Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
Priority-Constrained Descent optimizes hierarchical objectives while limiting distortion of the primary gradient.
Priority-Constrained Descent (PCD) is a gradient method for hierarchical multi-objective optimization in which a primary goal outranks secondary constraints such as sparsity, compression, or robustness. It preserves the primary descent direction and applies the smallest distortion needed to make progress on secondary objectives, controlled by tau in [0, 1]. The formulation is invariant to objective scaling and has exact closed-form solutions for two and three objectives. On structured compression, unstructured sparsity, low-rankness, and synthetic tasks, the authors report Pareto dominance and guaranteed secondary progress versus prior methods.
- PCD keeps the primary descent direction and applies only the distortion needed for secondary progress.
- A single tau in [0, 1] sets distortion strength; the method is scale-invariant.
- Closed-form solutions exist for two- and three-objective problems.
- Tests cover network compression, unstructured sparsity, low-rankness, and synthetic tasks.
From the original article
1 min read at huggingface.co
Deep learning problems rarely involve objectives that are equal in importance. A primary objective defines the goal, whilst secondary objectives, such as sparsity, compression, or robustness constrain the solution. While existing multi-objective methods have proven effective in practice, they have a clear symmetry problem and neglect the inherent objective hierarchy built into these objective spaces.
By Dara Varam, Mohamed I. Alhajri · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.