Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections
Researchers characterize minimal-norm univariate two-layer ReLU classifiers and show skip connections make every KKT point globally optimal.
The paper geometrically characterizes optimal binary classifiers for univariate two-layer ReLU networks under minimal-norm interpolation and weakly L2-regularized logistic loss. Unpenalized biases produce piecewise-affine interpolators that follow every label switch, while penalized biases yield a unique sparsest positive-margin classifier with one kink in each intermediate same-label segment. A free affine skip connection leaves those function-space solutions unchanged but makes every KKT point globally optimal. Numerical experiments across dataset complexity and network width support the predicted landscape and sparsity behavior.
- Unpenalized-bias interpolators are piecewise-affine and hug every label switch.
- Penalized biases yield a unique sparsest classifier with one kink per segment.
- A free affine skip connection makes every constrained KKT point globally optimal.
- Experiments support the predicted optimization landscape and sparsity effects.
From the original article
1 min read at arxiv.orgWe study minimal-norm interpolation and $\ell_2$-regularized logistic-loss minimization for binary classification by univariate two-layer ReLU networks. We give complete geometric characterizations of the optimal classifiers in function space, resolving how the solutions depend on whether hidden-layer biases are included in the parameter norm.
By Karolina Drabik · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.