Two-ReLU gradient descent collapses to bisector then specializes
Original title: Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics
Gradient flow on two-neuron ReLU model first collapses to bisector saddle before escaping to specialize to teacher.
Analysis of a minimal two-neuron ReLU teacher-student model under Gaussian inputs reveals multi-phase dynamics. Neurons collapse toward a bisector saddle before escaping and specializing. The loss landscape contains only global minima or saddles as critical points.
- Student neurons collapse to bisector saddle
- Every critical point with nonzero weights is global minimum or saddle
- Explains multi-phase symmetry breaking in multi-index models
From the original article
1 min read at arxiv.orgUnderstanding how gradient descent learns features remains a central challenge in neural network theory. We study this question in a minimal multi-index setting that already exhibits rich multi-phase gradient dynamics: a well-specified two-neuron ReLU teacher-student model under isotropic Gaussian inputs, trained by population gradient flow from small initialization.
By Binghua Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.