Mesoscopic theory describes SGD feature learning in wide two-layer networks
Original title: Mesoscopic learning dynamics of extensive-width neural networks
Gaussian overlap statistics track linear and quadratic learning until specialization, which quadratic shielding accelerates.
The paper studies two-layer networks learning target functions of the same architecture, with arbitrary nonlinearity, as width and input dimension grow proportionally. A mesoscopic description tracks the distribution of overlaps between learned and target features. Early on, a Gaussian closure reduces dynamics to collective variables for linear and quadratic components. Specialization is accelerated by quadratic shielding that filters gradient fluctuations.
- Standard overlap tracking would require a diverging number of parameters in this regime.
- The activation function determines the pattern of specialization and later feature refinement.
- Quadratic shielding filters gradient fluctuations after linear and quadratic components are learned.
From the original article
1 min read at arxiv.orgUnderstanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central problem in machine learning. We study this question in two-layer neural networks learning target functions represented by networks of the same architecture, with arbitrary nonlinearity, and with network width and input dimension growing proportionally.
By Shu Zhou · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.