Broken scale symmetries in undercomplete linear autoencoders
Theory paper shows SGD drives undercomplete linear autoencoders toward large decoder weights along the PCA solution manifold until hitting a finite-stepsize stability boundary.
The authors study scale symmetries in homogeneous networks, which gradient flow preserves but SGD breaks. For undercomplete linear autoencoders, they show SGD induces a directed scale drift favoring large decoder weights on the PCA solution manifold, with analytically tractable slow dynamics. The drift terminates at a finite-stepsize stability boundary, producing sharper solutions per the maximum Hessian eigenvalue, though different sharpness measures can diverge. This illustrates how loss geometry converts residual gradient noise into directed motion along functionally equivalent solutions.
- SGD breaks scale symmetry in undercomplete linear autoencoders, favoring large decoder weights
- Directed drift admits an analytically tractable effective description on slow timescales
- Drift ends at a finite-stepsize stability boundary, yielding sharper-than-balanced solutions
- Different sharpness measures can move in opposing directions
From the original article
1 min read at arxiv.orgNeural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD). A canonical example of such a symmetry is scale in homogeneous networks: one can scale up the parameters in one layer and down in the next without changing the network output.
By Farhad Pashakhanloo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.