Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
A symmetry-breaking account explains clustered Hessian spectra in trained deep networks, including ReLU, CNN, graph, and transformer models.
The paper argues that clustered Hessian spectra—a large bulk near zero plus isolated outliers—arise by departing from a nearby, highly symmetric reference configuration. In that reference, invariances beyond ordinary weight symmetries force high-dimensional kernels and eigenvalues of large multiplicity; returning to the trained model breaks the symmetry and produces the observed clusters. The framework includes a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models and the neural tangent kernel. The same mechanism yields analogous spectra in layerwise Hessians and the Gauss-Newton matrix.
- Hessian outliers emerge when a hidden symmetric reference is broken
- Symmetry forces high-multiplicity eigenvalues and large kernels
- Covers three-layer ReLU nets, CNNs, graph models, transformers, and the NTK
- The same structure appears in layerwise Hessians and the Gauss-Newton matrix
From the original article
1 min read at arxiv.orgHessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers. This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference.
By Yossi Arjevani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.