Double descent is the principle of least action
A statistical mechanics analysis explains double descent: finite-time diffusion induces effective weight decay that regularizes models as parameters grow.
The paper models stochastic gradient-based training as a particle diffusing over the training-loss energy landscape at an induced temperature, sampling parameters via a Boltzmann distribution. Finite training time carries an effective weight decay, making every parameter a quadratic degree of freedom governed by the equipartition theorem. Adding parameters at fixed training loss lowers the temperature and the L2 norm of the stationary path, increasing effective regularization and explaining the double descent phenomenon.
- Training framed as a particle obeying Boltzmann statistics on the loss landscape
- Finite-time diffusion induces effective weight decay per parameter
- Adding parameters lowers stationary-path L2 norm, strengthening regularization
From the original article
1 min read at arxiv.orgThe test error of a model plotted against its number of parameters $d$ falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics.
By Congzhou M Sha · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.