A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Theoretical framework bounds generalization for gradient descent with weight decay, deriving conditions that explain delayed generalization and grokking.
The paper proves convergence of gradient descent with weight decay to a neighborhood of global minimizers of the empirical l2 loss for a broad class of neural networks. It decomposes population error into data, optimization, and prediction variation errors, deriving cellwise and layerwise approximate-homogeneity bounds on prediction variation along the training trajectory. The resulting necessary and sufficient conditions explain layerwise generalization differences and provide a theoretical characterization of grokking.
- Proves gradient descent with weight decay converges near empirical loss minimizers
- Decomposes population error into data, optimization, and prediction variation terms
- Derives sufficient condition characterizing delayed generalization and grokking
From the original article
1 min read at arxiv.orgUnderstanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training.
By Yuqing Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.