A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Theoretical framework bounds generalization for gradient descent with weight decay, deriving conditions that explain delayed generalization and grokking.
The paper proves convergence of gradient descent with weight decay to a neighborhood of global minimizers of the empirical l2 loss for a broad class of neural networks. It decomposes population error into data, optimization, and prediction variation errors, deriving cellwise and layerwise approximate-homogeneity bounds on prediction variation along the training trajectory. The resulting necessary and sufficient conditions explain layerwise generalization differences and provide a theoretical characterization of grokking.
12