A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Paper derives a spectral theory showing weight decay drives NTK feature learning that controls grokking timescale and phase structure.
The paper provides a quantitative theory of grokking in homogeneous networks trained with squared loss and L2 weight decay, showing residuals after memorization feed back into NTK dynamics. It predicts the grokking timescale is controlled by the product of learning rate and weight decay, with feature learning slowing logarithmically near a critical decay threshold. Predictions were validated on modular addition using an 84x90 grid of trained MLPs and a 42x45 grid with a one-block Transformer, both recovering the predicted phase geometry and inverse-product scaling. Task-aligned Fourier structure continued emerging in the NTK after training accuracy saturated.
- Theory links weight decay-driven NTK evolution to delayed generalization in grokking
- Grokking timescale scales inversely with product of learning rate and weight decay
- Validated on 84x90 MLP grid and 42x45 Transformer grid in modular addition
- Critical decay threshold above which task-aligned NTK structure cannot generalize
From the original article
1 min read at arxiv.orgIn grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization.
By Lenz Pracher · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.