Paper describes numerical fixes for softmax and cross-entropy
The authors say the fixes remove the Slingshot effect and stop student logits drifting from the teacher in distillation.
Stories per day · 30 days
1 RSS
3 stories, newest first · one card per story
Stories tagged #grokking
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Paper derives a spectral theory showing weight decay drives NTK feature learning that controls grokking timescale and phase structure.
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Theoretical framework bounds generalization for gradient descent with weight decay, deriving conditions that explain delayed generalization and grokking.