Mesoscopic theory describes SGD feature learning in wide two-layer networks
Gaussian overlap statistics track linear and quadratic learning until specialization, which quadratic shielding accelerates.
Stories per day · 30 days
1 RSS
2 stories, newest first · one card per story
Stories tagged #feature-learning
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Paper derives a spectral theory showing weight decay drives NTK feature learning that controls grokking timescale and phase structure.