MuonGS relaxes weight-norm control for LLM pretraining
On dense models it reports about 15% token-equivalent speedup versus MuonH at matched validation loss.
Stories per day · 30 days
1 RSS
5 stories, newest first · one card per story
Stories tagged #weight-decay
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Weight decay causally regulates massive-activation scale in transformers, with peak magnitude scaling near lambda to the -1/2.
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Paper derives a spectral theory showing weight decay drives NTK feature learning that controls grokking timescale and phase structure.
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Researchers derive an exact law linking learning-rate schedules and weight decay in normalized networks, pinpointing when scale-invariant optimization destabilizes.
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Theoretical framework bounds generalization for gradient descent with weight decay, deriving conditions that explain delayed generalization and grokking.