When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Researchers derive an exact law linking learning-rate schedules and weight decay in normalized networks, pinpointing when scale-invariant optimization destabilizes.
The paper shows that normalization makes large parts of neural networks scale-invariant, creating a hidden feedback loop where learning-rate schedules and weight decay interact through the parameter norm to control the effective optimizer step. An exact discrete-time law with a single scalar quantity separates contraction- and expansion-dominated effective learning-rate regimes, and the balance point is intrinsically unstable, so constant learning rate with weight decay produces recurrent behavior instead of a stable equilibrium. A unified homogeneous-optimizer framework explains why adaptive methods stabilize more weakly under normalization. The law is validated with high precision on MLPs, CNNs, and GPT-2 across MNIST, CIFAR, WikiText, and OpenWebText, with code released on GitHub.
- Single scalar governs schedule and weight-decay forcing on the effective learning rate.
- Constant learning rate with weight decay cannot hold an interior equilibrium; behavior is recurrent.
- Unified framework explains weaker stabilization of adaptive optimizers under normalization.
- Predictions validated on MLP, CNN, and GPT-2 across MNIST, CIFAR, WikiText, OpenWebText.
From the original article
1 min read at arxiv.orgNormalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer.
By Hasan Amin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.