Learning rate decay can exponentially speed SGD on some nonconvex functions
Authors prove a benchmark class where decay uses exponentially fewer gradient queries than any fixed small step size.
Stories per day · 30 days
Relatedmixture-of-experts1nonconvex-optimization1optimization1optimization-theory1pretraining1sgd1weight-normalization1
2 stories, newest first · one card per story
Stories tagged #learning-rate
AIR schedule targets effective learning rate, not nominal rate
On 128 billion-token MoE pretraining, AIR lowers terminal training loss by about 0.01 versus cosine at matched cumulative learning rate.