Learning rate decay can exponentially speed SGD on some nonconvex functions
Authors prove a benchmark class where decay uses exponentially fewer gradient queries than any fixed small step size.
Stories per day · 30 days
1 RSS
2 stories, newest first · one card per story
Stories tagged #optimization-theory
Silver Rate Is (Almost) Optimal for Gradient Descent Acceleration
New optimization theory paper proves near-optimal lower bounds for gradient descent with predetermined stepsizes, confirming silver-schedule optimality.