Learning rate decay can exponentially speed SGD on some nonconvex functions
Original title: Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions
Authors prove a benchmark class where decay uses exponentially fewer gradient queries than any fixed small step size.
A theory paper argues that learning rate decay gives an asymptotic advantage for global optimization of a natural class of nonconvex functions. In the small-step regime, SGD with decay needs exponentially fewer gradient queries than SGD with any fixed learning rate. The authors say the class includes many popular benchmark functions and that annealing solves the related non-logconcave sampling task in polynomial time via weak Poincaré inequalities.
Unconfirmed: The exponential advantage is a theoretical claim for a defined function class, not an empirical training result.
- The claimed gain is the first exponential advantage shown for learning rate decay.
- The comparison is against any fixed learning rate in the small-step regime.
- Annealing is tied to a non-logconcave sampling task solved in polynomial time.
From the original article
1 min read at arxiv.orgStochastic Gradient Descent (SGD) is a workhorse algorithm for continuous optimization. It is a folk belief among deep learning practitioners that learning rate decay in SGD substantially improves performance. However, it has remained unclear whether learning rate decay offers an asymptotic advantage for global nonconvex optimization.
By Guneykan Ozgul · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.