Learning rate decay can exponentially speed SGD on some nonconvex functions
Authors prove a benchmark class where decay uses exponentially fewer gradient queries than any fixed small step size.
Stories per day · 30 days
5 stories, newest first · one card per story
Stories tagged #sgd
Broken scale symmetries in undercomplete linear autoencoders
Theory paper shows SGD drives undercomplete linear autoencoders toward large decoder weights along the PCA solution manifold until hitting a finite-stepsize stability boundary.
Sharp Stationary Gaussian Approximation for Constant-Stepsize SGD
Paper proves a sharp Gaussian approximation for the stationary law of constant-stepsize SGD.
Double descent is the principle of least action
A statistical mechanics analysis explains double descent: finite-time diffusion induces effective weight decay that regularizes models as parameters grow.
Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent
Quantile-k-Loss SGD filters corrupted component losses by quantile sampling, proving linear convergence while outperforming standard and min-k-loss SGD.