Optimal stepsize schedules for stochastic gradient descent in convex minimization
Original title: Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization
Exact analysis proves monotonically decreasing optimal stepsize schedules for SGD under bounded noise in smooth convex problems.
The work derives exact worst-case expected suboptimality for SGD final iterate with any schedule of N stepsizes in [0,1.5]^N. When variance is at least Ω(1/√N) the theory yields the minimax optimal schedule and proves its uniqueness. Schedules monotonically decrease, ruling out accelerations from fixed fractal schedules observed in the deterministic case.
- exact analysis for any N-step schedule
- matching objective and noise distributions
- monotonically decreasing optimal stepsize
- rules out accelerations in deterministic case
From the original article
1 min read at arxiv.orgThis work considers the design of stepsize schedules for stochastic gradient descent in smooth convex optimization. Throughout, we assume that gradient noise is conditionally zero-mean with bounded variance. Among problems with dimension $d\geq N+1$, we provide an exact worst-case analysis of the expected suboptimality of stochastic gradient descent's final iterate.
By Benjamin Grimmer · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.