Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
Clipped decentralized SGD reaches order-optimal rates under heavy-tailed noise with linear agent speed-up.
The paper studies clipped decentralized SGD for smooth non-convex objectives under bounded p-th moment heavy-tailed noise, with p between 1 and 2. It shows order-optimal convergence both in expectation and with high probability, together with a linear speed-up in the number of agents. A consensus-gap analysis attributes the result to clipping preserving gradient magnitude, unlike normalization, which can fail to converge in the decentralized setting. Numerical experiments are reported to support the theory.
- Clipped decentralized SGD is analyzed under heavy-tailed gradient noise.
- Rates are order-optimal in expectation and with high probability.
- The method obtains a linear speed-up in the number of agents.
- Clipping retains magnitude information that normalization can discard.
From the original article
1 min read at arxiv.orgHeavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization. While these methods are well understood in centralized settings, much less is known in decentralized ones, where applying a nonlinearity to local gradients affects both optimization and consensus.
By Aleksandar Armacki · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.