Sonar-MoE Enables Memory-Efficient MoE Training
Original title: Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
Sonar-MoE updates expert tensors in the backward pass to cut peak memory by 19-22% for MoE models.
MoE models consume growing GPU memory for expert gradients during training. Sonar-MoE moves expert updates to the backward pass and frees gradients immediately. It achieves bitwise-identical results to standard training without clipping experts. Peak memory drops 19-22% with 17-23% faster steps on OLMoE-1B-7B and Granite-3.1-3B-A800M.
- Peak memory reduced by 19-22%
- Updates are bitwise identical
- Held-out loss within 0.003 of baselines
- Faster steps than global-norm clipping
From the original article
1 min read at arxiv.orgMost parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away.
By Avni Mittal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.