Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Four scheduling techniques bound all memory peaks in long-context MoE training, enabling 120B-667B models at 1M-token context with up to 10.4x throughput.
The paper addresses memory peaks in long-context Mixture-of-Experts training by bounding four unbounded components: expert dispatch with the routing matrix, vocabulary projection, gradient checkpoint boundaries, and optimizer state. It introduces PipelinedLLEP (capped token contributions per dispatch chunk), Ring-DTP (ring circulation of activations or weight shards with online log-sum-exp), Selective Checkpoint Offload (SCO), and OffloadStreamAdamW, all preserving exact loss and gradients. Composed on MoE models from 120B to 667B parameters, the methods enable training at 1M context length, 8-32x the context reach of a tuned FSDP2 baseline, and up to 10.4x its throughput.
- Bounds four memory peaks: expert dispatch, vocabulary projection, checkpoint boundaries, optimizer state.
- Introduces PipelinedLLEP, Ring-DTP, Selective Checkpoint Offload, and OffloadStreamAdamW schedules.
- Cuts dispatch peak by up to 59.3% and vocabulary projection peak by 86.6% without throughput loss.
- Enables 120B-667B MoE training at 1M context, up to 10.4x FSDP2 baseline throughput.
- Products
- FSDP2
From the original article
1 min read at huggingface.co
Training a Mixture-of-Experts (MoE) model at long context or large batch size fails as soon as any one component's peak allocation exceeds device memory, so the target is every peak at once, not the average footprint.
By Shrey Pandit, Xuan-Phi Nguyen, Yiran Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.