Nereus: Adaptive Parallelism for LLM Post-Training
Nereus adapts parallelism during LLM reinforcement-learning post-training, cutting step latency 27.7% versus a fixed layout.
Nereus is a cost-aware runtime that retargets reinforcement-learning post-training when GPU availability, sequence length, memory pressure, or stage bottlenecks change. It represents each model-stage replica as an Elastic Model Unit and orders GPU transfers with a global transition graph. On a real-data trace, online tensor- and pipeline-parallel adaptation cut average step latency 27.7% versus a fixed layout with data-parallel scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions used 0.079% of runtime, and 8B PPO throughput rose 2.14–7.27× over OpenRLHF and 1.10–1.47× over Verl.
- Cost-aware controller admits a new plan only if the transition pays off.
- Elastic Model Units represent each model-stage replica's distributed state.
- Online TP/PP adaptation cut average step latency by 27.7%.
- Six transitions used 0.079% of a 1,024-GPU run.
- 8B PPO throughput improved up to 7.27× over OpenRLHF.
From the original article
1 min read at huggingface.co
Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time.
By Songlin Jiang, Tuo Shi, Sitong Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.