EchoMoE cuts MoE expert-parallel traffic by sending compressed differences
Original title: Communicate the Change: Efficient Expert Parallelism via Activation Compression
The retrofit needs about 8,500 times fewer tokens than full pretraining and ships with a vLLM inference integration.
EchoMoE modifies mixture-of-experts models so expert-parallel inference sends a compressed difference from an earlier residual reference instead of full hidden states. The paper reports 58% less inter-node communication volume and 22.6% higher throughput, with average benchmark performance within 3.8 percentage points. It also reports up to 2.38x inference throughput over bandwidth-constrained Ethernet and tolerance of quantized communicated representations. An implementation for pretrained MoE models and a vLLM integration are released.
- Each MoE layer normally needs dispatch and combine collectives.
- Reference states are refreshed periodically; only compressed differences are sent synchronously.
- Quality stayed within 3.8 percentage points across 18 benchmarks.
- Quantizing communicated representations caused no measurable quality loss, the authors say.
From the original article
1 min read at arxiv.orgScaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices.
By Fengyuan Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.