Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Stepped MoE combines elastic subnetworks with sparse routing so one model spans 1-4 billion parameters.
Stepped MoE combines elastically nested subnetworks with sparsely gated mixture-of-experts routing so one model can adapt to deployment limits and task needs. The backbone conditions on context and a target efficiency specification, giving inference-time control of the accuracy-efficiency trade-off. Experiments show 1, 2, 3, and 4 billion parameter settings that are 2-5% more accurate than dense counterparts on knowledge-intensive benchmarks and match static models on latency. Shared parameters reduce on-device disk use and let serving scale with available DRAM and compute.
- Combines elastic subnetworks with sparse MoE routing.
- One checkpoint spans 1 to 4 billion parameters.
- Beats dense models by 2-5% on knowledge benchmarks.
- Latency stays similar to comparable dense models.
- Shared weights reduce on-device disk use.
From the original article
1 min read at huggingface.co
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently.
By Arnav Kundu, Zhaoyang Xu, Bairu Hou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.