The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Edge0 streams 35B-parameter MoE inference from SSD on a 24GB machine at 20 tokens/s using a one-token-ahead prerouter, with framework and checkpoints open-sourced.
The paper (arXiv 2609.18063) presents Edge0, a streaming MoE inference engine whose per-layer prerouter predicts the next layer's expert routing one token ahead so SSD reads can be pre-staged with nothing dropped. An unmerged recovery LoRA trained on the student path compensates for quality lost to int4 quantization and routing replacement. On a single 24GB machine it serves a 35B MoE at 20 tok/s within 3GiB of peak active memory, within a few points of its fp16 teacher across five public benchmarks; framework, checkpoints, and adapters are open source.
- Per-layer prerouter predicts the next layer's expert routing one token ahead, enabling pre-staged SSD reads with zero dropped experts.
- Unmerged recovery LoRA trained on the student path recovers quality lost to int4 quantization and routing replacement.
- Serves a 35B MoE at 20 tok/s in 3GiB peak active memory on 24GB hardware, near fp16 teacher accuracy.
- An 8B tier is supported; framework, checkpoints, and adapters are released open source.
From the original article
1 min read at huggingface.co
Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19. 5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute.
By Yu Lin, Yiming Wang, Runyuan Cai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.