The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Edge0 streams 35B-parameter MoE inference from SSD on a 24GB machine at 20 tokens/s using a one-token-ahead prerouter, with framework and checkpoints open-sourced.
The paper (arXiv 2609.18063) presents Edge0, a streaming MoE inference engine whose per-layer prerouter predicts the next layer's expert routing one token ahead so SSD reads can be pre-staged with nothing dropped. An unmerged recovery LoRA trained on the student path compensates for quality lost to int4 quantization and routing replacement. On a single 24GB machine it serves a 35B MoE at 20 tok/s within 3GiB of peak active memory, within a few points of its fp16 teacher across five public benchmarks; framework, checkpoints, and adapters are open source.
33