CECAR cache-aware routing speeds on-device Qwen3-30B-A3B inference
Original title: CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
Authors report 12.5 tokens/s on a consumer GPU with 16 GB VRAM, a 4.5x speedup, with non-resident experts fetched from SSD.
CECAR is a Mixture-of-Experts inference system that jointly routes experts and manages cache. An ML cache policy predicts each expert's expected reuse distance and prefers experts with smaller predicted reuse. The authors argue binary cache-presence routing can be unstable and that even Belady-optimal cache hits are not enough for low-latency on-device inference. On Qwen3-30B-A3B, they say decode speed exceeds baselines that keep the full model resident in DRAM. Code is on GitHub.
Unconfirmed: Speed figures are the authors' measurements on one consumer-GPU setup.
- Non-resident experts are loaded from SSD rather than kept fully in GPU memory.
- The reported device has 16 GB of VRAM.
- Claimed throughput is 12.5 tokens/s, 4.5x over conventional MoE inference.
- Authors say cache hit rates stay bounded even under Belady's optimal policy.
- Code is at github.com/hoonki2002/CECAR.
From the original article
1 min read at arxiv.orgDespite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident.
By Byeongju Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.