Skip to content
NewA calmer, faster ZeroHour

CECAR cache-aware routing speeds on-device Qwen3-30B-A3B inference

Original title: CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

arXiv cs.AI / cs.LG / cs.CLby Byeongju Kim ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

Authors report 12.5 tokens/s on a consumer GPU with 16 GB VRAM, a 4.5x speedup, with non-resident experts fetched from SSD.

CECAR is a Mixture-of-Experts inference system that jointly routes experts and manages cache. An ML cache policy predicts each expert's expected reuse distance and prefers experts with smaller predicted reuse. The authors argue binary cache-presence routing can be unstable and that even Belady-optimal cache hits are not enough for low-latency on-device inference. On Qwen3-30B-A3B, they say decode speed exceeds baselines that keep the full model resident in DRAM. Code is on GitHub.

Unconfirmed: Speed figures are the authors' measurements on one consumer-GPU setup.

Key facts
  • Non-resident experts are loaded from SSD rather than kept fully in GPU memory.
  • The reported device has 16 GB of VRAM.
  • Claimed throughput is 12.5 tokens/s, 4.5x over conventional MoE inference.
  • Authors say cache hit rates stay bounded even under Belady's optimal policy.
  • Code is at github.com/hoonki2002/CECAR.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident.

arXiv cs.AI / cs.LG / cs.CL

By Byeongju Kim · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.