MoE channel selection works without calibration data, paper finds
Original title: Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
A weight-only reader keeps 98.1% of full accuracy at 17.8% of routed-expert reads and speeds MoE blocks versus vLLM.
The paper argues mixture-of-experts decoding does not need offline calibration to decide which expert channels to load. It says a threshold set for an 18% read budget on Prox actually averages 12.9% reads, while top-k matches or beats the threshold version in 40 of 43 comparisons and avoids repetition-loop collapse that calibrated thresholds cause on Gemma 4. A reader built from model weights uses a ternary sketch and a low-rank map; across three post-trained MoE models it retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels 1.60 to 1.90 times faster than optimized vLLM at batch one and up to 2.69 times faster at larger batches.
Unconfirmed: Accuracy and speed figures are the authors' benchmark results, not an independent reproduction.
- Top-k needs no calibration data and fixes the read count per token.
- All 23 generation-task comparisons favored or matched top-k at 13% and 18% budgets.
- Gemma 4 showed repetition-loop collapse under calibrated thresholds.
- Speedups are versus optimized vLLM on the same three models.
From the original article
1 min read at huggingface.co
Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data.
By Geunhyoung Kim, Sung Ju Hwang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.