Replacement operators affect training-free MoE expert pruning
Original title: What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning
Survivors gate scale choice dominates effects in pruning OLMoE, DeepSeek-V2-Lite and Moonlight models; tilted novice operator adds 1.2-1.8% parameters and improves accuracy.
Published training-free expert pruning methods change both expert removal and router behavior for removed experts. Holding mask fixed researchers compared replacement operators across three families plus Granite router on OLMoE, DeepSeek-V2-Lite, Moonlight. Largest effect comes from survivors gate scale. On routers trained without top-k renormalization renormalizing gates costs 0.8-8.5 points and reverses rerouting benefits. Tilted novice closed-form low-rank map adds 1.2-1.8% parameters at 50% pruning and gains 3.45 and 1.37 points over constant on OLMoE and Moonlight. Evaluation domain changes conclusions dramatically.
- Trained-scale rerouting best on DeepSeek-V2-Lite
- Tilted novice statistically indistinguishable from partial copy on two models
- MoNE constant keeps 92-97% dense accuracy but only 5-65% GSM8K and 0-17% code scores
- Calibration domain moves mask 4-9 times more than resampling
From the original article
1 min read at huggingface.co
Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests.
By Ezza Ali, Sehrish Bibi, Momina Moetesum · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.