Higher-order pruning of experts in mixture-of-experts language models
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
Researchers derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective for Mixture-of-Experts language models that provably minimizes an upper bound on pruning error by modeling cooperative expert interactions. They show REAP, a state-of-the-art first-order method, is a special case of HOPE with interaction terms ignored. Across three frontier MoE models up to 122B parameters, two calibration sets, and benchmarks covering math, instruction following, coding, and agentic tasks, HOPE achieves the best average rank (1.58 of 5 at 50% pruning versus 2.42 for REAP), with gains up to +6.1% on agentic coding.
- HOPE minimizes an upper bound on pruning error via second-order expert interactions
- REAP is a special case of HOPE with interaction terms ignored
- At 50% pruning, average rank 1.58 of 5 versus 2.42 for next-best REAP
- Largest gains on agentic coding workloads, up to +6.1%
- Evaluated on three frontier MoE models up to 122B parameters
Full article231 words · extracted from arxiv.org · click to collapse
Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective which provably minimizes an upper bound on the error resulting from pruning. We show that REAP (a state-of-the-art first-order pruning method) is a special case of HOPE where interaction terms are ignored. Across three frontier MoE models (up to 122B parameters), two distinct calibration sets, and multiple benchmarks (including math, instruction following, coding, and an agentic suite), we demonstrate that HOPE produces better pruning decisions than existing methods, and its advantage is most pronounced at high pruning rates and on challenging agentic workloads. At 50% pruning, HOPE outperforms all baselines and achieves an average rank of 1.58 out of 5 methods (versus 2.42 for the next-best method, REAP), with gains of up to +6.1% on agentic coding. Over all conditions, HOPE again achieves the best average rank and surpasses every other method in the majority of head-to-head comparisons. By preserving cooperative expert structure that first-order methods ignore, HOPE enables aggressive compression with minimal degradation, particularly on complex tasks where diverse expert combinations are invoked over long sequences.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.18916