Higher-order pruning of experts in mixture-of-experts language models
New HOPE method uses second-order objectives to prune Mixture-of-Experts LLMs, outperforming REAP at 50% pruning on models up to 122B parameters.
Researchers derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective for Mixture-of-Experts language models that provably minimizes an upper bound on pruning error by modeling cooperative expert interactions. They show REAP, a state-of-the-art first-order method, is a special case of HOPE with interaction terms ignored. Across three frontier MoE models up to 122B parameters, two calibration sets, and benchmarks covering math, instruction following, coding, and agentic tasks, HOPE achieves the best average rank (1.58 of 5 at 50% pruning versus 2.42 for REAP), with gains up to +6.1% on agentic coding.
- HOPE minimizes an upper bound on pruning error via second-order expert interactions
- REAP is a special case of HOPE with interaction terms ignored
- At 50% pruning, average rank 1.58 of 5 versus 2.42 for next-best REAP
- Largest gains on agentic coding workloads, up to +6.1%
- Evaluated on three frontier MoE models up to 122B parameters
From the original article
1 min read at arxiv.orgMixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative.
By Alex M. Tseng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.