Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Routing drift after MoE merging does not reliably indicate failure; source-route restoration often lacks task benefit.
The paper asks whether expert-routing drift after merging Mixture-of-Experts LLMs is evidence of routing failure, studying DeepSeekMoE, OLMoE, and Qwen3-MoE. Counterfactual swaps show most expert reassignments come from input shifts rather than same-layer parameter changes, and restoring source routes does not reliably improve next-token likelihood or task loss. Recoverable loss appears under deliberate router corruption, but source-specialist likelihood does not reliably mark useful local repairs. The authors release an analysis toolkit and Selective Router Repair code.
- Studies routing drift in DeepSeekMoE, OLMoE, and Qwen3-MoE
- Most expert reassignments come from input shifts, not parameter changes
- Source-route restoration does not reliably improve task loss
- Defines routing failure as loss recoverable by a routing intervention
- Releases a routing analysis toolkit and Selective Router Repair code
- AI models
- DeepSeekMoEOLMoEQwen3-MoE
From the original article
1 min read at huggingface.co
Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such routing drift is often interpreted as routing failure, raising a fundamental question that remains unclear: does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?
By Yuanyi Wang, Yanggan Gu, Su Lu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.