Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Researchers find MoE routing drift after merging does not reliably indicate routing failure.
The paper tests whether expert routing drift after merging Mixture-of-Experts LLMs indicates routing failure. Across DeepSeekMoE, OLMoE, and Qwen3-MoE, counterfactual interventions attribute most expert reassignments to input shifts rather than same-layer parameter changes. Source-route restoration poorly predicts next-token likelihood gains, so the authors define failure as task loss recoverable under a routing intervention. Selective Router Repair does not reliably convert source-specialist token advantages into beneficial corrections, and the toolkit and code are released.
- Most expert reassignments come from input shifts, not same-layer parameter changes.
- Source-route restoration does not reliably recover likelihood or task loss.
- Routing failure is task loss recoverable by a specified routing intervention.
- Selective Router Repair does not reliably find beneficial local corrections.
From the original article
1 min read at arxiv.orgModel merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such \emph{routing drift} is often interpreted as routing failure, raising a fundamental question that remains unclear: \emph{does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?
By Yuanyi Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.