Rethinking Routing Exploration in Modern Mixture-of-Experts
MirrorMoE fixes MoE routing under-exploration from shared experts by suppressing unneeded experts per token.
The paper argues that modern mixture-of-experts designs combining fine-grained expert segmentation with always-on shared experts create an exploration dilemma. Shared experts reduce token-representation variability just as finer segmentation combinatorially enlarges the routing space, so added expressiveness may go unused. MirrorMoE shrinks that space by token-adaptively suppressing unnecessary experts. The authors report consistent gains across model scales.
- Fine-grained experts enlarge the routing space while shared experts reduce exploration.
- That mismatch can leave extra MoE expressiveness unused.
- MirrorMoE shrinks the space by token-adaptively suppressing unnecessary experts.
- The method improved performance consistently across model scales.
From the original article
1 min read at arxiv.orgMixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token.
By Bingcong Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.