Multi-View MoE reuses multiple routers to shape expert computation
Original title: Multi-View MoE: Letting Every Router Shape Expert Computation
On a 24-layer 10B-A1B model trained on 708B tokens, nine-task mean rose 1.18 points.
Multi-View MoE aggregates scores from independently initialized routers and mixes their relative preferences into each selected expert's input. Each expert still runs one forward pass, so active-expert count, FLOPs, and all-to-all volume stay fixed. Full Multi-View improves a nine-task downstream mean by 1.18 percentage points; Multi-Router alone adds 0.6 points and raises rank correlation with expert utility.
- Single-router rankings agree weakly with expert utility in middle and deep layers.
- Rank correlation rises 8.1% globally and about 58% in the worst-aligned layer.
- Controlled mixing tests show expert- and token-dependent use of input differences.
From the original article
1 min read at arxiv.orgSparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts per token. Expert selection directly affects model capabilities under a fixed active-expert budget. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers.
By Zelin Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.