How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention in looped MoE models, lowering pretraining loss at equal parameters and compute.
Foil is a recipe for looping sparse mixture-of-experts models while holding expert parameters and per-token expert compute fixed. It flattens experts—halving expert layers, doubling experts per layer, and doubling passes—and gives each pass its own attention parameters while sharing experts and routers. At 20B tokens every Foil model beats the unflattened looped baseline on pretraining loss; at 100B tokens loss improves monotonically with flattening, and the most flattened variant is 0.012 nat lower at equal parameters and compute, with downstream accuracy on par or better.
- Foil halves expert layers while doubling experts per layer and passes.
- At 100B tokens the most flattened model is 0.012 nat below baseline.
- Untied attention produces more balanced and more confident routing.
- Looping gains and wider expert layers amplify each other.
From the original article
1 min read at arxiv.orgLooped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a question: how to loop a MoE?
By Shouren Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.