Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model recurrence and sparsity, predicting looped MoE loss and guiding compute-efficient design.
The paper introduces Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity with model size and data for looped Mixture-of-Experts transformers. A bounded, sparsity-conditional recurrence mapping estimates the effective-parameter gain from looping. The laws predict held-out loss more accurately than prior alternatives and recover dense and MoE laws as special cases. At matched compute and trillion-token scale, a law-derived looped MoE matches a roughly 2x larger non-looped MoE on reasoning, with about 3x active-parameter efficiency from sparsity and about 2x total-parameter efficiency from recurrence.
- Loop Scaling Laws jointly model recurrence, sparsity, size, and data.
- Sparsity yields about 3x active-parameter efficiency; recurrence about 2x on reasoning.
- At matched compute, a looped MoE matches a roughly 2x larger non-looped MoE.
- Laws recover standard dense and MoE scaling laws as special cases.
From the original article
1 min read at huggingface.co
Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation.
By Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.