Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model looped recurrence and MoE sparsity, matching larger models at equal compute.
The paper introduces Loop Scaling Laws, which jointly model recurrence, mixture-of-experts sparsity, model size, and data via a bounded sparsity-conditional recurrence mapping. The laws predict held-out loss of looped models more accurately than prior alternatives and recover dense and MoE scaling laws as special cases. Sparsity delivers about 3x active-parameter efficiency and recurrence about 2x total-parameter efficiency on reasoning; at trillion-token scale and matched training compute, a law-derived looped MoE matches a roughly 2x larger non-looped MoE on reasoning benchmarks.
- First scaling law jointly covering recurrence, sparsity, size, and data
- Recovers dense and standard MoE scaling laws as special cases
- Sparsity gives about 3x active-parameter efficiency
- At matched compute, looped MoE matches a roughly 2x larger model
From the original article
1 min read at arxiv.orgLooped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation.
By Yanbei Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.