How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Researchers show model growth via looped transformers improves scaling exponents; a 7.4B architecture matches GPT-3 13B with roughly 20x less compute.
The paper shows that architectural interventions, contrary to conventional wisdom, can modify pre-training scaling exponents and yield exponential performance gains with compute. Looped transformers with increasing loop counts provide a model growth mechanism; a 7.4B model-growth architecture matches GPT-3 13B on CORE with roughly 20x less compute, with efficiency gains that increase with scale. A boundary operator that normalizes and injects an earlier block also improves compute efficiency, and in data-constrained multi-epoch settings increasing loops with scale is compute-optimal.
- Architectural interventions can modify pre-training scaling exponents.
- 7.4B model-growth architecture matches GPT-3 13B on CORE with ~20x less compute.
- Boundary operator injecting an earlier normalized block also improves compute efficiency.
- In data-constrained multi-epoch training, increasing loop counts with scale is compute-optimal.
- Products
- GPT-3looped transformer
From the original article
1 min read at arxiv.orgScaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to exponential improvements in performance with increases in computation. As an anchoring point, we consider the architectural formulation of looped transformers.
By Zixi Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.