How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Researchers show model growth via looped transformers improves scaling exponents; a 7.4B architecture matches GPT-3 13B with roughly 20x less compute.
The paper shows that architectural interventions, contrary to conventional wisdom, can modify pre-training scaling exponents and yield exponential performance gains with compute. Looped transformers with increasing loop counts provide a model growth mechanism; a 7.4B model-growth architecture matches GPT-3 13B on CORE with roughly 20x less compute, with efficiency gains that increase with scale. A boundary operator that normalizes and injects an earlier block also improves compute efficiency, and in data-constrained multi-epoch settings increasing loops with scale is compute-optimal.
42