Scaling law favors full BPTT over truncated looped transformers
Original title: Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters.
A study of nearly 600 training runs varies model size, tokens, and forward and backward loop counts, then fits a scaling law for looping and truncation through effective parameters. Truncation saves compute, but the fitted law favors full backpropagation through time when loop counts are jointly optimized. With model size free, two fully trained loops are best and use only about 8% more training compute than the best non-looped model.
Unconfirmed: The scaling-law conclusions and loss comparisons are the authors' fitted results.
- Matched-compute runs show full BPTT through fewer loops beats truncated settings.
- At fixed model size, optimal loop count and token budget both grow with compute.
- Two fully trained loops are the best looped setting when model size is free.
- That setting uses about 8% more training compute than the best non-looped model.
From the original article
1 min read at huggingface.co
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping.
By Xuyuan Xiong, Siyu Chen, Zihan Shao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.