Improving Test-Time Scaling with Adaptive Looped Transformers
TaH2 adaptively loops tokens that benefit, steepening the AIME accuracy-compute slope 53 percent versus a non-looped baseline.
The paper studies whether looped transformers improve the accuracy-compute slope of test-time scaling as outputs get longer. Existing looped models often have steeper slopes than a non-looped baseline but underperform at matched compute because fixed-depth looping spends extra iterations on every token. TaH2 jointly post-trains the backbone and an iteration decider with lookahead depth supervision so extra loops go only to tokens that benefit. On AIME, it raises the slope 53 percent (2.74 versus 1.79) and beats the baseline peak by about 3.4 points at matched test-time compute, with the gain growing from +2.8 points at depth 2 to +3.9 at depth 8.
- Many tokens gain nothing from extra fixed-depth loop iterations.
- Lookahead labels jointly train the backbone and an iteration decider.
- AIME accuracy-compute slope improves 53 percent, 2.74 versus 1.79.
- Matched compute accuracy exceeds the baseline by about 3.4 points.
- Advantage grows from 2.8 points at depth 2 to 3.9 at depth 8.
From the original article
1 min read at arxiv.orgLooped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains underexplored.
By Yichen You · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.