Improving Test-Time Scaling with Adaptive Looped Transformers
TaH2 adaptively loops layers on hard tokens, steepening AIME test-time scaling by 53% over a non-looped baseline.
The paper studies whether looped transformers improve test-time scaling as decoding compute grows, finding fixed-depth looping often has a steeper accuracy-compute slope but still trails a non-looped baseline at matched compute. TaH2 adds an iteration decider, jointly post-trained with lookahead depth supervision, so extra loops are spent only on tokens that benefit. On AIME, TaH2 improves the accuracy-compute slope by 53% (2.74 versus 1.79) and exceeds the baseline peak by about 3.4 points at matched compute, with the gap growing to +3.9 points at depth 8.
- Fixed-depth looping wastes extra iterations on tokens that do not benefit.
- TaH2 jointly trains the backbone and an iteration decider with lookahead depth labels.
- AIME accuracy-compute slope rises 53%, from 1.79 to 2.74.
- About 3.4 points above the non-looped peak at matched test-time compute.
- Advantage grows from +2.8 points at depth 2 to +3.9 at depth 8.
From the original article
1 min read at huggingface.co
Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains underexplored.
By Yichen You, Tianyu Fu, Aosong Feng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.