Scheduling Recursive Reasoning in Looped Transformers
TAPS adapts recurrent step size in looped transformers, raising accuracy and cutting loops without retraining.
The paper analyzes how recurrent update scale affects terminal loss in looped transformers and introduces TAPS, a scheduler that balances persistent progress against centered fluctuation. TAPS adapts step size online and, under stated conditions, reduces expected terminal loss and reaches a target quality in fewer loops. Empirically it improves accuracy on structured reasoning tasks without retraining; folding the same principle into training adds further gains and up to 1.56x wall-clock speedup at matched baseline accuracy.
- Splits update-scale sensitivity into progress and fluctuation terms
- Adapts step size online without retraining the recurrent model
- Training-time TAPS adds up to 1.56x wall-clock speedup at matched accuracy
- Effective across recurrent architectures and inference strategies
From the original article
1 min read at huggingface.co
Recurrent reasoning models have attracted growing attention for scaling test-time computation, typically by iteratively refining latent states with shared parameters. However, these models apply each learned update with a fixed unit scale, which can be conservative when updates make persistent progress and overly aggressive when they fluctuate, limiting the benefit of additional loops.
By Boyuan Wang, Chengyao Yu, Jiaxi Ren · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.