Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
InfiLoop adds loop-native residuals so looped Transformers keep improving past 20,000 test-time steps.
The paper argues looped Transformers need dedicated residual connections because extra iterations can overwrite correct intermediate reasoning. InfiLoop combines content-based weighting with learned temporal decay and a constant-memory streaming recurrence. A 7M-parameter model reaches 97.9% exact accuracy on Sudoku-Extreme and 13.6% pass@2 on ARC-AGI-2, and Sudoku accuracy keeps rising beyond 20,000 effective test-time steps.
- Looped Transformers lose accuracy as noisy updates overwrite correct intermediate deductions.
- InfiLoop learns which past states to keep and how much of each update to accept.
- A 7M-parameter model reaches 97.9% exact accuracy on Sudoku-Extreme.
- It scores 13.6% pass@2 on ARC-AGI-2 and still improves past 20,000 steps.
From the original article
1 min read at huggingface.co
In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions.
By Pengxiang Li, Dilxat Muhtar, Di He · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.