Decoding Looped Transformers Better for (Almost) Free
Training-free LoopCD improves looped Transformer decoding and can cut recurrent compute by up to 48%.
LoopCD is a training-free contrastive decoder for looped Transformers that uses an earlier recurrent state to guide the final next-token prediction, either in logit space or hidden-state space. Across four looped Transformer families it improves full-depth decoding: LoopCD-Logits raises Ouro-2.6B-Thinking AIME 2024 pass@1 from 61.88% to 73.33%, and LoopCD-Hidden raises Huginn HumanEval pass@1 from 22.56% to 31.71%. The gains let models halve recurrent loops while matching or beating unguided full-depth baselines, cutting forward FLOPs by 22.5% to 48.2%.
- LoopCD contrasts the final loop with an earlier recurrent pass, with no extra training.
- LoopCD-Logits raises Ouro-2.6B-Thinking AIME 2024 pass@1 from 61.88% to 73.33%.
- LoopCD-Hidden lifts Huginn HumanEval pass@1 from 22.56% to 31.71%.
- Halving recurrent loops still matches full-depth baselines, cutting FLOPs 22.5% to 48.2%.
- AI models
- Ouro-2.6B-ThinkingHuginn
From the original article
1 min read at arxiv.orgLooped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training.
By Weihao Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.