Decoding Looped Transformers Better for (Almost) Free
LoopCD uses earlier recurrent states for contrastive decoding, boosting looped Transformers while cutting inference FLOPs.
Researchers introduce LoopCD, a training-free contrastive decoding method for looped Transformers that contrasts the final prediction with an earlier recurrent pass. LoopCD-Logits works in logit space with one extra output pass, while LoopCD-Hidden uses hidden states with no output overhead. On Ouro-2.6B-Thinking, LoopCD-Logits raises AIME 2024 pass@1 from 61.88% to 73.33%, and LoopCD-Hidden lifts Huginn HumanEval pass@1 from 22.56% to 31.71%. Those gains allow halving recurrent loops while matching full-depth baselines, cutting forward FLOPs by 22.5% to 48.2%.
- LoopCD contrasts final and earlier loop predictions without extra training.
- LoopCD-Logits lifts Ouro-2.6B-Thinking AIME 2024 pass@1 from 61.88% to 73.33%.
- LoopCD-Hidden raises Huginn HumanEval pass@1 from 22.56% to 31.71%.
- Halving recurrent loops still matches baselines while cutting FLOPs 22.5–48.2%.
- AI models
- Ouro-2.6B-ThinkingHuginn
From the original article
1 min read at huggingface.co
Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training.
By Weihao Liu, Huangjie Zheng, Tianrong Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.