What Makes Recurrence Effective in Looped Language Models?
Looped language models gain reasoning from recurrence beyond training, but extra loops can hurt knowledge.
The paper studies when recurrence helps looped language models, where parameter sharing adds inference depth without new parameters. Across budgets below, within, and beyond the training horizon, extra recurrence can improve reasoning past the training horizon but degrades knowledge, and harder reasoning items do not reliably benefit more. Where recurrent iterations and distinct layers are placed also matters; non-recurrent output layers improve robustness to under-unrolling. Conventional initial-state injection is limited, while channel-wise history-state injection with timestep conditioning better preserves knowledge under extended unrolling.
- Recurrence can improve reasoning past the training horizon while hurting knowledge.
- Harder reasoning items do not consistently gain more from extra loops.
- Layer placement and non-recurrent outputs matter beyond raw effective depth.
- Channel-wise history-state injection plus timestep conditioning is more robust.
- AI models
- LoopLM
From the original article
1 min read at huggingface.co
Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness.
By Xinlin Zhuang, Siyuan Wang, Imran Razzak · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.