Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Learned depth priors and orthogonal injection improve looped language models while enabling cheaper KV sharing.
Recurrence in looped language models is costly, but states near fixed points need less of the path that reached them. The authors learn a depth prior from prediction feedback and use orthogonal input injection so the state cannot amplify or cancel the injected input. From 100M to 1.6B parameters, both changes lower perplexity versus Huginn's prior and existing injection schemes. At 1.6B, the learned prior with a 3x smaller KV cache matches fixed-depth downstream averages, while related methods speed prefill up to 1.79x and RL updates 2x.
- Near-fixed-point states enable truncated backprop and terminal KV sharing.
- Learned depth prior plus orthogonal injection lower perplexity from 100M to 1.6B.
- At 1.6B, a 3x smaller KV cache matches full-cache fixed-depth downstream average.
- A distilled student prefills up to 1.79x faster; RL updates are 2x faster.
From the original article
1 min read at huggingface.co
Every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning (RL). The closer recurrent states get to fixed points, the less the path to them matters. This enables truncated backpropagation in training; terminal key-value (KV) sharing for decoding with almost no loss in accuracy; a distilled student that prefills up to 1.
By Benhao Huang, Chufan Shi, Junlin Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.