Recurrent Looped Transformer
Recurrent Looped Transformers track algorithmic state far beyond training length via per-token feedback.
The Recurrent Looped Transformer splits eight layers between a parallel causal encoder and a recurrent decoder so computation grows with sequence length at a fixed per-token cost. Trained on at most 40 bits, two splits reached 100% parity accuracy at 256 bits in every seed, while an eight-layer Transformer stayed at chance. On swap-based S5 tracking at eight times the training length, RLT reached 97% final-state accuracy versus under 1%, and on longer modular arithmetic up to 93% versus 33%. Ablations show the gains require feedback; four-token chunking kept 64-bit parity at 99% but cut length-64 S5 accuracy from 100% to 20%.
- Layers split between a parallel encoder and a recurrent decoder.
- Two splits reach 100% parity at 256 bits after 40-bit training.
- S5 tracking hits 97% versus under 1% for a Transformer.
- Removing feedback drops parity and S5 accuracy to chance.
- AI models
- Recurrent Looped Transformer
From the original article
1 min read at huggingface.co
State tracking requires an update at every input, but the depth a Transformer applies to each token is fixed regardless of sequence length. We introduce the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder and a recurrent decoder.
By Yifan Zhang, Jichen Feng, Shihan Qin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.