Loopback Transformer beats baselines with fewer parameters
Original title: Loopback Transformer: Towards a Unified View of Residual Looped Transformers
On ClimbMix it uses 40-75% of baseline Transformer parameters and posts higher mean accuracy at 1.4B and 3B.
The Loopback Transformer treats residual looped models as depth memory: it stores individual block updates on a separate path from the shared stack input so input strength, total update weight, and allocation across updates can be controlled separately. Geometric decay sets total update weight and a recency prior, while attention allocates that weight. On ClimbMix, with training tokens and executed depth matched, it reaches lower validation loss than standard Transformers using 40-75% of their parameters, excluding embeddings and the output head, from 191M to 3B reference scales. It also leads seven other looped residual designs at 191M, 374M, and 778M.
- Parameter counts exclude token embeddings and the output head.
- Mean accuracy over thirteen downstream tasks is higher at 1.4B and 3B.
- At 191M, 374M, and 778M it has the lowest validation loss in the comparison.
- Those three scales also have the highest thirteen-task mean among eight looped designs.
- Attention learns allocation; geometric decay supplies the recency prior.
- AI models
- Loopback Transformer
From the original article
1 min read at huggingface.coLooped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it.
By Xuan Ouyang, Zefan Cai, Chenlong Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.