Skip to content
NewA calmer, faster ZeroHour

Loopback Transformer beats baselines with fewer parameters

Original title: Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Hugging Face daily papersby Xuan Ouyang, Zefan Cai, Chenlong Zhang ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On ClimbMix it uses 40-75% of baseline Transformer parameters and posts higher mean accuracy at 1.4B and 3B.

The Loopback Transformer treats residual looped models as depth memory: it stores individual block updates on a separate path from the shared stack input so input strength, total update weight, and allocation across updates can be controlled separately. Geometric decay sets total update weight and a recency prior, while attention allocates that weight. On ClimbMix, with training tokens and executed depth matched, it reaches lower validation loss than standard Transformers using 40-75% of their parameters, excluding embeddings and the output head, from 191M to 3B reference scales. It also leads seven other looped residual designs at 191M, 374M, and 778M.

Key facts
  • Parameter counts exclude token embeddings and the output head.
  • Mean accuracy over thirteen downstream tasks is higher at 1.4B and 3B.
  • At 191M, 374M, and 778M it has the lowest validation loss in the comparison.
  • Those three scales also have the highest thirteen-task mean among eight looped designs.
  • Attention learns allocation; geometric decay supplies the recency prior.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it.

Hugging Face daily papers

By Xuan Ouyang, Zefan Cai, Chenlong Zhang · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.