LooM Enables Loop-Invariant Memory for Causal Encoder-Decoder Models
Original title: LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
LooM loops only decoder layers after the first in causal encoder-decoder models to provide budgets from 3 to 21 executions per token without growing cache size.
Researchers propose LooM for looped transformers in causal encoder-decoder language models. It loops only decoder layers after the first, which read a global memory written once per token. Each layer publishes the KV entry for a token only on its first visit so the cache does not grow with the loop count. These layers loop independently providing budgets from 3 to 21 executions per token. One model trained on randomly sampled budgets covers all without a router. With enough loops LooM outperforms a single-pass baseline on language modeling, zero-shot accuracy and long-context perplexity while using the same cache size. Training budget distribution decides what extra depth buys.
- Loops only decoder layers after the first in CED
- Cache does not grow with loop count
- Budgets from 3 to 21 layer executions per token
- One model covers all budgets without router
- Outperforms single-pass baseline on multiple metrics
From the original article
1 min read at arxiv.orgLooped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn.
By Lele Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.