Extender adds a log channel to cut Transformer attention memory
Original title: The Extender: A Log-Structured Transformer
At 924M parameters and extension size 32, authors report matching short-context accuracy and beating Transformers on long-context RULER tasks.
The Extender is a Transformer variant that keeps the residual stream and adds a concatenation channel. Each layer appends a small extension while attention key-value projections read only that channel, so persistent attention memory falls from 2Ld_model to the sum of extension sizes. With extension size 32, the authors say it matches Transformer accuracy on short-context CORE tasks from 199M to 924M parameters and exceeds it on long-context RULER at 924M. For their 1664-wide, 924M model, they report a persistent attention memory footprint 104 times smaller than multi-head attention.
Unconfirmed: Accuracy and memory figures are the authors' reported results, not independently verified.
- Both the feed-forward network and queries still see the residual stream.
- Key-value projections take only the concatenation channel as input.
- Authors say memory savings grow with model width.
- The 924M model they report is 1664-wide.
From the original article
1 min read at huggingface.co
We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual h, a superposition channel. The Extender adds a concatenation channel x: each layer ell emits both a residual update δ_ell which is added to h, and a much smaller extension ε_ell which is appended to x.
By Jakob Eriksson · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.