Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Triadic linear attention extends linear attention to third-order tensor states, improving long-context modeling and recall over state-enlargement alternatives.
The paper proposes triadic linear attention, which writes the triadic outer product of a key, a second key, and a value into a third-order tensor state, read by contracting both key axes with two queries. An E-dimensional second key yields an E-fold state size increase while adding only two projections. The method is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, it substantially improves long-context language modeling and recall, outperforming other state-enlargement approaches.
- Third-order tensor state written via triadic outer product of two keys and value
- E-dimensional second key gives E-fold state increase with only two projections
- Compatible with delta rule, data-dependent forgetting, chunkwise-parallel training
- Improves long-context modeling and recall on Gated DeltaNet and scalar-gated variants
From the original article
1 min read at huggingface.co
Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states.
By Oliver Sieberling, Bharat Runwal, David Jin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.