KronLSTM applies a Kronecker Delta Rule to long-context linear attention
Original title: Kronecker Delta Rule for Long Context Linear Attention
Authors report stable bounded memory without extra normalization, and open-source code and models are released.
KronLSTM extends the xLSTM family with a Kronecker Delta Rule update that revises targeted associations along a single direction in matrix space. The authors describe the memory dynamics as a Markov chain that converges to a steady state, with the expected memory-matrix norm remaining uniformly bounded, so extra normalization is not required. Experiments show it is competitive with and often outperforms existing linear attention models, including the (Gated) DeltaNet family, especially on long-context benchmarks. Code and models are released as open source.
- The update is meant to reduce interference versus ordinary delta-rule memory revisions.
- Strongest reported gains are on long-context benchmarks.
- Code is on GitHub under NX-AI and models are on Hugging Face.
From the original article
1 min read at arxiv.orgWe propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space.
By Anamaria-Roberta Hartl · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.