KronLSTM adds a Kronecker Delta Rule for long-context attention
Original title: Kronecker Delta Rule for Long Context Linear Attention
The open-source architecture revises associative memory along one matrix direction and often beats DeltaNet-family models on long context.
KronLSTM extends the xLSTM family with a Kronecker Delta Rule update for associative matrix memory. The revision corrects targeted associations while changing memory along a single direction, which the authors say reduces interference versus delta-rule updates. They describe the memory dynamics as a Markov chain with a bounded expected matrix norm, avoiding extra normalization. Experiments show it is competitive with, and often better than, existing linear attention models, especially on long-context benchmarks. Code and models are open-source.
- Memory revision is limited to one direction in matrix space.
- Expected memory-matrix norm stays uniformly bounded over the full sequence.
- No additional memory normalization or stabilization is required, the paper says.
- Code is on GitHub and models are published in a Hugging Face collection.
From the original article
1 min read at huggingface.co
We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space.
By Anamaria-Roberta Hartl, Levente Zólyomi, Michael List · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.