Muir Model Improves Linear Attention for Large Contexts
Muir outperforms transformers and gated delta models on long-context tasks while training faster
Stories per day · 30 days
Relatedlong-context4quantization3inference2open-source2xlstm2associative-memory1attention1autonomous-research1delta-rule-attention1gated-deltanet1gdn1gpu-optimization1gradient-computation1kronlstm1language-modeling1mamba1memory1merlin1
25 stories, newest first · one card per story
Stories tagged #linear-attention
Systolic Linear Attention optimizes gated delta attention for Google TPUs
The method achieves 1.6x to 2.5x faster performance on TPU v7x and 11-24% faster training for MoE models compared to KDA.
ReWY speeds up delta-rule attention gradients
ReWY reformulates backward passes for delta-rule linear attention reducing seven local WY matrix products to three and achieving 1.15-1.81x speedup on GPUs.
Gradient-aware allocation and sliding windows for low-bit GDN states
On Qwen3.5-4B and 9B, GGA plus PW-S8 beats STEPQuant@4 by 38.39 and 41.21 points on seven-task averages.
MERLIN uses key moments to route multi-state linear attention
Reported long-context QA gains are +8.3 and +7.0 points on Gated DeltaNet-1.3B and Mamba-2-780M, with larger LongBench lifts.
PACE quantizes recurrent language-model states to four bits
On hybrid models from 4B to 35B parameters, mixed four- and eight-bit heads approach eight-bit DAMP at about two-thirds of its memory.
PACE quantizes linear-attention recurrent states to four bits post hoc
Mixed four- and eight-bit PACE matches eight-bit DAMP on the three larger hybrid models at about two thirds of its memory.
Multi-agent AI team finds linear attention designs that beat GDN2
Agents with distinct disciplines, private histories, and adaptive allocation report gains over GDN2 at 1.3B scale.
KronLSTM applies a Kronecker Delta Rule to long-context linear attention
Authors report stable bounded memory without extra normalization, and open-source code and models are released.
KronLSTM adds a Kronecker Delta Rule for long-context attention
The open-source architecture revises associative memory along one matrix direction and often beats DeltaNet-family models on long context.
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
NoteBank lifts pretrained linear-attention models to million-token retrieval by replaying sparse episodic memory.
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
HLA-WM improves long-horizon video world-model consistency without training while cutting memory use.
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Hybrid Linear Attention adds query-dependent chunk routing to Gated DeltaNet, lifting long-context scores on Qwen3.5.
LOCI: Spatial Linear Memory for Streaming World Models
LOCI uses geometry-conditioned spatial memory so video world models recall revisited scenes with less peak memory.
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
STEPQuant matches FP32 recurrent-state accuracy at about 6 bits while cutting serving memory.
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
STEPQuant quantizes delta-rule recurrent states by error lifetime and location, matching FP32 accuracy at 6 bits.
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Triadic linear attention extends linear attention to third-order tensor states, improving long-context modeling and recall over state-enlargement alternatives.
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Copying MLP weights and distilling attention lets linear Vision Transformers match or beat pretrained Softmax ViTs.
GTR: Gated Token Recurrence for Efficient Dense Prediction
GTR introduces a softmax-free recurrent vision backbone reaching 58.9 box AP on COCO with sub-2ms latency, targeting high-resolution dense prediction and edge deployment.
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Video DeltaNet combines softmax and linear attention for video generation, cutting DiT denoising of a 14.3-second 768p video to 6.7 seconds.
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Video DeltaNet combines local Softmax attention with linear memory, enabling 14.5x faster 768p video generation on eight NVIDIA B200 GPUs.
SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
SpectralShift extends Gated DeltaNet context windows via spectral reparameterization of alpha projections, improving long-context capabilities without architectural changes.
Kalman Delta Networks: Uncertainty-aware Associative Memory
Kalman Delta Networks add uncertainty tracking to linear-attention associative memory, improving perplexity and downstream accuracy at 750M and 1.3B scales.
Kalman Delta Networks: Uncertainty-aware Associative Memory
Researchers propose Kalman Delta Networks, adding Kalman-filter uncertainty tracking to delta-rule linear attention, improving perplexity and accuracy at 750M and 1.3B parameters.