Gated DeltaNet (AI model): news timeline & CVEs

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

NoteBank lifts pretrained linear-attention models to million-token retrieval by replaying sparse episodic memory.

The paper introduces NoteBank, a training-free sparse episodic memory that stores selected text and replays it through a linear-attention model's native forward pass. At one million tokens, Gated DeltaNet's mean accuracy on six RULER retrieval tasks rises from 1.7% to 70.3%, with no significant drop from 4K to 1M. The method adds at most 0.025% analytical FLOPs, and persistent storage is about 1/2,600 of a matched Transformer MHA KV cache. Experiments also report gains for Mamba-2-8B and post-trained Qwen3.5-9B on LongBench and MemoryAgentBench.

Hugging Face daily papersOct 10, 20:00 UTC · 10d agoAI research3· 2 reads

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Hybrid Linear Attention adds query-dependent chunk routing to Gated DeltaNet, lifting long-context scores on Qwen3.5.

Hybrid Linear Attention (HLA) adds query-dependent chunk mixing to Gated DeltaNet by treating each completed chunk as an exact affine state transition and interpolating it with the identity using content-dependent gates. Across adapted Qwen3.5 models from 0.8B to 9B, HLA beats native GDN and fixed chunk mixing by up to 5.57 points on LongBench-V2 and 3.97 on RULER. A from-scratch 1.3B model trained for 100B tokens at 4K context also improves RULER from 4K to 32K, with the gain rising from 0.83 to 4.22 points.

Hugging Face daily papersOct 4, 20:00 UTC · 16d agoAI research1· 1 read

Gated DeltaNet

Timeline2 articles

Appears with

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing