History Replays Itself: Unleashing Long-Context Capability in Linear Attention
NoteBank lifts pretrained linear-attention models to million-token retrieval by replaying sparse episodic memory.
The paper introduces NoteBank, a training-free sparse episodic memory that stores selected text and replays it through a linear-attention model's native forward pass. At one million tokens, Gated DeltaNet's mean accuracy on six RULER retrieval tasks rises from 1.7% to 70.3%, with no significant drop from 4K to 1M. The method adds at most 0.025% analytical FLOPs, and persistent storage is about 1/2,600 of a matched Transformer MHA KV cache. Experiments also report gains for Mamba-2-8B and post-trained Qwen3.5-9B on LongBench and MemoryAgentBench.
- NoteBank is training-free and needs no separate retriever or reader.
- Gated DeltaNet RULER accuracy rises from 1.7% to 70.3% at 1M tokens.
- Added cost is at most 0.025% analytical FLOPs versus full KV cache.
- Gains also cover Mamba-2-8B, Qwen3.5-9B, LongBench, and MemoryAgentBench.
Full article217 words · extracted from huggingface.co · click to collapse
Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds leqslant0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.15715