History Replays Itself: Unleashing Long-Context Capability in Linear Attention
NoteBank lifts pretrained linear-attention models to million-token retrieval by replaying sparse episodic memory.
The paper introduces NoteBank, a training-free sparse episodic memory that stores selected text and replays it through a linear-attention model's native forward pass. At one million tokens, Gated DeltaNet's mean accuracy on six RULER retrieval tasks rises from 1.7% to 70.3%, with no significant drop from 4K to 1M. The method adds at most 0.025% analytical FLOPs, and persistent storage is about 1/2,600 of a matched Transformer MHA KV cache. Experiments also report gains for Mamba-2-8B and post-trained Qwen3.5-9B on LongBench and MemoryAgentBench.
56