DAMI improves long-context retrieval in gated linear attention models
Original title: DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
DAMI jointly configures memory parameters for better retrieval accuracy across multiple architectures.
Linear attention scales well but struggles with long-context retrieval due to finite-state memory. In gated linear recurrences, memory retention depends on state decay shaped by gate parameters and input activations. DAMI assigns geometric timescales across heads and a low-high-low input-sensitivity profile across layers. A small activation sample calibrates the system. It improves average RULER-style retrieval accuracy by 5.29 to 16.71 percentage points on GDN, KDA, GDN2, Mamba-2 and RWKV-7.
- DAMI uses geometric timescales across heads
- Low-high-low input-sensitivity profile across layers
- Calibrates with small activation sample
- Improves retrieval by 5.29-16.71 points
From the original article
1 min read at huggingface.co
Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations.
By Zhangchi Feng, Harry Jiang, Yangyifei Luo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.