MERLIN uses key moments to route multi-state linear attention
Original title: Multi-State Linear Attention with Generalized Second-Order Routing
Reported long-context QA gains are +8.3 and +7.0 points on Gated DeltaNet-1.3B and Mamba-2-780M, with larger LongBench lifts.
The paper proposes MERLIN, which stores the first and second moments of each segment's keys so a query can recover both the mean and variance of its alignment. It uses those statistics to aggregate per-state readouts in multi-state linear attention, rather than mean alignment alone. On multi-state Gated DeltaNet-1.3B and Mamba-2-780M, the authors report gains on RULER, long-context QA, and LongBench, including against baselines using 4x more memory.
Unconfirmed: The reported gains are the authors' experimental results.
- Prior MSLA aggregation used only mean alignment between a query and segment keys.
- MERLIN recovers mean and variance of alignment inside each segment.
- LongBench gains are +13.5% and +12.0% on the two model variants.
- It beats some baselines that consume four times more memory.
From the original article
1 min read at arxiv.orgSelf-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state.
By Hee Seung Hwang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.