Muir improves linear attention with gating and readout
Original title: Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Muir reduces perplexity by 1.8-2.0% over GDN-2 and supports 10M-token extrapolation.
Linear attention suffers from erasure-write sensitivity and readout interference. Muir uses sensitivity-allocated low-rank projections and matched-filter readout gates. It outperforms baselines in perplexity, accuracy, and long-context recall while matching throughput.
- Lowers perplexity by 1.8-2.0% over GDN-2
- Improves commonsense accuracy by up to 1.9 points
- 100B-token validation loss reached 1.2x faster
- Maintains near-100% passkey retrieval to 1M tokens
From the original article
1 min read at arxiv.orgLinear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates.
By Tushaar Gangavarapu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.