Muir Model Improves Linear Attention for Large Contexts
Original title: Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Muir outperforms transformers and gated delta models on long-context tasks while training faster
The paper introduces Muir, a linear attention model that uses sensitivity-allocated gating and matched-filter readout. It lowers perplexity by 1.8-2.0% over GDN-2 and improves commonsense accuracy. It extrapolates to 10M tokens and trains 2.7x faster than transformers at 32K.
- 166M-1.34B parameters
- Reaches 100B-token validation loss 1.2x faster
- Improves extractive recall over GDN-2
- Maintains near-100% passkey retrieval to 1M tokens
- AI models
- Muir
From the original article
1 min read at huggingface.co
Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates.
By Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.