Muir Linear Attention Outperforms Transformers on Long-Context Tasks
First merged summary.
Muir linear attention model outperforms transformers and gated delta models on long-context tasks while training 2.7x faster.
Muir is a linear attention model that uses sensitivity-allocated gating and matched-filter readout to address erasure-write sensitivity and readout interference. It lowers perplexity by 1.8-2.0% over GDN-2, improves commonsense accuracy by up to 1.9 points, and reaches 100B-token validation loss 1.2x faster. The model extrapolates to 10M tokens with near-100% passkey retrieval and supports model sizes from 166M to 1.34B parameters.
- Lowers perplexity by 1.8-2.0% over GDN-2
- Trains 2.7x faster than transformers at 32K context
- Reaches 100B-token validation loss 1.2x faster
- Maintains near-100% passkey retrieval to 1M tokens
- Improves commonsense accuracy by up to 1.9 points
- Extrapolates to 10M tokens
What each outlet adds
beyond the shared facts- Hugging Face daily papersModel parameter range and improved extractive recall over GDN-2
- arXiv cs.AI / cs.LG / cs.CLExplanation of linear attention limitations and specific accuracy gains
- AI models
- Muir
Coverage
oldest first · one row per article · UTC- · 6d ago
Hugging Face daily papersFirst report
Muir Model Improves Linear Attention for Large Contexts
Muir outperforms transformers and gated delta models on long-context tasks while training faster
- · 6d ago
arXiv cs.AI / cs.LG / cs.CL
Muir improves linear attention with gating and readout
Muir reduces perplexity by 1.8-2.0% over GDN-2 and supports 10M-token extrapolation.