Multi-Stream Attention Residuals
MSAR adds multi-head depth attention to residuals, with a 1.58× iso-loss speedup over standard residuals.
Multi-Stream Attention Residuals (MSAR) extends Attention Residuals from single-stream depth attention to multi-head depth attention. Stream diversity comes from multi-scale causal convolutions, and retrieved streams are mixed adaptively into a single-stream layer input without persistent multi-stream storage. Across 12 downstream benchmarks at 18B and 28B mixture-of-experts scales, MSAR outperformed standard residuals, manifold-constrained Hyper-Connections, and AttnRes. Scaling-law experiments report a 1.58× iso-loss compute speedup over the standard residual baseline, versus 1.35× for AttnRes and 1.30× for mHC.
- MSAR generalizes Attention Residuals to multi-head depth attention.
- Streams use multi-scale causal convolutions and lightweight adaptive mixing.
- It avoids persistent multi-stream storage or communication.
- MSAR beat residual, mHC, and AttnRes on 12 benchmarks at 18B and 28B MoE.
- Iso-loss compute speedup is 1.58× versus 1.35× for AttnRes and 1.30× for mHC.
From the original article
1 min read at huggingface.co
Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis.
By Xiaohan Qin, Xiangdong Zhang, Tuo Dai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.