CoWindow Attention: Full Causal Coverage Is a Collective Property
CoWindow Attention covers causal history collectively, cutting 128K training latency up to 8.6x versus FullAttn.
CoWindow Attention (CoWA) distributes access to the causal history across KV heads so their union covers the full context while each head attends sparsely to distant tokens. The pattern needs no learned router and is used for both training and inference. At 8K, 100% collective coverage reaches 89.73% accuracy versus 89.97% for FullAttn. At 128K with tensor parallelism, training latency falls 7.4x forward and 8.6x backward, and decoding latency falls 3.0x, while 0.6B–14B scaling and continued 32B training track FullAttn perplexity and downstream scores.
- CoWA splits long-range causal history across KV heads
- No learned router; pattern is fixed for train and inference
- 8K ablation: 89.73% accuracy versus 89.97% for FullAttn
- At 128K, training latency drops 7.4x forward and 8.6x backward
- 14B and 32B models match FullAttn on knowledge and retrieval
From the original article
1 min read at huggingface.co
FullAttn repeatedly exposes the complete causal history to every attention head, creating substantial redundant computation and memory traffic even with IO-efficient dense kernels. We introduce CoWA, a structured attention architecture that distributes access to the causal history across KV heads.
By Jingze Shi, Zhangyang Peng, Xianduo Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.