Block Sparse Attention with Log-Linear Complexity
PISA selects sparse attention blocks with pyramid Top-K routing, reducing complexity to O(N log N).
PISA is a block-sparse attention method that avoids scoring every query-block pair. It builds O(log N) pooled key levels and uses LogSumExp scoring on bounded candidate sets to route from coarse to fine, giving O(N log N) complexity. Hardware-aware Triton kernels fuse hierarchical routing for training and inference without materializing the query-key score matrix. On language modeling, it matches a baseline on commonsense reasoning and performs better on retrieval tasks.
- Pyramid Top-K selection reduces block-sparse attention to O(N log N).
- LogSumExp scoring is applied only to bounded candidate sets.
- Triton kernels fuse routing without materializing the score matrix.
- Retrieval improves while commonsense reasoning stays comparable to baseline.
From the original article
1 min read at huggingface.co
Scaling language models to long contexts is limited by the quadratic cost of self-attention. Block sparse attention offers an efficient alternative, but selecting the retained blocks remains a bottleneck. Conventional block selection requires scoring all query-block pairs and therefore remains quadratic in sequence length.
By Bohao Tang, Zhen Qin, Yuqi Pan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.