Fast Relative-Position Attention Algorithm Speeds Up Transformers
Original title: Fast and Exact Relative-Position Attention via Structure-Aware Kernels
Exact tiled computation incorporates additive position terms into FlashAttention without redundant indexing or memory overhead.
The paper addresses bottlenecks in relative-position attention mechanisms for transformers by proposing an exact algorithm that adds content-position or bias-position terms to attention scores. It uses offset-aware tiling to reuse key/query-relation products across token pairs in forward and backward passes and applies offset-based gradient sharding to eliminate contention. The unified Triton implementation outperforms naive and specialized baselines, delivering 1.8-6.4x faster forward and 2.0-21x faster backward passes for 8K-token sequences while cutting peak memory by up to 128x and enabling 1.8x faster end-to-end transcription with 2.4x less memory.
- Uses offset-aware tiling and product reuse across passes
- Eliminates atomic contention via gradient sharding
- Unified Triton code supports multiple relative-position mechanisms
- Enables long-context transcription 1.8x faster
- Reduces memory up to 128x over baselines
From the original article
1 min read at huggingface.co
Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms.
By Delyan Boychev, Ivo Petrov, Dimitar I. Dimitrov · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.