Systolic Linear Attention optimizes gated delta attention for Google TPUs
Original title: Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs
The method achieves 1.6x to 2.5x faster performance on TPU v7x and 11-24% faster training for MoE models compared to KDA.
Gated-delta linear attention layers such as GDN, KDA, and GDN-2 are tuned for GPUs but not optimal for TPUs with wide systolic matrix units. The authors present Systolic Linear Attention (SLA) which reshapes gated delta attention around the matrix unit by widening keys to 256 with grouped value heads. On TPU v7x, a full SLA-R4 layer runs 1.6x--2.5x faster than the open-source Tokamax KDA kernel. In a 2,048-chip TPU v7x pre-training study of sparse Mixture-of-Experts models from 6.5B to 63.1B parameters, SLA trains 11%--24% faster than KDA at similar perplexity.
- Reshapes gated delta attention for 256x256 systolic units
- Widens keys to 256 with grouped value heads
- Groups decay gates into four 64-channel tiles
- Computes chunk solve with block matrix products
- 11%--24% faster training for MoE models
From the original article
1 min read at arxiv.orgGated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make per-channel decay rescaling and sequential in-block solves cheap.
By Gagik Amirkhanyan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.