FLAMIA kernels accelerate attention on NVIDIA Blackwell GPUs
Original title: FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
FLAMIA achieves 1.10-1.40x speedup for long-context attention using MXFP8 on B200 GPUs with token-block scaling.
Attention computation is a key cost in Transformer training at long contexts. FLAMIA is a family of kernels for NVIDIA Blackwell GPUs that reuses quantized operands and adds safeguards against scale leakage. It fuses quantization with architecture-specific optimizations and supports variable-length documents in packed layouts, preserving training stability on Llama-style models.
- MXFP8 path accelerates forward and backward attention by 1.10-1.40x on B200
- Uses token-block scaling to reuse quantized operands across orientations
- Validates stability on pretraining Llama-style models
- Supports document-local scaling for packed layouts
From the original article
1 min read at huggingface.co
Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes.
By Fedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.