FuTransformer speeds Llama-style inference with FP8 prefill and INT4 decode
Original title: FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
On one H100, prefill reaches 1.29 PFLOP/s and is up to 1.48x faster than compiled PyTorch FP8.
Researchers describe FuTransformer, a CUDA Llama-style Transformer block for NVIDIA Hopper GPUs. Prefill uses FP8 with delayed scaling and no padding; decode uses INT4 projection weights and an INT4 KV cache. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8. INT4 decode of Llama-3.1-8B takes 2.8 ms per step at batch one, versus 8.1 ms for compiled transformers.
- Prefill reaches 1.29 PFLOP/s on one H100.
- The FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path.
- INT4 decode is 1.3-2.4x faster than the faster PyTorch BF16 or INT4 block.
- WikiText-2 perplexity rises from 6.24 to 7.23 with round-to-nearest INT4 weights.
- Vendors
- NVIDIA
- Products
- H100
- AI models
- Llama-3.1-8B
From the original article
1 min read at arxiv.orgTransformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs.
By Wei Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.