Attention Quantization for Tabular Foundation Models
FP8 quantization of attention queries, keys, and values speeds tabular foundation model inference up to 1.7x with no accuracy loss.
The paper develops an FP8 quantization strategy targeting attention calculations (queries, keys, values) in tabular foundation models, arguing attention matters more than weight or KV cache quantization given their differing size and serving patterns versus LLMs. Aligning quantization error between test rows and training rows proves crucial, since misalignment causes drastic accuracy drops. A Triton kernel using explicit FP8 matrix multiplication achieves up to 1.7x speedup over regular 16-bit kernels, with no relevant accuracy loss on TabPFN-v3 and TabICLv2 across TabArena and BeyondArena benchmarks.
- FP8 attention quantization matters more than weight/KV-cache quantization for tabular models
- Aligning train-test quantization error prevents drastic accuracy drops
- Triton kernel with explicit FP8 instructions achieves up to 1.7x speedup
- No relevant accuracy loss on TabPFN-v3 and TabICLv2 across benchmarks
From the original article
1 min read at arxiv.orgWith the recent rise and adoption of tabular foundation models, optimizing their inference performance becomes an emerging field for efficiency research. While the models are architecturally similar to transformer-based large language models (LLMs), the size and serving patterns differ significantly.
By Jonas M. Kübler · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.