Attention Quantization for Tabular Foundation Models
FP8 quantization of attention queries, keys, and values speeds tabular foundation model inference up to 1.7x with no accuracy loss.
The paper develops an FP8 quantization strategy targeting attention calculations (queries, keys, values) in tabular foundation models, arguing attention matters more than weight or KV cache quantization given their differing size and serving patterns versus LLMs. Aligning quantization error between test rows and training rows proves crucial, since misalignment causes drastic accuracy drops. A Triton kernel using explicit FP8 matrix multiplication achieves up to 1.7x speedup over regular 16-bit kernels, with no relevant accuracy loss on TabPFN-v3 and TabICLv2 across TabArena and BeyondArena benchmarks.
- FP8 attention quantization matters more than weight/KV-cache quantization for tabular models
- Aligning train-test quantization error prevents drastic accuracy drops
- Triton kernel with explicit FP8 instructions achieves up to 1.7x speedup
- No relevant accuracy loss on TabPFN-v3 and TabICLv2 across benchmarks
Full article150 words · extracted from arxiv.org · click to collapse
With the recent rise and adoption of tabular foundation models, optimizing their inference performance becomes an emerging field for efficiency research. While the models are architecturally similar to transformer-based large language models (LLMs), the size and serving patterns differ significantly. We show that the focus should be on the attention calculation and less on weight or KV cache quantization, which are more popular in LLMs. We develop a quantization strategy for queries, keys, and values to FP8 and use explicit FP8 matrix multiplication instructions to speed up the attention calculation. We find that it is crucial to align the quantization error in the test rows with the quantization error in the training rows, as otherwise the accuracy drops drastically. Our Triton kernel achieves a speedup up to 1.7x over regular 16-bit kernels, and we show that on TabPFN-v3 and TabICLv2 there is no relevant accuracy loss across TabArena and BeyondArena.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.13031