GAAC losslessly compresses BF16 KV caches about 1.41×
Original title: GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
On NVIDIA DGX Spark, 64K-token decode attention sped up 1.19–1.27× and resident KV memory fell about 26%.
Group-Adaptive Additive Codec encodes BF16 exponents with calibrated per-channel codes while keeping KV values bit-for-bit. A fused attention kernel rebuilds tiles on the fly so the full cache is not decompressed. Across five tested models it reports 1.405–1.407× BF16 compression, smaller gains on FP8 formats, and speedups on NVIDIA DGX Spark and Apple M4.
- FP8 E4M3 caches compressed 1.09–1.14× and E5M2 caches 1.25–1.30×.
- End-to-end W4A16 decode steps sped up 1.04–1.15× on DGX Spark.
- Serving throughput rose 13–29% under comparable KV-memory budgets.
- Apple M4 attention speedups were 1.02–1.25×.
- Products
- NVIDIA DGX SparkApple M4
From the original article
1 min read at arxiv.orgWeight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices.
By Yipin Guo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.