Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom introduces per-query bit-depth scanning of offloaded 4-bit KV caches, making long-context decoding 1.67x faster on Qwen3-8B at one million tokens.
Fathom is a key-scan method for sparse attention over KV caches resident in host memory, where each query adaptively chooses how many bits of each key channel to read using reverse water-filling over variance-weighted channel importance. The 4-bit K cache is stored channel-major as bit planes, so reading a prefix of planes is an exact t-bit quantizer. At one million tokens on Qwen3-8B, decode steps are 1.67x faster in GPU time than 136-bit scans used by Double Sparsity, Loki, and SparQ r=32, and 92-bit scans match the accuracy of 136-bit scans on real coding-agent sessions. The method requires the index to be offloaded to host memory and matches exact top-k decoding on RULER-style tasks.
- Per-query bit budgets via reverse water-filling over channel importance
- 4-bit K cache stored channel-major as bit planes enables exact t-bit prefix quantizers
- 1.67x faster decoding than 136-bit scans at 1M tokens on Qwen3-8B
- 92-bit scans match most accurate 136-bit scan on real coding-agent sessions
- Requires index in host memory; no speedup when index fits in GPU memory
- Tags
- decodinghost-memory-offloadinference-optimizationkv-cachelong-contextquantizationsparse-attention
- Vendors
- Hugging Face
- Products
- Fathom
- AI models
- Qwen3-8B
From the original article
1 min read at huggingface.co
When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read.
By Vivek Kalyanarangan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.