KVGuard detects KV cache safety bugs in LLM serving
Original title: KVGuard: A KV Cache Safety Checker for LLM Serving Systems
KVGuard covers all 29 historical bugs from vLLM and SGLang plus 8 new ones with 3% overhead.
KVGuard provides the first systematic memory-safety abstraction for KV cache management in LLM serving systems that use PagedAttention. It performs five targeted checks across CPU allocation, block tables, and GPU access to catch double-free, use-after-free, out-of-bounds, uninitialized read, and permission violations that silently corrupt outputs. A prototype on vLLM and SGLang reports no false positives and adds minimal overhead while detecting issues existing sanitizers miss.
- Covers all 29 historical KV cache safety bugs from vLLM and SGLang
- Finds 8 previously unknown safety violations
- 3.0% geomean TPOT overhead on Qwen 2.5 14B
- At most 0.11% KV cache memory overhead
- Two orders of magnitude below NVIDIA Compute Sanitizer
- AI models
- Qwen 2.5 14B
From the original article
1 min read at arxiv.orgModern LLM serving systems manage the KV cache through PagedAttention, a custom GPU allocator that partitions a single CUDA-allocated region into fixed-size physical blocks and maps each request to those blocks through a CPU-built block table.
By Jonguk Cheong · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.