TideCache raises hybrid LLM serving throughput on CPU-GPU superchips
Original title: TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming
In SGLang on GB300 and GH200, it finishes up to 2.04x as many agent turns versus SGLang with HiCache.
TideCache manages heterogeneous caches for hybrid open-weight language models on CPU-GPU superchips. A transparent page pool, layer-wise cache streamer, and request scheduler keep generated KV resident while streaming global-attention prefixes. Across seven model and platform configurations it cuts overall P50 per-token latency by 20-59% and improves agent P50 TTFT by 1.40-3.05x versus SGLang with HiCache.
- The streamer prefetches the next global-attention prefix into shared GPU slots.
- HBM released by streaming is reused for other caches and additional requests.
- The scheduler picks resident or streamed execution from request size and queue state.
- The system is implemented in SGLang and evaluated on GB300 and GH200.
From the original article
1 min read at arxiv.orgHybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory.
By Jinbin Luo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.