Skip to content
NewA calmer, faster ZeroHour

TideCache raises hybrid LLM serving throughput on CPU-GPU superchips

Original title: TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

arXiv cs.AI / cs.LG / cs.CLby Jinbin Luo ()1 min read2 · 1 read
AI summary · x-ai/grok-4.7Summaries by Vani AI

In SGLang on GB300 and GH200, it finishes up to 2.04x as many agent turns versus SGLang with HiCache.

TideCache manages heterogeneous caches for hybrid open-weight language models on CPU-GPU superchips. A transparent page pool, layer-wise cache streamer, and request scheduler keep generated KV resident while streaming global-attention prefixes. Across seven model and platform configurations it cuts overall P50 per-token latency by 20-59% and improves agent P50 TTFT by 1.40-3.05x versus SGLang with HiCache.

Key facts
  • The streamer prefetches the next global-attention prefix into shared GPU slots.
  • HBM released by streaming is reused for other caches and additional requests.
  • The scheduler picks resident or streamed execution from request size and queue state.
  • The system is implemented in SGLang and evaluated on GB300 and GH200.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory.

arXiv cs.AI / cs.LG / cs.CL

By Jinbin Luo · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.