When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Remote KV-cache hits slowed GLM-5.3 prefill until sharding and async loads cut p99 from 83s to 0.67s.
A study of remote KV-cache reuse for GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention, on 8xB300 nodes found LMCache plus Valkey left median time-to-first-token unchanged but raised 16k-token p99 from 3.96s to 17.3s. Remote hits took 0.88s versus 0.31s to recompute, and synchronous loads pushed a hit-heavy stream to p99 83.32s with GPUs 7% busy. Sharding, per-rank servers, asynchronous loading, and three client bug fixes reduced that p99 to 0.67s while serving 98.5% of prompt tokens remotely, lifting prefill throughput 14% and 34% with an upstream kernel fix. Changes were submitted to LMCache.
55