Evolving Infra adapts LLM serving to agent workloads at runtime
Original title: Evolving Infra: Agent-Adaptive LLM Serving at Scale
Authors report up to 4.1x lower mean latency and 14x lower P95 TTFT than NVIDIA Dynamo on GLM-5.2 and Kimi K2.6.
Agentic workloads stress prefill–decode serving because coarse signals miss changing agent behavior, causing load imbalance and eviction of reusable KV caches. Evolving Infra decouples compute allocation from KV cache management so nodes can switch between prefill and decode. An online planner matches the split to workload mix, and an agent-aware manager uses context signals for eviction and transfer. Authors test coding, math, and computer-use tasks on up to 256 GB300 GPUs.
Unconfirmed: The latency and TTFT gains are reported by the paper's authors and are not independently confirmed.
- Existing PD systems use coarse signals that only partly capture agent workloads.
- Nodes switch between prefill and decode after compute and KV cache are decoupled.
- Tasks cover coding, math, and computer use with GLM-5.2 and Kimi K2.6.
- The reported baseline is Dynamo, called NVIDIA's production framework.
- Scale in the evaluation reaches up to 256 GB300 GPUs.
- Vendors
- NVIDIA
- Products
- Evolving InfraDynamo
From the original article
1 min read at huggingface.coAgentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload.
By Hao Kang, Weili Xu, Ziyang Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.