Skip to content
NewA calmer, faster ZeroHour

Evolving Infra adapts LLM serving to agent workloads at runtime

Original title: Evolving Infra: Agent-Adaptive LLM Serving at Scale

Hugging Face daily papersby Hao Kang, Weili Xu, Ziyang Li ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

Authors report up to 4.1x lower mean latency and 14x lower P95 TTFT than NVIDIA Dynamo on GLM-5.2 and Kimi K2.6.

Agentic workloads stress prefill–decode serving because coarse signals miss changing agent behavior, causing load imbalance and eviction of reusable KV caches. Evolving Infra decouples compute allocation from KV cache management so nodes can switch between prefill and decode. An online planner matches the split to workload mix, and an agent-aware manager uses context signals for eviction and transfer. Authors test coding, math, and computer-use tasks on up to 256 GB300 GPUs.

Unconfirmed: The latency and TTFT gains are reported by the paper's authors and are not independently confirmed.

Key facts
  • Existing PD systems use coarse signals that only partly capture agent workloads.
  • Nodes switch between prefill and decode after compute and KV cache are decoupled.
  • Tasks cover coding, math, and computer use with GLM-5.2 and Kimi K2.6.
  • The reported baseline is Dynamo, called NVIDIA's production framework.
  • Scale in the evaluation reaches up to 256 GB300 GPUs.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload.

Hugging Face daily papers

By Hao Kang, Weili Xu, Ziyang Li · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.