TAO jointly schedules LLM agent sessions and tool returns
Original title: TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
Across 12 reported settings, TAO cuts latency by up to 26.8% and mean TTFT by up to 36.3% versus the best baseline for each metric.
TAO is a tool-aware joint scheduler for LLM agentic serving, where inference runs on clustered GPUs and external tools run on distributed CPUs. Its value interface combines near-term tool readiness, decaying node-specific cache credit, and token footprint. A dynamic program selects and places waiting sessions under capacity constraints. The authors also report up to 29.5% higher decode throughput versus the best evaluated baseline for that metric.
Unconfirmed: Latency and throughput gains are the authors' measurements against evaluated baselines.
- Existing systems overlook tool-return readiness and use heuristic GPU admission.
- Placement uses node-specific cache credit with time-aware decay.
- Gains are reported across 12 model-workload-concurrency settings.
From the original article
1 min read at huggingface.co
Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions.
By Shuhuai Li, Shengdong Ye, Dongdong Ge · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.