Skip to content
NewA calmer, faster ZeroHour

Meta's Cap-and-Track steers LLM inference using capacity refusals

Original title: Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity

arXiv cs.AI / cs.LG / cs.CLby Benjamin Novotny ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

In a production A/B, a classifier ran on 43% fewer H100s than a queued fixed-rate shaper.

Meta describes Cap-and-Track, a dispatcher for asynchronous LLM inference across provider capacity and internal GPUs, used for post and live-video understanding. Cap refuses work at a saturated resource before a long wait; Track raises or lowers dispatch using only those refusals, without predicting output length. Durable priority queues hold excess and refused work for retry. The authors report lower GPU use in a same-traffic A/B, with longer waits for lower-priority work at backfill peaks.

Unconfirmed: GPU and goodput figures are Meta's production A/B and replay results, not an independent benchmark.

Key facts
  • The shaper carries more than 300 million online requests and hundreds of terabytes of media daily.
  • A backfill peak handled 1.1 billion requests in 24 hours.
  • Steady replay stayed within 1% of goodput of a rate tuned above measured capacity.
  • That replay had about a quarter of the tuned rate's refusal share.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs.

arXiv cs.AI / cs.LG / cs.CL

By Benjamin Novotny · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.