Meta's Cap-and-Track steers LLM inference using capacity refusals
Original title: Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity
In a production A/B, a classifier ran on 43% fewer H100s than a queued fixed-rate shaper.
Meta describes Cap-and-Track, a dispatcher for asynchronous LLM inference across provider capacity and internal GPUs, used for post and live-video understanding. Cap refuses work at a saturated resource before a long wait; Track raises or lowers dispatch using only those refusals, without predicting output length. Durable priority queues hold excess and refused work for retry. The authors report lower GPU use in a same-traffic A/B, with longer waits for lower-priority work at backfill peaks.
Unconfirmed: GPU and goodput figures are Meta's production A/B and replay results, not an independent benchmark.
- The shaper carries more than 300 million online requests and hundreds of terabytes of media daily.
- A backfill peak handled 1.1 billion requests in 24 hours.
- Steady replay stayed within 1% of goodput of a rate tuned above measured capacity.
- That replay had about a quarter of the tuned rate's refusal share.
From the original article
1 min read at arxiv.orgA fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs.
By Benjamin Novotny · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.