Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Study shows LLM test-time scaling cost depends on generation schedule: eight serial A100 calls use ~4.7x more GPU energy than one batched call.
The paper shows that candidate count N alone poorly describes the systems cost of sampling-based test-time scaling. Fixing N=8 with Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts, going from 1 to 8 candidates improved accuracy by 8.4 and 18.4 percentage points respectively, but eight serial generation calls on A100 GPUs consumed 4.64-4.86x more gross GPU-device energy and had 5.77-6.12x higher P95 latency than one batched 8-candidate call. The authors argue evaluations should report generation schedules and GPU-level metrics, and recommend fewer, larger batches when candidates are independent and memory allows.
- N=8 serial calls cost 4.64-4.86x more GPU energy than one batched call
- P95 latency 5.77-6.12x higher for serial schedules on A100
- Accuracy gains from 1 to 8 candidates: +8.4 pts (Phi-3-mini), +18.4 pts (Qwen2.5-1.5B)
- Recommends reporting generation schedule and GPU metrics in evaluations
- Products
- A100
- AI models
- Phi-3-miniQwen2.5-1.5B
From the original article
1 min read at huggingface.co
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed.
By Mobina Kashaniyan, Ali Jannesari · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.