ATEST adapts LLM test-time sampling to live GPU energy
Original title: ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference
On four A100 datasets, ATEST-Online stayed inside the energy budget and tied or led accuracy as load rose.
Fixed multi-sample test-time scaling ignores query difficulty and energy spikes under GPU contention. The authors measured up to a 4.1-fold energy increase for the same four-sample query as load rose. ATEST-Online samples incrementally, watches plurality stability and live GPU conditions, and learns step energy online; ATEST-Calibrated instead picks a budget from offline models. On GSM8K at high load, Online reached 60.0 percent accuracy with no deferrals, versus 44.0 percent and 42.4 percent deferrals for Calibrated.
- Default mode needs no offline calibration.
- Transfer tests used an NVIDIA L40S and Qwen2.5-3B.
- Including profiling, Online used 42 times less GPU time than Calibrated.
- Online control raises tail latency and lowers short-task throughput.
- Vendors
- NVIDIA
- AI models
- Phi-3-miniQwen2.5-3B
From the original article
1 min read at huggingface.co
Test-time scaling (TTS) improves large language model (LLM) reasoning by sampling multiple answers, but fixed sampling budgets ignore both query difficulty and the changing energy cost of inference under GPU contention. We measure up to a 4. 1-fold increase in energy for the same four-sample query as GPU load rises.
By Mobina Kashaniyan, Ali Jannesari · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.