When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Elo-per-token analysis shows LLM agents' marginal gains drop below independent sampling at scale; parallel sessions beat one long session.
The paper proposes Elo-per-token analysis, using a Bradley-Terry model to measure how agent performance scales with token budget on open-ended tasks with continuous scoring. Across four agents and four benchmarks with sessions up to 100M tokens, agents initially convert tokens to Elo faster than independent sampling but eventually slow below the linear-in-log-compute reference. The authors define a scaling inflection point and show that splitting 100M tokens across parallel sessions on FrontierCS Polyomino Packing gains +264 Elo over one long session and +355 over ten short sessions. Human contestants on shared AtCoder Heuristic Contest tasks improve superlinearly, indicating headroom over current agents.
- Agents' marginal Elo gains eventually fall below independent-sampling reference
- Scaling inflection point defined where marginal gains match independent sampling
- Splitting 100M tokens across sessions gains +264 Elo on Polyomino Packing
- Human contestants still improve superlinearly, showing agent headroom
From the original article
1 min read at huggingface.co
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectories.
By Kaiyuan Liu, Qiuyang Mang, Bo Peng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.