Infrathon tests language agents on long-horizon AI infrastructure tasks
Original title: Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Across 150 runs and over 10,000 GPU-hours, 135 still improved in the second half of a 24-hour budget.
Researchers introduce Infrathon, a benchmark of long-horizon AI infrastructure optimization tasks, and evaluate six models over 150 runs with 24-hour budgets. Rankings change with the evaluation horizon, and shorter tests can miss later gains. Claude Opus 5.5 raises the mean KV-cache compression ratio from 6.82% with the human-designed Quant-VideoGen codec to 19.86%. Trajectory analysis cites task violations, unreliable testing, and premature completion as barriers.
Unconfirmed: Performance figures and failure modes are reported by the paper's authors and are not independently verified here.
- Six models were evaluated across 150 runs.
- Each run had a 24-hour budget, totaling over 10,000 GPU-hours.
- 135 of 150 runs improved during the second half of the budget.
- Claude Opus 5.5 reached a 19.86% mean KV-cache compression ratio versus 6.82% for Quant-VideoGen.
- The authors identify task violations, unreliable testing, and premature completion as barriers.
- Products
- Quant-VideoGen
- AI models
- Claude Opus 5.5
From the original article
1 min read at huggingface.co
In this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours.
By Zhengyu Hu, Zheyuan Xiao, Yike Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.