cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents
cua-speedrun standardizes virtual-machine infrastructure to measure computer-use agent speed, cost, and efficiency.
cua-speedrun is a standardized benchmark for the speed and cost of computer-use agents, which operate GUIs to complete computer tasks. It uses a uniform virtual machine, a shared execution pipeline, and a common agent interface so one implementation can run across benchmarks. Evaluations on four CUA benchmarks measure reasoning effort, agent harnesses, and environment latency against performance, speed, and cost. No model family is best on all three, open-weight models are off the frontier, and more reasoning or faster environment I/O can have counterintuitive effects on completion time. The authors also show most benchmark task sets can be reduced without losing statistical power.
- Existing CUA benchmarks confound speed because infrastructure and latency differ.
- cua-speedrun uses one VM setup, pipeline, and agent interface.
- No single model family is best on performance, speed, and cost together.
- No open-weight model sits on the reported frontier.
- Smaller task sets can preserve statistical power for faster comparisons.
From the original article
1 min read at arxiv.orgComputer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost.
By Pranjal Aggarwal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.