OSWorld-Pro: Process-based Evaluation for Computer Use Agents
OSWorld-Pro adds process-level subgoals so computer-use agent failures can be diagnosed beyond final outcomes.
OSWorld-Pro is a process-based benchmark for computer-use agents, with more than 300 tasks and 2,800 subgoals grounded in over 67,000 human annotations. Human-aligned LLM judges score subgoal completion so progress can be tracked across sequentially dependent steps, rather than only final deliverables. Claude Opus 5 reaches 75.7 percent on OSWorld-Pro versus 83.4 percent on OSWorld. The authors highlight failure modes such as subgoal-irrelevant actions and click-based GUI mistakes.
- More than 300 tasks and 2,800 subgoals with 67,000 human annotations.
- LLM judges measure progress across sequentially dependent subgoals.
- Claude Opus 5 scores 75.7 percent, below 83.4 percent on OSWorld.
- Noted failures include irrelevant actions and imprecise click inputs.
Full article184 words · extracted from arxiv.org · click to collapse
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24890