OSWorld-Pro: Process-based Evaluation for Computer Use Agents
OSWorld-Pro adds process-level subgoals so computer-use agent failures can be diagnosed beyond final outcomes.
OSWorld-Pro is a process-based benchmark for computer-use agents, with more than 300 tasks and 2,800 subgoals grounded in over 67,000 human annotations. Human-aligned LLM judges score subgoal completion so progress can be tracked across sequentially dependent steps, rather than only final deliverables. Claude Opus 5 reaches 75.7 percent on OSWorld-Pro versus 83.4 percent on OSWorld. The authors highlight failure modes such as subgoal-irrelevant actions and click-based GUI mistakes.
- More than 300 tasks and 2,800 subgoals with 67,000 human annotations.
- LLM judges measure progress across sequentially dependent subgoals.
- Claude Opus 5 scores 75.7 percent, below 83.4 percent on OSWorld.
- Noted failures include irrelevant actions and imprecise click inputs.
- Vendors
- Anthropic
- Products
- OSWorldOSWorld-Pro
- AI models
- Claude Opus 5
From the original article
1 min read at arxiv.orgEvaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement.
By Zhilin Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.