CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering
CUA-SWE tests agents that repair software using code changes and visual GUI feedback.
CUA-SWE is a benchmark, environment, and evaluation pipeline for software engineering that requires computer use as well as code changes. Agents must modify code and configuration, run commands, interact with live software, and inspect visual feedback, including cases where specifications exist only in the GUI. The suite spans four domains and scores each task with deterministic tests for required behavior. The reported evaluation examines how frontier agents connect screenshots and graphical interaction with source-level execution to produce verified repairs.
- CUA-SWE combines coding, command execution, GUI interaction, and visual inspection.
- Some required specifications are available only through the running application's interface.
- Tasks cover four software-engineering domains with deterministic correctness tests.
- Evaluation studies how frontier agents link screenshots to code changes and repairs.
From the original article
1 min read at huggingface.co
Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored.
By Prince Zizhuang Wang, Chenhao Liang, Zelong Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.