CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Researchers introduce CUA-Universe, a pipeline turning real desktop software into hybrid GUI+CLI agent environments, lifting a 9B model's OSWorld success rate.
CUA-Universe is an environment-to-data pipeline that converts real desktop applications into hybrid GUI+CLI environments, scaling to 16 applications via App-Forge, Task-Weave, and Path-Steer. Training on its harvested trajectories shifted a 9B model toward effective GUI+CLI orchestration, yielding +39.3 points on CUA-Verse, +16.8 points success rate on OSWorld, and +7.84 points on OSWorld-MCP while cutting steps and tokens by up to 57% and 60%. The work addresses the scarcity of scalable hybrid environments for computer-use agents.
- Pipeline scales to 16 real applications with discovered or generated CLI surfaces
- 9B model improves on CUA-Verse, OSWorld, and OSWorld-MCP benchmarks
- Path-Steer harvests verified hybrid trajectories for post-training
- Cuts steps up to 57% and tokens up to 60% versus baselines
From the original article
1 min read at arxiv.orgComputer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state.
By Haoting Shi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.