Hybrid Computer Use unifies interfaces for AI agents
Original title: Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents
HCU combines programmatic, semantic (AX), and visual (GUI) interaction channels to enable flexible channel selection and combination in computer-use agents.
Computer-use agents tackle tasks from everyday desktop activities to long-horizon professional workflows. Interfaces centered on screenshots and mouse-and-keyboard actions impose a human-interaction prior while leaving semantic and programmatic capabilities underexposed. HCU exposes these capabilities and lets models choose and combine them. It supports efficient AX interaction with compact observations and batch actions reported as AX diffs. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among compared systems. All evaluated models improve with HCU in efficiency and task success.
- Highest Partial and Binary scores on OSWorld 2.0
- AX tools return compact observations and AX diffs
- Reduces round trips during channel switching
- All evaluated models improve with HCU
- Supports flexible channel selection across three modalities
From the original article
1 min read at huggingface.co
Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed.
By Haoqing Wang, Xiaoyi Zhang, Sicheng Fan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.