JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
JarvisGUI benchmark tests GUI agents on cross-device workflows across Android, Windows, and Ubuntu, revealing major gaps in state transfer and long-horizon reasoning.
JarvisGUI is a dynamic benchmark that formulates GUI tasks as input-output transformations under a lightweight type system, automatically composing multi-step cross-device workflows across Android, Windows, and Ubuntu virtual environments. Evaluation shows state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a capability gap invisible to existing single-device benchmarks.
- Dynamic benchmark composes typed cross-device workflows across Android, Windows, Ubuntu
- Open-source GUI agents fail at state transfer and long-horizon dependencies
- Typed input-output task formulation enables automatic workflow composition
From the original article
1 min read at arxiv.orgReal-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments.
By Zixiang Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.