Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
EvoSkill-GUI introduces training-free skill revision for GUI agents, boosting MobileWorld, AndroidWorld, and OSWorld success by up to 16.2%.
EvoSkill-GUI is a training-free framework in which each GUI-agent skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, and failure cases. A reflect-revise-reuse loop lets the executor make in-rollout revisions while an isolated critic diagnoses failed trajectories under strict information isolation. On MobileWorld, AndroidWorld, and OSWorld it improves multiple base models by up to +16.2%, +6.0%, and +10.5% respectively, with evolved skill libraries transferring to related tasks. Code is released on GitHub.
- Skills are multi-file packages with retrieval metadata, executable plans, and failure-recovery rules.
- Reflect-revise-reuse loop revises skills at deployment time without training.
- Isolated critic diagnoses failed trajectories under strict information isolation.
- Gains: +16.2% MobileWorld, +6.0% AndroidWorld, +10.5% OSWorld.
- Evolved skill libraries continue benefiting related tasks instead of being rebuilt.
Full article206 words · extracted from huggingface.co · click to collapse
GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \emph{reflect-revise-reuse} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.17653