Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
EvoSkill-GUI introduces training-free skill revision for GUI agents, boosting MobileWorld, AndroidWorld, and OSWorld success by up to 16.2%.
EvoSkill-GUI is a training-free framework in which each GUI-agent skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, and failure cases. A reflect-revise-reuse loop lets the executor make in-rollout revisions while an isolated critic diagnoses failed trajectories under strict information isolation. On MobileWorld, AndroidWorld, and OSWorld it improves multiple base models by up to +16.2%, +6.0%, and +10.5% respectively, with evolved skill libraries transferring to related tasks. Code is released on GitHub.
- Skills are multi-file packages with retrieval metadata, executable plans, and failure-recovery rules.
- Reflect-revise-reuse loop revises skills at deployment time without training.
- Isolated critic diagnoses failed trajectories under strict information isolation.
- Gains: +16.2% MobileWorld, +6.0% AndroidWorld, +10.5% OSWorld.
- Evolved skill libraries continue benefiting related tasks instead of being rebuilt.
- Organizations
- ZJU-REAL
From the original article
1 min read at huggingface.co
GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution.
By Bofan Chen, Boxuan Zhang, Fei Tang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.