Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
Work2Skill turns human GUI demos into reusable skills that raise computer-use agent scores.
Work2Skill induces reusable computer-use agent skills from human GUI demonstrations. WorkRecorder aligns screen recordings, mouse and keyboard events, and accessibility-tree data to build WorkDemo-350, 350 screened successful desktop demonstrations. An agentic induction method extracts workflow rules, step dependencies, and completion checks. With four shared execution models, induced skills beat a No Skill baseline by 4.67-10.54 points on MyPCBench Bounded Action and 3.11-7.50 points on a 286-task OSWorld-Verified subset.
- WorkRecorder aligns screen video, input events, and accessibility-tree data.
- WorkDemo-350 has 350 screened, successful human desktop GUI demonstrations.
- Induced skills encode workflow rules, step dependencies, and completion checks.
- Gains are 4.67-10.54 points on MyPCBench and 3.11-7.50 on 286 OSWorld-Verified tasks.
From the original article
1 min read at huggingface.co
Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details.
By Yuqi Huang, Hui Li, Yifei Cao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.