Do Self-Evolving Skills Generalize to Held-Out Tasks?
Of 21 self-evolving skills that improve in training, only five fully transfer their gains to held-out tasks.
The paper tests whether self-evolving skills that improve on training tasks also help held-out tasks of the same kind. Five self-evolving methods and a one-shot skill were compared on six benchmarks with a matched model, agent, and train/test split. Of 21 skills that improved in training, five kept all of the gain on test, 13 kept part, and three kept none, often by hard-coding task-specific details. An LLM judge matched test rankings for 86% of skill pairs but predicted single-edit effects poorly. Proposed Generalizable Skill Optimization writes a fresh skill per task and scored highest on all six benchmarks.
- Five of 21 improved skills fully transferred; three transferred none
- No existing method was best on every benchmark
- An LLM judge matched test rankings on 86% of skill pairs
- GSO writes a new skill per task and led all six benchmarks
From the original article
1 min read at arxiv.orgAI agents can externalize what they learn from past tasks into reusable \emph{skills}, such as procedures, checklists, code, or other executable artifacts, that can be retrieved and reused when solving new tasks. Self-evolving skill methods keep rewriting these skills after each round of practice on training tasks, and the skill is then used on new tasks of the same kind.
By Xihao Piao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.