ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
ViSkill stores visual skill cards for VLM agents, reaching 0.89 success and 0.91 with cold-start on grid tasks.
ViSkill encodes successful interactions as composite visual skill cards that vision-language-model agents retrieve for inference and reward shaping, then distills new successes back into the library. An optional cold-start accelerates early learning. On Sokoban, FrozenLake, and PrimitiveSkill it reaches 0.89 overall success, or 0.91 with cold-start, outperforming the evaluated proprietary and open-source baselines and converging faster than PPO. Code is released by ZJU-REAL.
- Visual skill cards preserve geometry that text-only skills discard.
- Retrieved skills guide inference and reward shaping in a closed loop.
- Overall success is 0.89, rising to 0.91 with cold-start.
- It beats evaluated baselines and converges faster than standard PPO.
- Organizations
- Zhejiang University
- AI models
- ViSkill
From the original article
1 min read at arxiv.orgSkill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure.
By Hongxing Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.