ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
ViSkill gives VLM agents reusable visual skill cards, reaching 0.89 success and 0.91 with cold start.
ViSkill encodes successful interactions as composite visual skill cards that VLM agents can retrieve for inference and reward shaping. New successful trajectories are distilled back into the library, so skill accumulation and policy improvement reinforce each other, with an optional cold start. On Sokoban, FrozenLake, and PrimitiveSkill, overall success is 0.89 and rises to 0.91 with cold-start initialization. It outperforms the evaluated proprietary and open-source baselines and converges faster than standard PPO.
- Visual skill cards preserve geometry lost in text-only skills
- Retrieved skills guide inference and reward shaping
- Overall success is 0.89, or 0.91 with cold start
- It converges faster than standard PPO on three environments
- Organizations
- ZJU-REAL
From the original article
1 min read at huggingface.co
Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure.
By Hongxing Li, Dingming Li, Yixin Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.