Frontier VLM Agents Show Limited Success in Robot Generalist Tasks
Original title: Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena
Closed-weight models lead in all five domains while open-weight VLMs show lower planning and manipulation success.
Robot generalists must interpret a scene act on it and satisfy every condition of a task. The Embodied Agent Arena evaluates ten VLM agents across Geometry Spatial Reasoning Affordance Task Planning and Manipulation using 1000 cases. Closed-weight models lead all five domains whereas open-weight VLMs show useful spatial reasoning but substantially lower planning and manipulation success. Agents achieve early subgoals yet lose object control bind actions to wrong regions or stop before required physical states hold. Additional computation does not uniformly resolve failures.
- 1000 cases combining 32 established sources with 168-case GeoProbe benchmark
- Closed-weight models lead all five domains
- Open-weight VLMs lower planning and manipulation success
- Early subgoals achieved yet object control lost
- Code repair restores execution without completing tasks
From the original article
1 min read at huggingface.co
A robot generalist must interpret a scene, act on it, and satisfy every condition of a task. How far do frontier vision-language models (VLMs) meet these demands? We introduce Embodied Agent Arena, evaluating ten VLM agents across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation.
By Haojian Huang, Pukun Zhao, Zexi Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.