MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
MotorMind lets a general VLM control robots zero-shot, reaching 66.7% on LIBERO-PRO.
MotorMind is a robot-manipulation harness that maps mid-level actions proposed by a general vision-language model to deterministic control, with asynchronous monitoring and background memory updates. It uses no task-specific policy training, coding agents, or extra grounding tools such as SAM3. It reports 66.7% success on base LIBERO-PRO suites and 53.8% under perturbations, against at most 13.3% and 19.2% for prior zero-shot methods, and 95% average success on a real xArm6. Stronger VLM backbones improve results; remaining failures are mainly visual grounding, embodied reasoning, and action knowledge.
- No task-specific policies, coding agents, or SAM3 grounding tools are used.
- Base LIBERO-PRO success is 66.7% versus at most 13.3% for prior zero-shot methods.
- Under perturbations, success is 53.8% versus at most 19.2%.
- The same interface averages 95% success on a real xArm6 robot.
Full article258 words · extracted from huggingface.co · click to collapse
Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.38078