RoboValue benchmark tests robotic value models on sim and real tasks
Original title: RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models
Errors remain in instruction grounding, progress tracking, and recovery even when outcome scores look strong.
Researchers introduce RoboValue, a fine-grained sim-and-real benchmark for robotic value models that score task execution from visual observations and instructions. Shared interfaces cover task-state understanding, temporal progress, failure and recovery reasoning, and value consistency. The dataset spans 15 simulation and 20 real-world manipulation tasks, with expert demonstrations and annotated trajectories. Zero-shot and one-shot tests under embodiment and environment shifts show inconsistent value gains for the same subtask.
- Four capability dimensions share one evaluation interface.
- Dataset includes 15 simulation and 20 real-world tasks.
- Models were tested zero-shot and one-shot under embodiment and environment shifts.
- The same semantic subtask can get different value gains across valid orders.
- A public leaderboard is at robovalue-benchmark.github.io.
From the original article
1 min read at huggingface.co
Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations cover only selected aspects of execution understanding, leaving the reliability of value feedback insufficiently tested. We introduce RoboValue, a fine-grained sim-and-real benchmark for unified evaluation of robotic value models.
By Shengbang Liu, Zhengye Du, Zhilong Wan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.