RoboColosseum evaluates VLA policies across robot embodiments
Original title: RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
RoboColosseum shows frontier agents rival specialized VLA policies but fine-tuning leads to rank reversals and embodiment-specific performance differences.
RoboColosseum is a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. It evaluates five vision-language-action policies and three frontier AI agents across four embodiments using open and fine-tune tracks with 5699 pairwise comparisons at 124 sites. Open Track results show frontier agents rival specialized policies but all struggle on open-world tasks. Fine-tuning to 20 tasks per embodiment improves performance with GR00T N1.7 leading on SO-101 and pi0.5 on Franka and Bimanual-YAM.
- 5699 pairwise Open Track comparisons at 124 sites
- Frontier agents rival specialized VLA policies
- Weak cross-embodiment rank agreement (Kendall W=0.24)
- Fine-tuning causes substantial rank reversals
- No policy completes any G1 Humanoid tasks but progress ranges 13.3-23.2%
From the original article
1 min read at huggingface.co
General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments.
By Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.