RoboJudge-9B tops MLLM judges of embodied video generation
Original title: RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
Trained on 12K videos, the 9B model reaches Pearson 0.719, 0.089 above Gemini-3.7-Flash.
Researchers introduce RoboJudge-Bench to test whether multimodal language models agree with humans when judging embodied video rollouts. The benchmark scores Physical Adherence and Instruction Alignment on 800 annotated videos from eight corpora and ten generators. General-purpose models agree less on physical adherence, and general-domain video judges do not transfer consistently. Fine-tuning a 9B model on RoboJudge-Train with supervised then reinforcement learning produces RoboJudge-9B, the strongest judge they evaluate.
- Videos span grippers, human hands, and dexterous hands.
- RoboJudge-Train has 12K videos and 25K human-verified scores with model-assisted rationales.
- The Pearson gain of 0.089 over Gemini-3.7-Flash is statistically significant.
- The same training also improved results on VideoPhy2 and Physion-Eval.
- AI models
- RoboJudge-9BGemini-3.7-Flash
From the original article
1 min read at huggingface.co
Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated.
By Siyuan Ma, Yu Qi, Cheng Ye · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.