World Models' Last Exam in Physics
A 40-task physics benchmark finds video world models remain physically inconsistent, with the best scoring 57.76.
World Models' Last Exam in Physics is a measurement-based benchmark of physical consistency in video world models. It includes 40 controlled tasks across mechanics, optics, fluids, thermal and phase-change phenomena, electromagnetism, and surface tension, scored without reference videos. Tests of eight models on 1,280 videos found persistent inconsistencies, with the best overall score at 57.76 out of 100. The evaluator matched human judgments better than a direct vision-language model baseline.
- Forty tasks cover mechanics, optics, fluids, thermal effects, electromagnetism, and surface tension.
- Eight models were tested on 1,280 videos; the best scored 57.76.
- Scoring uses quantitative physical measurements rather than reference videos.
- The evaluator agreed with humans more than a direct vision-language baseline.
From the original article
1 min read at huggingface.co
Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems. Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests largely focus on mechanics.
By Mingju Gao, Qingle Liu, Yuzhao Peng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.