RealityBench tests VLMs on documenting physical work
Original title: RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work
Across 26 models, documentation is strong for in-frame text but weak for measurements and small spatial relations.
RealityBench models unconstrained physical work in Blender across eleven environments, scoring whether a view is acceptable proof-of-work and whether a vision-language model can document task-relevant state. Top models largely solve documentation when the answer is text in frame, but struggle with measurements, small parts, and spatial relations. Acceptability is hard because human raters disagree; the best models match a single non-author rater against a majority scorer. On three documentation questions with real images, rankings from rendered frames transfer.
Unconfirmed: Human raters disagree on acceptability, so model gaps versus a majority label are not fully settled.
- Tasks include rooftop solar installation and vehicle repair, not only kitchens or warehouses.
- Most traced wrong rejections come from misreading the scene, not the requirement.
- Acceptability cost for the best models is within a human pass range.
From the original article
1 min read at huggingface.co
Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation.
By Andrew Perrault, Anmol Takiar, Dominik Wienecki · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.