WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
WorldAuditBench tests multimodal agents at finding 3D-world anomalies; frontier models reach 6.6–42.3% versus humans at 83.4%.
WorldAuditBench is a benchmark for auditing interactive 3D worlds for anomalies such as floating objects, traversable walls, and scene-inconsistent objects. It includes 213 tasks across 13 environments built with Unreal Engine 5 and Three.js, covering five anomaly families. Five frontier models were tested under a fixed exploration budget in two setups: VLA exploration followed by VLM identification, and an end-to-end VLM agent. Success rates ranged from 6.6% to 42.3%, well below human performance of 83.4%.
- Benchmark has 213 anomaly tasks across 13 environments and five families.
- Environments were built with Unreal Engine 5 and Three.js.
- Five frontier models scored 6.6% to 42.3% under a fixed budget.
- Human auditors reached 83.4% on the same tasks.
- Compares cascaded VLA-then-VLM auditing with an end-to-end VLM agent.
Full article243 words · extracted from arxiv.org · click to collapse
As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or objects inconsistent with the surrounding scene. Multimodal AI systems, including vision-language models (VLMs) and vision-language-action models (VLAs), have shown potential for automating this task. However, 3D world auditing is complex, requiring the close coupling of two distinct capabilities: action, to navigate the 3D world and search for anomalies systematically and efficiently; and visual reasoning, to understand the environment and identify anomalies from multimodal observations. It remains largely unexplored whether multimodal agents can effectively couple these two capabilities, using visual reasoning to identify potential anomalies while taking actions to validate them. In this paper, we introduce WorldAuditBench, a benchmark for 3D world auditing comprising 213 anomaly tasks across 13 environments built with Unreal Engine 5 and Three.js, spanning five anomaly families. We evaluate five frontier models under a fixed exploration budget using two auditing paradigms: VLA-based exploration followed by VLM-based anomaly identification, and an end-to-end VLM agent in which visual reasoning directly guides action selection. Across the evaluated models and two paradigms, success rates range from 6.6% to 42.3%, substantially below human performance (83.4%). Through the task of world auditing, WorldAuditBench provides a testbed for studying how multimodal agents couple action and visual reasoning in interactive 3D environments, while highlighting current limitations in their ability to gather and interpret evidence during exploration.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.40325