4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
4DCodeBench evaluates AI agents reconstructing dynamic scenes from video as executable graphics programs; frontier models handle static scenes but fail on complex dynamics.
4DCodeBench is a benchmark for 4D inverse graphics through code generation, where agents must reconstruct dynamic scenes from video as executable graphics programs, including abstractions like physical simulations. The evaluation set mixes real-world videos and synthetic scenes covering deformation, fluid flow, and fracture. Extensive benchmarking of frontier models shows that strong static reconstruction capability does not yet translate into reliable reconstruction of complex dynamics. The benchmark is open-sourced on GitHub as a testbed for tracking progress.
- Agents translate video observations into executable graphics programs with physical simulations.
- Covers deformation, fluid flow, and fracture in real and synthetic videos.
- Frontier models' static reconstruction skill does not transfer to dynamic scenes.
Full article124 words · extracted from arxiv.org · click to collapse
We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs. To accomplish this, agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that can interpret the dynamics of the world through code. Our benchmark is available at https://github.com/4DCodeBench/4DCodeBench
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.03715