BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender
Blender-VideoBench evaluates agentic video understanding by having agents programmatically reconstruct real videos in Blender scenes.
BVB (Blender-VideoBench) tests whether multimodal agents truly understand videos by requiring programmatic reconstruction of real-world videos as animated Blender scenes via a lightweight Mini-BVB harness under identical sandbox and cost constraints. Evaluation uses Dual VQA for spatiotemporal fact preservation and Latent Similarity for perceptual match, combined in a square-root mean overall score. Across 51 configurations from 10 model families, the best model reaches 88.6 Latent Similarity but retains only 53.7% of source-correct spatiotemporal answers, showing semantic retention remains the main challenge.
- 51 configurations from 10 model families evaluated programmatically
- Best model hits 88.6 Latent Similarity but only 53.7% factual retention
- Latent Similarity correlates strongly with human preference in blind study
From the original article
1 min read at huggingface.co
Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmarks still evaluate models mainly through question answering. If an agent truly understands a video, it can reconstruct it programmatically.
By Yolo Y. Tang, Daiki Shimada, Jiayue Meng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.