VibeVideoBench scores agents that render video by writing code
Original title: VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
Across 2,151 rollouts, the strongest of nine systems meets every core requirement on 76.3% of 239 tasks.
VibeVideoBench evaluates multimodal agents that write code to render frames and synthesize sound rather than using diffusion video models. Its 239 English and Chinese tasks span 12 categories and require a finished MP4 with sound within one hour, inside a sandbox with no image or video generator. Scoring uses program checks, about 13,000 rubric checks, and five quality dimensions judged by three VLM panels. GPT-6 Astra and Claude Opus 5.5 rank first and second with both judges and human annotators. Counting missing videos as zero changes rankings; three quarters of lost videos are late, slow, or unstarted final renders.
- Agents run in their native harness and have one hour to deliver.
- How often agents inspect their own frames does not predict quality.
- Judge rankings closely match blind human preferences.
- Delivery failures form a separate ranking axis from requirement satisfaction.
- Products
- VibeVideoBench
- AI models
- GPT-6 AstraClaude Opus 5.5
From the original article
1 min read at huggingface.co
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation.
By Muxi Diao, Wenbin Xu, Lele Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.