MotionGraphicsBench scores SVG motion edits with verification programs
Original title: MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification
Pass@1 across 47 configurations ranged from 6.0% to 67.7%, and verifiers agreed with humans 93.3% of the time.
MotionGraphicsBench is an expert-curated set of 100 instances that ask models to animate a static SVG from a motion-edit prompt, using real Lottie designs. Each instance includes a deterministic verification program that checks prompt requirements instead of scoring against a reference or a VLM judge. On 180 animations, the programs agreed with humans 93.3% of the time, above 83.3% for a GPT-6 Astra baseline. Across 47 configurations, pass@1 spanned 6.0% to 67.7%; at 32 attempts, pass@32 still spanned 38.0% to 95.0%.
- The task supplies all visual elements so scoring isolates motion generation.
- Programs were built with a multi-agent pipeline and synthesized test cases.
- Fifteen models were tested at highest reasoning effort with 32 attempts each.
- Common failures miss poses, trajectories, or coordinated events.
- The dataset is released as figma-ai/motion-graphics-bench.
- Organizations
- Figma
- AI models
- GPT-6 Astra
From the original article
1 min read at huggingface.co
Large language models can generate motion graphics from static images and natural language requests, but it remains difficult to evaluate how well models can follow requests that combine spatial, appearance, and temporal requirements. We introduce MotionGraphicsBench, an expert-curated benchmark of 100 instances for animating an input static SVG with a motion edit prompt.
By Jiaju Ma, Ashvini Kumar Jindal, Chuhua Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.