Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut
Timeline-Bench finds frontier coding agents finish only about 27 percent of 56 realistic video-editing tasks.
Timeline-Bench comprises 56 real video-editing tasks that require an agent to turn raw production assets into a finished video passing format, content, brief, and quality tests. Quality checks were calibrated on 2,582 blind judgments by 43 editors. Across 16 agents pairing frontier models with harnesses such as Codex, Claude Code, and OpenCode, the best system, GPT-6 Astra in Codex with editorial guidance, resolved 15 of 56 tasks (26.8 percent); the average resolved 14.0 percent. Human editors preferred the reference edit in 83.5 percent of judgments, and 562 of 771 unresolved runs failed only the quality test.
- Benchmark has 56 tasks that require a finished video edit.
- Best agent, GPT-6 Astra in Codex, solved 15 of 56 tasks.
- Average agent resolved 14.0 percent of tasks.
- 562 of 771 unresolved runs failed only the quality test.
- Editors preferred reference edits in 83.5 percent of judgments.
Full article204 words · extracted from arxiv.org · click to collapse
AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we introduce Timeline-Bench, a benchmark of 56 real video-editing tasks, each asking an agent to turn raw production material into a finished video. Tasks range from selecting dialog takes and shaping interview footage into a story to cutting commercials from product shots, voiceovers and graphics. Every task provides a brief, source assets, a container and a set of tests. A task is resolved when the output passes every test. The tests check the delivery format, the content and the brief's explicit requirements, and include a quality test calibrated on 2,582 blind judgments by 43 video editors. We evaluate 16 agents that pair frontier models with coding-agent harnesses such as Codex, Claude Code and OpenCode. The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of the 56 tasks (26.8%), and the average agent resolves 14.0%. Human editors prefer the reference edit in 83.5% of judgments. Most unresolved runs (562 of 771) fail only the quality test: agents perceive footage through stills and transcripts and check their renders for defects, not craft. We release the tasks, verifier and per-run results at https://timelinebench.tensortest.com.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.35143