Co-Design Bench scores coding agents on real user revisions
Original title: Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
The strongest of ten endpoints passes 69.9% of criteria but fully satisfies only 26.4% of tasks.
Co-Design Bench is a reference-free benchmark of 273 tasks drawn from 37,463 deduplicated queries across 26,314 real user coding sessions. Multimodal Editing and Repository Editing cover localized edits with optional image, audio, video, or document inputs and coordinated repository changes. Tasks include 2,331 hidden expert acceptance criteria graded by a judge-then-refute protocol without gold code or target screenshots. Under a shared agentic harness, repository edits spanning three or four change types scored lower for all ten endpoints than those spanning one or two.
- Tasks come from 26,314 real coding sessions and 37,463 deduplicated queries.
- Hidden expert criteria total 2,331.
- Capability dimensions are Grounded Understanding, UI Transformation, and Functional Implementation.
- All ten endpoints did worse on edits with three or four change types.
From the original article
1 min read at huggingface.co
Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions.
By Xilin Wei, Shengyuan Ding, Xinyu Fang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.