RefCraft: strongest agent scores 52.12 on visual webpage tasks
Original title: RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development
The benchmark has 300 tasks pairing instructions with screenshots from one to five reference pages, judged by keypoints plus programmatic and VLM checks.
RefCraft evaluates whether agents can build webpages from natural-language instructions and visual references rather than text alone or pure visual reconstruction. Agents must select useful parts of one to five reference screenshots, render, inspect, and iteratively refine open-ended pages. Across nine frontier multimodal models, the strongest system scores 52.12 out of 100. The authors say visual-reference-conditioned agentic generation remains a substantial challenge.
- Tasks use independently verifiable keypoints plus programmatic and VLM judges.
- Existing webpage benchmarks often reconstruct visuals or follow text instructions only.
- Nine frontier multimodal language models were evaluated.
From the original article
1 min read at arxiv.orgBeyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks.
By Yuchi Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.