ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
ProgramDistill is a benchmark evaluating coding agents on reconstructing web app features from reference applications, testing nine frontier agents.
ProgramDistill evaluates coding agents on features discovered through interaction with fully functional reference applications, factorizing apps into features with replayable behaviors verified via gold patches. Its mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and built 4,063 tasks without human intervention. On cumulative full-application reconstruction workflows, GPT-6 Astra achieved 49.2% and Claude Opus 5 28.8% success. In partial reconstruction, success drops from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8.
- mine-craft-patch pipeline yields 1,975 replay-verified behaviors across 26 applications
- 4,063 evaluation tasks constructed without human intervention
- GPT-6 Astra 49.2% and Claude Opus 5 28.8% on cumulative full-reconstruction workflows
- Success falls sharply as partial-restoration depth increases from 1 to 8
- Designed for diagnosing coding agents and curriculum-based training
- Products
- GPT-6 AstraClaude Opus 5
- Organizations
- Hugging Face
- AI models
- ProgramDistill
From the original article
1 min read at huggingface.co
Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications.
By Jeonghye Kim, Minseon Kim, Young Jin Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.