IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
IdeaAMBIG benchmark with 660 instances measures whether LLMs can spot and fix underspecified research-method details for faithful implementation.
Researchers introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances (163 real-world gaps from reproducibility reports and GitHub issues, 497 controlled synthetic gaps) built from papers, codebases, and reproduction artifacts. It evaluates codification-readiness assessment, defect localization, and clarification action generation. Across 13 LLMs, the best model achieved only 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% clarification success when given the annotated defect. An oracle study showed gold resolutions raise the codification-ready rate from 14% to 98%, identifying defect localization as the main bottleneck.
- 660-instance benchmark spans 163 real-world and 497 synthetic specification gaps.
- Best of 13 LLMs achieved 9.6% defect recovery on real-world instances.
- With defects annotated, best clarification success reached 80.6%.
- Gold resolutions lift codification-ready rate from 14% to 98%.
From the original article
1 min read at arxiv.orgA research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.
By Yiling Ma · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.