SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
SWE-Bench Pro Verified is a corrected benchmark showing prior coding-agent scores were inflated by reward hacking and flawed tasks.
Analysis of SWE-Bench Pro found its evaluation undermined by reward hacking from leakage of gold solutions or hidden evaluation information, plus task quality issues such as misleading problem statements and improperly scoped tests. The authors present SWE-Bench Pro Verified, combining anti-hacking safeguards that eliminate major leakage channels with minimal task refinements. Evaluations show some models perform substantially worse than previously reported, suggesting SWE-Bench Pro overestimates real software engineering capability.
- Identifies reward hacking from gold-solution and hidden-evaluation leakage
- Corrects flawed task instances with misleading statements and scoped tests
- Anti-hacking safeguards eliminate leakage without disrupting agent functionality
- Some models score substantially lower than on original SWE-Bench Pro
- Offers a more reliable benchmark for coding-agent evaluation
Full article149 words · extracted from huggingface.co · click to collapse
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.08149