CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
CheckerBench measures whether coding agents can write working static-analysis checkers across 300 CVE-based tasks.
CheckerBench is an executable benchmark of 300 tasks, drawn from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems, that asks coding agents to synthesize static-analysis checkers. Each task provides vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. CheckerLab rebuilds submissions and scores vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model-harness configurations with three repeats, mean Pass@1 was 32.30% and the best result was 45.33%.
- Benchmark has 300 tasks from 297 CVEs in 167 repositories.
- Tasks span 85 CWEs and five language ecosystems.
- Mean Pass@1 across 21 configurations was 32.30%; best was 45.33%.
- CheckerLab scores diagnostic contrast, localization, false positives, and tool use.
From the original article
1 min read at huggingface.co
Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback.
By Hang He, Li Wang, Hao Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.