CyberBinGym tests AI agents exploiting binary patch diffs
Original title: CyberBinGym: Can AI Agents Turn Binary Patches into Real Exploits?
The strongest agent recovered the flaw in 88 percent of 100 tasks and produced a working exploit in 20 percent.
CyberBinGym gives agents stripped vulnerable and patched binaries from real flaws, without source code or vulnerability details, and asks them to recover and exploit the bug. The benchmark has 100 tasks from 53 projects, six vulnerability families, and three analysis levels. Scoring separates crash-based recovery from end-to-end exploitation and audits trajectories for reward hacking. Supplying the patched binary raised exploit success on a matched subset, and some exploits still worked with standard hardening.
Unconfirmed: Results describe a controlled benchmark, not confirmed attacks against live unpatched systems.
- Tasks come from real vulnerabilities in 53 widely used projects.
- On 20 matched tasks, the patched binary raised exploit rate from 10 to 25 percent.
- With hardening enabled, the agent still exploited two of 20 previously successful tasks.
- The authors say exploitation remains unreliable in this controlled setting.
From the original article
1 min read at arxiv.orgSecurity fixes usually reach users as compiled updates, but deployment delays leave unpatched systems exposed. During this window, comparing patched and unpatched binaries can reveal how to exploit a vulnerability, turning a defensive update into actionable information for attackers.
By Lambang Akbar Wijayadi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.