SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?
SWEeper-Bench tests whether coding agents can find and fix interaction-only bugs across 200 real web apps.
SWEeper-Bench contains 200 tasks built from real bug fixes in widely used open-source web applications. Agents receive only the codebase and an open-ended request, then must design interactive tests because the bugs are hard to see from code alone. An agentic verifier scores each repair by what a user would observe in the browser. The best of 15 frontier agents passes 59.0%, and trajectory analysis shows the largest failure is identifying the buggy behavior.
- 200 tasks drawn from real fixes in 200 open-source web apps
- Agents get only the codebase and an open-ended request
- Best of 15 frontier agents passes 59.0% of tasks
- Main bottleneck is triggering or recognizing the buggy behavior
- An agentic verifier judges repairs by browser-visible behavior
- Products
- Focalboard
From the original article
1 min read at huggingface.co
Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive.
By Yang Yao, Haozhe Chen, Bingyi Kang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.