SWE-CoWork: coding agents finish under 8% of org assignments
Original title: SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Across 101 real repositories, leading models pass about 63% of fail-to-pass tests but complete fewer than 8% of tasks.
SWE-CoWork places coding agents in a simulated engineering organization of 26 colleagues with distinct roles and bounded knowledge. Agents start from a brief handoff, recover requirements from colleagues and workplace records, and are scored by private executable tests. Eleven models show a large gap between partial implementation and full delivery. Giving complete requirements upfront raises fail-to-pass rates by about 16 points for Seed Evolving and GPT-6 Astra, but complete task success does not improve.
- The benchmark has 101 assignments, 11 languages, and 12 application domains.
- Claude-Opus-5.5 sends 93% of first card questions to accountable owners.
- That model still obtains only 64% of colleague-held requirements.
- 31% of requirements reported as completed fail their tests.
- Deferred questions go unfollowed, and implementations stay incomplete after requirements are acquired.
From the original article
1 min read at arxiv.orgA coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains.
By Xinping Lei · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.