SWE-CoWork shows coding agents rarely finish full assignments
Original title: SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Leading models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.
SWE-CoWork places coding agents in simulated engineering organizations with 26 colleagues across 101 real repositories, 11 languages, and 12 domains. Agents must recover distributed requirements, reconcile conflicts, and pass private executable tests. Eleven models were evaluated; the two leaders satisfy about 63% of fail-to-pass tests but complete under 8% of assignments. Full requirements upfront raise fail-to-pass scores by about 16 points for Seed Evolving and GPT-6 Astra without changing complete-task success. Claude-Opus-5.5 asks accountable owners first 93% of the time but obtains only 64% of colleague-held requirements, and 31% of requirements marked complete still fail tests.
Unconfirmed: Scores are the paper's evaluation of eleven models; most model names are not listed.
- The benchmark spans 101 assignments, 11 languages, and 12 domains.
- Each simulation includes 26 colleagues with bounded knowledge.
- Upfront requirements add about 16 fail-to-pass points for two named models.
- 31% of requirements reported complete still fail their tests.
From the original article
1 min read at huggingface.coA coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains.
By Xinping Lei, Minghao Liu, Ruijie Zhu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.