SWE-CoWork finds coding agents complete under 8% of tasks
First merged summary.
SWE-CoWork finds leading coding models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.
SWE-CoWork places coding agents in simulated engineering organizations of 26 colleagues with distinct roles and bounded knowledge, across 101 real repositories, 11 languages, and 12 domains. Agents start from a brief handoff, recover requirements from colleagues and workplace records, reconcile conflicts, and are scored by private executable tests. Eleven models were evaluated; the two leaders satisfy about 63% of fail-to-pass tests but fully complete fewer than 8% of assignments. Giving complete requirements upfront raises fail-to-pass scores by about 16 points for Seed Evolving and GPT-6 Astra without improving complete-task success. Claude-Opus-5.5 asks accountable owners first 93% of the time but obtains only 64% of colleague-held requirements, and 31% of requirements marked complete still fail tests.
- The benchmark spans 101 assignments, 11 languages, and 12 domains.
- Each simulation includes 26 colleagues with bounded knowledge.
- Upfront requirements add about 16 fail-to-pass points for Seed Evolving and GPT-6 Astra.
- Complete-task success does not improve when full requirements are given upfront.
- Claude-Opus-5.5 obtains 64% of colleague-held requirements despite asking owners first 93% of the time.
- 31% of requirements reported complete still fail tests; deferred questions go unfollowed.
What each outlet adds
beyond the shared facts- arXiv cs.AI / cs.LG / cs.CLDeferred questions go unfollowed; implementations stay incomplete after requirements are acquired.
- Hugging Face daily papersAgents must reconcile conflicts while recovering distributed requirements.
Coverage
oldest first · one row per article · UTCSWE-CoWork shows coding agents rarely finish full assignments
Leading models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.
SWE-CoWork: coding agents finish under 8% of org assignments
Across 101 real repositories, leading models pass about 63% of fail-to-pass tests but complete fewer than 8% of tasks.