Skip to content
NewA calmer, faster ZeroHour

SWE-CoWork: coding agents finish under 8% of org assignments

Original title: SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

arXiv cs.AI / cs.LG / cs.CLby Xinping Lei ()1 min read1 · 1 read
Part of a story covered by 2 sourcesSWE-CoWork finds coding agents complete under 8% of tasksMerged summary & timeline
AI summary · x-ai/grok-4.7Summaries by Vani AI

Across 101 real repositories, leading models pass about 63% of fail-to-pass tests but complete fewer than 8% of tasks.

SWE-CoWork places coding agents in a simulated engineering organization of 26 colleagues with distinct roles and bounded knowledge. Agents start from a brief handoff, recover requirements from colleagues and workplace records, and are scored by private executable tests. Eleven models show a large gap between partial implementation and full delivery. Giving complete requirements upfront raises fail-to-pass rates by about 16 points for Seed Evolving and GPT-6 Astra, but complete task success does not improve.

Key facts
  • The benchmark has 101 assignments, 11 languages, and 12 application domains.
  • Claude-Opus-5.5 sends 93% of first card questions to accountable owners.
  • That model still obtains only 64% of colleague-held requirements.
  • 31% of requirements reported as completed fail their tests.
  • Deferred questions go unfollowed, and implementations stay incomplete after requirements are acquired.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains.

arXiv cs.AI / cs.LG / cs.CL

By Xinping Lei · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.