Skip to content
NewA calmer, faster ZeroHour

SWE-CoWork shows coding agents rarely finish full assignments

Original title: SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Hugging Face daily papersby Xinping Lei, Minghao Liu, Ruijie Zhu ()1 min read
Part of a story covered by 2 sourcesSWE-CoWork finds coding agents complete under 8% of tasksMerged summary & timeline
AI summary · x-ai/grok-4.7Summaries by Vani AI

Leading models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.

SWE-CoWork places coding agents in simulated engineering organizations with 26 colleagues across 101 real repositories, 11 languages, and 12 domains. Agents must recover distributed requirements, reconcile conflicts, and pass private executable tests. Eleven models were evaluated; the two leaders satisfy about 63% of fail-to-pass tests but complete under 8% of assignments. Full requirements upfront raise fail-to-pass scores by about 16 points for Seed Evolving and GPT-6 Astra without changing complete-task success. Claude-Opus-5.5 asks accountable owners first 93% of the time but obtains only 64% of colleague-held requirements, and 31% of requirements marked complete still fail tests.

Unconfirmed: Scores are the paper's evaluation of eleven models; most model names are not listed.

Key facts
  • The benchmark spans 101 assignments, 11 languages, and 12 domains.
  • Each simulation includes 26 colleagues with bounded knowledge.
  • Upfront requirements add about 16 fail-to-pass points for two named models.
  • 31% of requirements reported complete still fail their tests.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains.

Hugging Face daily papers

By Xinping Lei, Minghao Liu, Ruijie Zhu · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.