Skip to content
NewA calmer, faster ZeroHour
StoryAI researchinfo

SWE-CoWork finds coding agents complete under 8% of tasks

2 sources · 2 articlesFirst reported Updated ()
What's new

First merged summary.

Merged summary · x-ai/grok-4.7 · rewritten as coverage arrivesSummaries by Vani AI

SWE-CoWork finds leading coding models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.

SWE-CoWork places coding agents in simulated engineering organizations of 26 colleagues with distinct roles and bounded knowledge, across 101 real repositories, 11 languages, and 12 domains. Agents start from a brief handoff, recover requirements from colleagues and workplace records, reconcile conflicts, and are scored by private executable tests. Eleven models were evaluated; the two leaders satisfy about 63% of fail-to-pass tests but fully complete fewer than 8% of assignments. Giving complete requirements upfront raises fail-to-pass scores by about 16 points for Seed Evolving and GPT-6 Astra without improving complete-task success. Claude-Opus-5.5 asks accountable owners first 93% of the time but obtains only 64% of colleague-held requirements, and 31% of requirements marked complete still fail tests.

Key facts
  • The benchmark spans 101 assignments, 11 languages, and 12 domains.
  • Each simulation includes 26 colleagues with bounded knowledge.
  • Upfront requirements add about 16 fail-to-pass points for Seed Evolving and GPT-6 Astra.
  • Complete-task success does not improve when full requirements are given upfront.
  • Claude-Opus-5.5 obtains 64% of colleague-held requirements despite asking owners first 93% of the time.
  • 31% of requirements reported complete still fail tests; deferred questions go unfollowed.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

What each outlet adds

beyond the shared facts
  • arXiv cs.AI / cs.LG / cs.CLDeferred questions go unfollowed; implementations stay incomplete after requirements are acquired.
  • Hugging Face daily papersAgents must reconcile conflicts while recovering distributed requirements.

Coverage

oldest first · one row per article · UTC
  1. · 2d ago Hugging Face daily papersFirst report

    SWE-CoWork shows coding agents rarely finish full assignments

    Leading models pass about 63% of fail-to-pass tests but fully complete fewer than 8% of 101 assignments.

  2. · 1d ago arXiv cs.AI / cs.LG / cs.CL

    SWE-CoWork: coding agents finish under 8% of org assignments

    Across 101 real repositories, leading models pass about 63% of fail-to-pass tests but complete fewer than 8% of tasks.