τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction
New τ^τ-bench tasks coding agents with building deployable customer-service agents; best config, Claude Opus 5, passes only 23.9% of simulations.
Researchers introduce τ^τ-bench, an end-to-end benchmark where a developer agent must build a complete customer-service agent from real business records, a client with requirements, a production API, an inherited codebase, and cost/model limits, then is scored by deploying it against held-out simulated users. Across 53 tasks in four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations versus an 82.2% expert-authored reference ceiling. Failure modes mirror those of human developers: shallow queries instead of deep record comprehension, almost no client communication, and shipping the first architecture that runs rather than experimenting.
- Benchmark makes agent construction itself the scored task
- Claude Opus 5 + Claude Code passes 23.9% of simulations
- Expert-authored reference ceiling scores 82.2%
- Failures: shallow queries, poor client communication, little architecture experimentation
- Vendors
- Anthropic
- Products
- Claude Opus 5Claude Code
- AI models
- Claude Opus 5
From the original article
1 min read at huggingface.co
LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement.
By Quan Shi, Keshav Dhandhania, Karthik Narasimhan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.