BizBench tests long-horizon payment agents on constraint-valid execution
Original title: BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business
Claude Opus 5 and GPT-6 Sol reach only 64.0% and 61.3% strict success on 50 payment tasks.
BizBench is a multi-turn benchmark of long-horizon business agents, grounded in procedures from a leading payments company. Its environment includes 789 knowledge documents and 601 discoverable tools, with 50 tasks covering resource limits, time dependencies, prior commitments, and conflicting or outdated guidance. Deterministic checks score both goal attainment and execution validity. Claude Opus 5 and GPT-6 Sol reach strict success of 64.0% and 61.3%. Qwen3.8-27B hits 42.0% goal attainment but 32.7% strict success; DeepSeek-V4.1-Flash averages 191.3 tool calls, about 70% more than reference runs.
Unconfirmed: The payments company behind the procedures is not named.
- Each task is validated by a reference execution without a single prescribed path.
- Outcome-only scoring can hide invalid intermediate steps.
- The domain is payments, where policy violations can cause financial or regulatory harm.
From the original article
1 min read at huggingface.co
Existing long-horizon agent benchmarks often emphasize final task success while providing limited assessment of intermediate execution validity. Yet real-world business operations require agents to navigate extensive, potentially outdated or conflicting policies while satisfying critical constraints throughout execution.
By Enhan Zhao, Yongqi Tong, Xuejun Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.