CRM-Agent-Bench tests tool-using agents on 555 CRM tasks
Original title: CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
Sixteen systems score 64.32% to 83.60% strict success, with leaders diverging sharply in cost and domain reliability.
CRM-Agent-Bench is an execution-grounded benchmark of 555 stateful, multi-turn customer-relationship tasks across nine business domains and 36 capability tags. Success is the strict conjunction of task-specific checks over actions, arguments, order, response, target state, and protected state. Sixteen systems on the audited final release are scored for coverage, latency, and cost.
- Each task uses an executable contract drawn from those declared checks.
- Leading systems cluster in success but differ in cost and per-domain reliability.
- The benchmark structure is described as portable for later model generations.
- Code is published at github.com/salesforce-misc/crm-agent-bench.
- Products
- CRM-Agent-Bench
- Organizations
- Salesforce
From the original article
1 min read at huggingface.co
Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss.
By Monojit Banerjee, Zhujun Lan, Defu Cao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.