UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents
UndoBench shows tool agents hit 83.54% task success but only 46.72% conditional recovery, with retries duplicating effects.
UndoBench separates tool-using agents' task competence from fault recovery using 36 workflows and 36 fault scenarios across eight enterprise domains and counterfactual paired trials. On 12 held-out workflows, 5,760 executions showed 83.54% nominal competence but only 46.72% conditional recovery success, and naive retry duplicated external effects in 53.33% of trials. Recovery is phase-dependent: methods behave similarly before mutation, collapse during partial mutation, and improve after commit when verification and server-side idempotency are used. Commercial API models showed the same gap.
- Covers 36 workflows and 36 fault scenarios across eight enterprise domains.
- Nominal competence was 83.54% but conditional recovery fell to 46.72%.
- Naive retry duplicated external effects in 53.33% of trials.
- During partial mutation, retry, idempotency, and journaling all collapsed.
- Commercial API models reproduced the competence-recovery gap.
From the original article
1 min read at huggingface.co
Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery.
By Dolly Sah, Tanmay Sah, Harshul Jain · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.