From Evidence to Action: How Tool-Using Agents Fail
SafeActBench finds tool-using agents often act before evidence is established, despite strong static judgments.
The paper studies where the evidence-to-action chain breaks for tool-using agents, from deciding whether to act through single actions and dependent workflows. Across ten model-harness configurations, strong static action assessment often coexists with much weaker interactive execution. Agents frequently stop after incomplete investigation or act before required evidence exists; once that evidence is obtained, single actions are usually reliable, while multi-step workflows still miss prerequisites. SafeActBench contains 656 cases across six operational domains and five protocols, scored with a provenance-bound Evidence Ledger and a deterministic trajectory evaluator.
- SafeActBench contains 656 cases across six domains and five protocols.
- Strong static action judgments can coexist with much weaker interactive execution.
- Many failures start before execution, from incomplete investigation or premature action.
- Single-action execution is usually reliable once required evidence is obtained.
- Multi-action workflows still miss prerequisites and leave execution incomplete.
From the original article
1 min read at huggingface.co
Tool-using agents make consequential changes to external state, yet correct outcomes do not guarantee that their actions were supported by evidence established beforehand. We study where this evidence-to-action chain breaks as agents move from deciding whether to act to executing single actions and dependent workflows.
By Hongzhan Lin, Shidong Cao, Ziyang Luo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.