Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
Seven tool-using agents almost always call failed retrieval useless but rarely stop unless the harness forces it.
In a retrieval environment with controlled source failures, seven tool-using agents judged a failing source's results useless 97-100% of the time yet rarely stopped on that judgment. Prompt changes altered when they stopped but not what they stopped on: memory permission and a reasoning mode caused early stops, a stated budget pushed 7-8B models to the deadline, and stopping rules or call costs were followed only partly. Stopping tracked the evidence only when the harness required an answer after five consecutive results judged useless, which improved failing-source success and stayed fixed when the budget doubled. A pre-registered replication on 300 fresh questions confirmed both the dissociation and the rule's effect.
- Seven agents judged failing results useless 97-100% of the time
- Most rarely stopped because of that judgment
- Prompt cues changed stop timing more than the evidence used
- A five-useless-result harness rule raised success for every model
- A 300-question replication confirmed the gap and the rule
From the original article
1 min read at huggingface.co
An agent whose tool keeps returning nothing useful should stop relying on it. In a retrieval environment with controlled source failures, we separate how agents judge results from what they do. We compare stopping at the same step after longer and shorter runs of results the agent judged useless; this contrast is zero for clock- or deadline-driven stopping.
By Chubin Zhang, Zhenglin Wan, Xingrui Yu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.