Closed-World Resolution Against Tool Hallucination in LLM Agents
Benchmark across ten LLMs documents 322 tool hallucinations and 154 more on MCP, showing model scale does not help and gates cannot reject fabricated calls.
The paper presents a five-class taxonomy (H1-H5) of tool hallucination in LLM agents, where models call nonexistent tools or pass arguments no schema declares, a blind spot no gating defense can reject since no gate made the decision. Across ten hosted models on two invocation surfaces, researchers measured 322 genuine hallucinations, concentrated on the unconstrained raw-JSON surface (34 versus 3), with model scale offering no benefit as a 675B model matched a 7-8B one. On the Model Context Protocol, merging servers into one namespace produced 154 hallucinations, including from frontier models that were clean on the single-registry surface. The versioned Hallucinated-Tools Benchmark (HTB) is released for comparable resolver evaluation.
- Five-class taxonomy (H1-H5) of tool hallucination, with proof that defense must precede any gate.
- 322 genuine hallucinations measured across ten hosted models on two invocation surfaces.
- Fabricated-tool calls concentrate on unconstrained raw-JSON surfaces: 34 versus 3.
- MCP server merging creates collision and shadowing surfaces causing 154 hallucinations, including frontier models.
- Versioned Hallucinated-Tools Benchmark (HTB) released for comparable resolver evaluation.
Full article264 words · extracted from arxiv.org · click to collapse
Tool-augmented large language model (LLM) agents fail in a way no tool-selection or tool-security method addresses: they call tools that do not exist and pass arguments no schema declares. Existing defenses either pick the right tool (selection) or constrain what an agent may do with real tools (gating), both of which presuppose the emitted call refers to a real tool at all. We show this is a structural blind spot: a hallucinated call is by construction not a decision any gate made, so no gate can reject it. This paper is primarily a measurement and benchmark study. We give a five-class taxonomy of tool hallucination (H1-H5) and, as a reference point, the Resolution Rung: a training-free, closed-world resolver (registry membership plus a signature check) whose interest is where it must sit, not what it computes. We prove hallucination defense must precede any causal gate, and characterize the one irreducible residue (borrowed arguments schema-indistinguishable from a valid call). Across ten hosted models under two invocation surfaces we measure 322 genuine hallucinations; fabricated-tool calls concentrate on the unconstrained raw-JSON surface (34 vs. 3), and model scale does not help (a 675B model matches a 7-8B one). We then extend to the Model Context Protocol, where merging several servers into one namespace creates hallucination surfaces a single registry cannot express (a second taxonomy, M1-M5); on the live MCP surface we measure 154 hallucinations, including from frontier models that were clean on the single-registry surface, because collisions and shadowing are structural to the merge. We release the versioned Hallucinated-Tools Benchmark (HTB) so any resolver is comparable across submissions.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.19425