DialFork benchmark measures procedural hallucination in dialogue agents
Original title: DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents
The set has 36,278 forked dialogues, and FlowScore beats semantic similarity across seven encoders.
Researchers introduce DialFork to measure procedural hallucination, where conversational agents stay fluent and factually plausible but leave the intended process. Conformant dialogues are forked at controlled turns and injected with one of four hallucination types. Three tasks cover set-level monitoring, single-dialogue retrieval, and the first off-track turn. Dialog-flow metrics FlowKL and FlowScore compare how dialogues unfold rather than their content.
- Construction yields 36,278 dialogues across 24 tasks in 12 domains.
- Thirty human annotators validate the injected hallucination types.
- FlowScore beats baselines on seven encoders at dialogue level.
- DialFork is released under the MIT license.
From the original article
1 min read at huggingface.co
Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process.
By Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.