PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs
PHRBench evaluates how 18 LLMs recover from hallucinated premises across 4,820 controlled cases.
PHRBench is a controlled benchmark of post-hallucination reasoning across four domains and 18 large language models. It scores trajectories by hallucination compliance, avoidance, and heuristic correction, defining an insightful trajectory as correction that reaches the correct answer. Across 4,820 instances, successful recovery is relatively rare and linked to more frequent belief updates. Prompt properties of the hallucination predict recovery, with a lightweight model reaching AUROC 0.847.
- Covers four domains and 18 large language models.
- Scores compliance, avoidance, and heuristic correction separately from final accuracy.
- Successful recovery is rare and tied to more belief updates.
- A lightweight predictor reaches AUROC 0.847 for recovery.
From the original article
1 min read at arxiv.orgHallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood.
By Linghao Meng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.