To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech
VeriSpeak tests whether audio language models can verify 3,879 spoken claims, with retrieval plus reasoning reaching 86.1%.
VeriSpeak is a public probe benchmark of 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels, for fact-checking by large audio language models. Experiments show a consistent text-to-speech gap: models that verify written claims reliably often fail on the same claims when spoken. Retrieval alone adds limited benefit because models conflate retrieved evidence with the spoken claim, while retrieval plus explicit reasoning improves comparison, and a thinking-tuned model reached 86.1% accuracy. The dataset is released on Hugging Face.
- VeriSpeak contains 3,879 spoken claims with balanced true and false labels.
- Claims cover temporal, geographical, and relational facts from spoken media.
- Models that verify written claims often fail the same claims when spoken.
- Retrieval alone helps little because models conflate evidence with the claim.
- Retrieval plus explicit reasoning reached 86.1% on a thinking-tuned audio model.
Full article180 words · extracted from arxiv.org · click to collapse
Online misinformation increasingly appears in spoken formats such as news clips, podcasts, interviews, political speeches, and social media videos, creating a need for fact-checking systems that can verify claims directly from speech. We introduce VeriSpeak, a probe benchmark for studying speech-based fact verification in Large Audio Language Models (LALMs). VeriSpeak contains 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels. The benchmark is designed to examine whether factual verification ability transfers from text to speech, and whether retrieval-augmented LALMs can use textual evidence to correctly support or refute spoken claims. Our experiments reveal a consistent text-speech modality gap: LALMs that verify written claims reliably often fail on the same claims when spoken. Moreover, retrieval alone provides limited gains because models frequently conflate retrieved evidence with the spoken claim. In contrast, retrieval combined with explicit reasoning improves claim-evidence comparison, with a thinking-tuned LALM reaching 86.1% accuracy. VeriSpeak highlights that effective speech misinformation detection requires not only speech understanding, but also grounded reasoning over retrieved evidence. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/abhiram4572/VeriSpeak.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.30227