SmoothScore measures overconfident claims by LLM research agents
Original title: Quantifying Claim Smoothing by LLM Research Agents
Across 7 systems and 4 datasets, the authors find agents make stronger claims than the cited evidence supports.
The paper studies LLM research agents that read large document collections and write long reports. It defines “smoothing” as collapsing conflicting sources into a confident narrative that is more decisive than the citations support. SmoothScore extracts conclusive claims, retrieves and clusters relevant evidence, and scores each claim against that evidence. Across 7 systems and 4 datasets, the authors report that agents consistently overstate what the underlying documents warrant.
Unconfirmed: The paper does not name the seven systems or four datasets in the abstract.
- SmoothScore scores claims after clustering evidence by high-level aspects.
- The failure is loss of conflicting or uneven evidence in a single narrative.
- The study covers 7 systems and 4 datasets.
- Findings concern research-report synthesis rather than a deployed incident.
From the original article
1 min read at arxiv.orgDeep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing.
By Tiasa Singha Roy · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.