Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
Evaluation of twelve LLMs on 222 clinical questions shows verbatim quotes rarely substantiate claims; claude-opus-5 fully substantiates only 37.1%.
The authors build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring citation attachment, verbatim quote production, and claim substantiation. Most models attach verbatim quotes to over 90% of claims from prompting alone, though lightweight models like claude-haiku-4.5 struggle. Quotes frequently fail to substantiate claims: claude-opus-5 quotes 98.0% of claims but fully substantiates only 37.1%, exposing a capability gap for verifiable clinical QA.
- Twelve LLMs tested on 222 synthetic clinical questions across four practice guidelines
- Most models attach verbatim quotes to over 90% of claims from prompting alone
- claude-opus-5 produces quotes for 98.0% of claims but fully substantiates only 37.1%
- Results quantify the gap in building verifiable clinical QA systems
From the original article
1 min read at arxiv.orgLarge language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but these often point to broad texts, leaving time-pressed clinicians unable to verify them efficiently.
By Jiashuo Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.