Evidence-aware retrieval scores reported for a breast-lesion RAG system
Original title: How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions
The selected hybrid retriever reached 0.94 on Recall@3, R-Precision, MRR@3, and nDCG@3 for 17 of 18 queries.
Researchers describe an evidence-aware retrieval protocol for a clinical RAG system over breast-lesion management literature. An 18-document corpus and an 18-query benchmark tested dense embeddings, BM25, and cross-encoder reranking across chunk sizes and overlaps. The chosen hybrid setup, 384-token chunks with 20% overlap, placed accepted evidence in the top three results for 17 of 18 queries. Expanding annotations raised recognized evidence without changing rankings, and the authors say the small set still needs independent validation.
Unconfirmed: The authors say the small development set and retrieval-informed curation still require independent validation, including radiologist review.
- Chunk sizes of 128 to 1024 tokens and overlaps of 0 to 20% were tested.
- Curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51.
- Generation was scored on a 512-token index at K=1 to 5.
- deepseek-v4-pro served as the LLM judge for DeepEval claim-level metrics.
- Cross-encoder reranking produced the highest retrieval scores.
From the original article
1 min read at arxiv.orgPurpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used.
By Miguel Mochizuki Silva · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.