A Living Benchmark for Information Retrieval from Electronic Health Records
Clinicians validate BRIE, an auto-generated EHR benchmark showing leading LLMs often omit clinically important facts.
Researchers present a framework that automatically generates question-answer pairs from longitudinal electronic health record notes. Nineteen clinicians validate the generator, producing BRIE, a continuously maintainable benchmark for clinical information retrieval. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, especially when answers require synthesis across multiple documents and encounters. Because generation is validated, BRIE can produce multiple clinician-varying answers and refresh content to limit benchmark leakage.
- Nineteen clinicians validate an automatic EHR question-answer generator named BRIE.
- Nine LLMs and five inference strategies were compared.
- Models often omit facts requiring synthesis across notes and encounters.
- Refreshable items and multiple answers address leakage and clinician variation.
From the original article
1 min read at arxiv.orgLarge language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements.
By Jordan L. Cahoon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.