A Living Benchmark for Information Retrieval from Electronic Health Records
Clinicians validate BRIE, an auto-generated EHR benchmark showing leading LLMs often omit clinically important facts.
Researchers present a framework that automatically generates question-answer pairs from longitudinal electronic health record notes. Nineteen clinicians validate the generator, producing BRIE, a continuously maintainable benchmark for clinical information retrieval. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, especially when answers require synthesis across multiple documents and encounters. Because generation is validated, BRIE can produce multiple clinician-varying answers and refresh content to limit benchmark leakage.
- Nineteen clinicians validate an automatic EHR question-answer generator named BRIE.
- Nine LLMs and five inference strategies were compared.
- Models often omit facts requiring synthesis across notes and encounters.
- Refreshable items and multiple answers address leakage and clinician variation.
Full article170 words · extracted from arxiv.org · click to collapse
Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.30205