NPC-RubricBench shows LLM cancer-care errors reverse by phase
Original title: Patient-specific rubric-based evaluation of large language models across longitudinal cancer care
Across 405 patients, strong diagnosis did not carry into later care, and prompt style traded quality against faithfulness.
NPC-RubricBench evaluates multistage clinical reasoning for nasopharyngeal carcinoma using longitudinal records from 405 patients at two cancer centers. It has 941 patient-phase units across six phases and 17,688 patient-specific criteria over five dimensions, scored only on decision-time evidence. Among 30 general-purpose and medical LLMs, high diagnostic performance did not improve downstream care. Explicit decision errors dominated staging at 48.2%, while omissions dominated initial work-up and follow-up at 33.4% and 30.4%.
- Medically adapted models had higher omission rates than general-purpose models in the set tested.
- Broad open-ended prompts lowered Decision Quality across all 30 models.
- Checklist-like prompts raised Decision Quality but cut Evidence Faithfulness and Safety Adherence.
- Each assessment is restricted to clinical evidence available at decision time.
- Products
- NPC-RubricBench
From the original article
1 min read at arxiv.orgLarge language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway.
By Wei-Wei Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.