Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis
PRISM reports high AUROCs for emergency diagnosis, deterioration, ICU admission, and mortality.
PRISM is a retrieval-augmented multimodal language model for clinical prognosis that combines per-feature numerical tokenization, recency-weighted longitudinal ECGs, and a dual-modality population memory of mean-pooled similar encounters. On the 1,443-task MDS-ED benchmark with a 96-hour pre-arrival window, it reports AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. With benchmark-window inputs only, AUROC is 0.8895 versus 0.8937 for UniPACT and 0.8889 for MDS-ED. Supervised fine-tuning on external MC-MED reaches 0.8611 AUROC, above locally trained gradient boosting at 0.8227.
- PRISM fuses numerical tokens, recency-weighted ECGs, and population-memory embeddings.
- Reported AUROCs: 86.2% diagnosis, 93.1% deterioration, 91.8% ICU, 92.8% mortality.
- Benchmark-window AUROC was 0.8895, near UniPACT at 0.8937 and MDS-ED at 0.8889.
- Fine-tuning on external MC-MED reached 0.8611 versus 0.8227 for gradient boosting.
Full article219 words · extracted from arxiv.org · click to collapse
Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.13510