ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
ModaLens image-swap audit shows report availability cuts MedGemma-27B image sensitivity on MIMIC-CXR from 20.94% to 4.26% answer changes.
ModaLens is a paired image-swap audit measuring how report availability affects image sensitivity in report-conditioned medical VLMs. On MedGemma-27B across 3,199 paired MIMIC-CXR cases from 293 patients (14 questions per case), generated answers changed on 4.26% of image-swap trials with the report versus 20.94% without it, a 16.7-point paired difference (95% CI 15.6-17.7). The original prompt with a lowercase first-token readout gave 4.70% versus 17.07%, and the direction replicated in two further model lineages. Labels derived from reports limit conclusions about visual correctness; code, prompts, and run records are publicly released.
- Report availability reduced image-swap answer changes from 20.94% to 4.26%.
- Audit spans 3,199 paired MIMIC-CXR cases from 293 patients, 14 questions each.
- Effect replicates in two additional model lineages under alternative readouts.
- Report-derived labels limit claims about visual correctness of answers.
- Code, exact prompts, and per-number run records released on GitHub.
From the original article
1 min read at huggingface.co
A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image.
By Sebastián Andrés Cajas Ordóñez, Maximin Lange, Quang Bui · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.