ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
ModaLens image-swap audit shows report availability cuts MedGemma-27B image sensitivity on MIMIC-CXR from 20.94% to 4.26% answer changes.
ModaLens is a paired image-swap audit measuring how report availability affects image sensitivity in report-conditioned medical VLMs. On MedGemma-27B across 3,199 paired MIMIC-CXR cases from 293 patients (14 questions per case), generated answers changed on 4.26% of image-swap trials with the report versus 20.94% without it, a 16.7-point paired difference (95% CI 15.6-17.7). The original prompt with a lowercase first-token readout gave 4.70% versus 17.07%, and the direction replicated in two further model lineages. Labels derived from reports limit conclusions about visual correctness; code, prompts, and run records are publicly released.
- Report availability reduced image-swap answer changes from 20.94% to 4.26%.
- Audit spans 3,199 paired MIMIC-CXR cases from 293 patients, 14 questions each.
- Effect replicates in two additional model lineages under alternative readouts.
- Report-derived labels limit claims about visual correctness of answers.
- Code, exact prompts, and per-number run records released on GitHub.
Full article179 words · extracted from huggingface.co · click to collapse
A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens, a paired image-swap audit, measures how report availability changes image sensitivity: MedGemma-27B on 3,199 paired MIMIC-CXR cases from 293 patients, all 14 questions per case (13 finding-specific and one composite), each image replaced by one from another study, usually of the same patient, with question and report fixed. Under an explicit answer instruction, the model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it, a paired increase of 16.7 points (patient-clustered 95 percent CI 15.6 to 17.7), so report availability reduces image-swap sensitivity under this protocol; the original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change. The labels are derived from reports, which limits conclusions about visual correctness; the direction replicates in two further model lineages. Code, the exact prompts and a run record for every number are at https://github.com/criticaldata/MODALENS.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.15635