Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs
Clinical LLM tests find accuracy outruns counterfactual validity, with frequent answer changes under demographic prefixes.
Six LLMs were evaluated on 150 MedQA USMLE questions using counterfactual-validity and demographic-prefix tests. Of 900 counterfactual attempts, 228 (25.3%) were valid. Across 5,400 demographic comparisons, 1,097 answers changed (20.3%), and automated judging raised 3,128 stereotype flags. MedGemma 27B led with 87.1% accuracy, 63.3% counterfactual validity, a 16.0% answer-change rate, and mean explanation demographic dissonance of 0.169, while OpenBioLLM and GLM ranked worst on instability and stereotype flags. All judgments were automated without clinician validation.
- Only 25.3 percent of 900 counterfactual attempts were judged valid.
- Demographic prefixes changed 20.3 percent of 5,400 answer comparisons.
- MedGemma 27B reached 87.1 percent accuracy but 63.3 percent counterfactual validity.
- Automated judging raised 3,128 stereotype flags without clinician validation.
Full article221 words · extracted from arxiv.org · click to collapse
Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustness. We evaluated six LLMs on 150 MedQA USMLE questions using two automated perturbation tests to assess their performance. The counterfactual validity (CFV) test asked each model to make a minimal, plausible clinical change that would make a different answer correct. The demographic robustness test added six demographic prefixes to the same vignette and compared the answers and explanations with a no demographic baseline. Of the 900 CFV attempts, 228 (25.3 %) were valid and 672 were invalid. Across 5,400 demographic comparisons, 1,097 answers were changed (20.3%). Automated judging identified 3,128 stereotype evidence flags, including 1,932 in the broad Other category. MedGemma 27B achieved the highest accuracy (87.1%) and CFV (63.3%), lowest answer change rate (16.0%), and low mean Explanation Demographic Dissonance (EDD) score (0.169). However, its accuracy still exceeded its CFV, indicating that correct answers do not guarantee reliable performance on the counterfactual validity task. OpenBioLLM had the highest answer change rate and EDD, whereas GLM had the highest stereotype flag rate. These findings show that accuracy, CFV, answer stability, EDD, and stereotype evidence capture different evaluation aspects. Because all judgments were automated and no clinician validation was available, the results support safety screening but do not establish clinical deployability of the model.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.32807