Open Language Models Perform on Spanish Medical Residency Exam
Original title: Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination
Larger open models rank in the top 600-3000 candidates on the Spanish MIR exam but abstention does not improve outcomes and images are not always used.
Accuracy on medical exams is evidence for clinical use of language models. Four open models of 8-27 billion parameters were evaluated on 1,552 Spanish MIR questions. The three larger models would rank among the top 600-3000 candidates with 41-43 specialties open. Abstention never helps as even low-confidence answers are correct more than 25% of the time. Medical fine-tuning reduced overconfidence without improving separation of correct answers and images were not always used.
- Three larger models rank top 600-3000
- 41-43 specialties open
- Abstention does not improve accuracy
- Fine-tuning reduced overconfidence without better separation
- Images left accuracy unchanged in 13-38% cases
From the original article
1 min read at arxiv.orgAccuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the…
By Miguel Fernandez-de-Retana · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.