MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations
Original title: MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
Models score 33.3 to 100% on calculation tasks with 1,200 seeded items; five models detect at least 96.7% of planted errors with at most 4% false alarms.
MedPhysBench tests language models on radiation therapy calculations and error detection using 10 generators for medical physics tasks including TG-51 calibration. Reference answers come from code implementing governing protocols and a grader applies tolerance bands. On a held-out test split of 1,200 items, 12 models scored up to 100%. Five models detect at least 96.7% of mostly large planted errors with low false alarms. Claude models show specific behaviors in inverse-square correction.
- Ten seeded generators for calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection.
- Reference answers from code implementing governing protocols.
- Grader converts units and applies tolerance bands.
- Held-out test split of 1,200 items (half for two models).
- Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms.
- AI models
- Claude Opus 5.5Claude Sonnet 5.5
From the original article
1 min read at arxiv.orgMedical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it. MedPhysBench has ten seeded generators of calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, up to clinical tasks such as a TG-51 calibration worksheet.
By Omar Nusrat · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.