InclusionMed Index scores frontier medical models from 17.5 to 49.1
Original title: InclusionMed: Benchmarking AI for Inclusive Healthcare
The first Atlas covers five benchmarks and three of nine role-capability cells, with rank changing by role and capability.
InclusionMed is an open medical AI evaluation initiative. Its Atlas organizes benchmarks across three healthcare roles and three AI capabilities. The first release selected four public benchmarks from 388 candidates and adds private InclusionMed-DxFiles, for five benchmarks covering three of nine role-capability cells. Frontier large language models scored 17.5 to 49.1 out of 100 on the InclusionMed Index. Rank varied by role or capability, and the ten highest-ranked models scored higher on patient communication than on clinical decision-making.
Unconfirmed: Index scores and cost comparisons are the authors' reported results, not an independent audit.
- Four public benchmarks were chosen from 388 candidates after audits and independent reproduction.
- Models with similar Index scores differed substantially in estimated inference cost.
- The Index also reports scores by healthcare role and by AI capability.
From the original article
1 min read at huggingface.coAdvancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, but benchmark coverage remains uneven. We introduce InclusionMed, an open medical AI evaluation initiative for inclusive healthcare. The InclusionMed Atlas organizes benchmarks across three healthcare roles and three AI capabilities.
By The InclusionMed Collaboration · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.