AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
AI Consilium shows model choice beats format for clinical LLM second opinions on NEJM cases.
The authors built a multidimensional clinical benchmark from 300 diagnostically resolved NEJM-MedQA cases, with 110 used for physician reference ratings and judge calibration and 190 held out. Eight single-model pipelines produced role-specific opinions and consilium reports under free-form and structured reasoning, and a physician-calibrated LLM judge scored 3,040 final reports. Model identity separated quality more than format; structured reports scored higher in seven of eight pipelines, with management at 2.740 versus 2.436. GPT-5.6-Sol led every component and reached consensus in the fewest review rounds.
- Uses 300 resolved NEJM-MedQA cases; a judge scored 3,040 reports.
- Model identity separated quality more strongly than response format.
- Structured reports beat free-form in seven of eight pipelines on management.
- GPT-5.6-Sol led every component and reached consensus in fewest rounds.
Full article187 words · extracted from arxiv.org · click to collapse
Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark using 300 diagnostically resolved NEJM--MedQA cases: 110 for physician reference ratings and Judge calibration and 190 for held-out evaluation. Eight single-model pipelines generated independent role-specific opinions and interactive-consilium reports under free-form and structured clinical-reasoning conditions. A physician-calibrated LLM Judge evaluated 3,040 final reports and the corresponding structured primary-role outputs. Model identity separated report quality more strongly than response format. Structured final reports had higher overall mean scores in seven of eight model pipelines, with their clearest advantage in management (2.740 vs. 2.436). Compared with the structured pre-consilium primary-role opinion, the complete structured synthesis had higher pooled scores across all five shared criteria, with the largest increases in problem prioritization, factual grounding, and reasoning coherence. GPT-5.6-Sol led every evaluated component and reached consensus in the fewest review rounds. These findings show that the benchmark can distinguish model capability, the contribution of multidisciplinary synthesis, criterion-specific effects of response structure, deliberation dynamics, and inference requirements within one common-case evaluation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.22298