MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
MindEval2 benchmarks 18 language models on mental-health support using clinician rubrics and realistic simulated patients.
MindEval2 evaluates language models used for mental-health support. Mental Health Quality Rubrics, developed with PhD-level psychologists, score responses against explicit operational criteria instead of opaque Likert scales. MindSim simulates harder patients with designed profiles, tools, and memory updated between sessions so interactions can be longitudinal. Across 18 proprietary and open-weight models, frontier systems improved surface quality and boundaries, but weak guidance calibration remained the dominant failure mode.
- MHQR rubrics were developed with PhD-level psychologists.
- MindSim uses profiles, tools, and cross-session memory.
- Eighteen proprietary and open-weight models were evaluated.
- Guidance calibration, not surface quality, caused most remaining failures.
From the original article
1 min read at arxiv.orgWith people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realistic setting by simulating multi-turn interactions with synthetic patients and evaluating them with an LLM-as-a-judge.
By John Mendonça · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.