Audit finds 41 models change mental health advice by who asks
Original title: Curbside Chatbots: Auditing the Mental Health Interventions Large Language Models Recommend to Clinicians, Caregivers, and Patients
Across 19 vignettes and 35,055 responses, models often gave clinicians consultation-style answers and lay users simpler, hedged advice.
Researchers audited what 41 language models recommend when the same clinical information is presented by a clinician, a caregiver, or the person themselves. They used 19 case vignettes and five questions, collecting 35,055 free-text responses and extracting named interventions with a validated judge. Twenty-nine models also rated full option lists on a seven-point Likert scale. Models largely recommended options they rated well, but behavior differed by footing, and specific intervention prompts more often drew options rated below zero, usually with conditions.
Unconfirmed: The authors say the instrument, code, and corpus will be released on publication and are not yet public.
- The audit covered 41 language models and 19 case vignettes.
- Researchers collected 35,055 free-text responses to five questions.
- Twenty-nine models rated option lists on a seven-point Likert scale.
- Clinicians more often received consultation-style responses than lay users.
- The instrument, code, and corpus will be released on publication.
From the original article
1 min read at arxiv.orgUsers are asking language models mental health questions and are receiving answers. We audited what 41 language models recommend to users when the same clinical information reaches them from three footings: a clinician describing a patient, a caregiver describing a loved one, or a person describing themselves.
By Matthew Flathers · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.