Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training
Instrument-anchored LLM judges transfer counseling-quality ratings across domains and beat in-domain training.
The paper studies transfer of expert overall-impression scores for counseling quality across three German simulated-counseling corpora, totaling 195 expert-rated sessions after scale equating on one corpus. Leave-one-domain-out training reaches nested Spearman rho of 0.54, compared with at most 0.48 trained in-domain, and the paired gap stays at +0.12 when training-set sizes are matched. Session-level construct scores from small open-weight LLMs, largely taken from the experts' rating instruments, drive performance: an ensemble from three model families reaches 0.51 language-only, while a nonverbal block adds only +0.03. Repairing one corpus where 16% of diarized segments had the wrong speaker is worth +0.07.
- Study covers 195 expert-rated German simulated counseling sessions.
- Leave-one-domain-out Spearman rho is 0.54 versus at most 0.48 in-domain.
- Matched-size comparison still shows a paired gap of +0.12.
- Instrument-derived constructs lift one judge from 0.32 to 0.41.
- A three-family language ensemble reaches 0.51; nonverbal audio adds +0.03.
From the original article
1 min read at arxiv.orgAutomatic assessment of communication quality in dyadic counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. We study cross-domain transfer of expert overall-impression prediction across three German corpora of simulated counseling (two general-practice medical, one school-related parent-teacher; $n=195$ expert-rated sessions, one corpus after scale equating).
By Tobias Hallmen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.