Verifiable Social Reasoning for LLM Assistants
Researchers introduce Fuse, a multi-agent simulation benchmark evaluating LLM social reasoning, showing models are overly sensitive to biased user framing.
The paper introduces Fuse, a multi-agent simulation framework for evaluating LLM assistants' social reasoning in consultation settings. A target agent with a hidden motive interacts with an agent representing the user, who then consults the evaluated assistant, providing verifiable ground truth by construction; simulation faithfulness was validated through a human study with 24k annotations. Across 12 LLMs, user mediation compounded the difficulty of social reasoning, models showed systematic sensitivity to biased user framing, and longer conversations did not reliably improve performance. The framework and a 21k-example dataset are open-sourced.
- Multi-agent simulation provides verifiable ground truth for motive inference
- Evaluated 12 LLMs; validated with a 24k-annotation human study
- User mediation compounds inherent social-reasoning difficulty
- Models systematically sensitive to biased user framing
- Fuse and a 21k-example dataset are open-sourced
From the original article
1 min read at huggingface.co
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth.
By Amir Taubenfeld, Zorik Gekhman, Avigail Grinstein-Dabush · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.