Verifiable Social Reasoning for LLM Assistants
Fuse, a multi-agent simulation with hidden motives, evaluates LLM social reasoning, revealing compounding difficulty from user mediation and bias sensitivity.
Fuse is a multi-agent simulation framework in which a target agent with a hidden motive interacts with other agents including one representing the user, who consults the evaluated assistant to infer the motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. Applied to 12 LLMs, it shows user mediation compounds social reasoning difficulty, models are systematically sensitive to biased user framing, models may need more details than humans, and longer conversations do not always improve performance. The framework and a 21k-example dataset are open-sourced.
- Hidden-motive multi-agent simulation yields verifiable ground truth for social reasoning
- Evaluated 12 LLMs; faithfulness validated with 24k human annotations
- LLMs show systematic sensitivity to biased user framing
- Longer conversations do not always improve prediction accuracy
- Framework and 21k-example dataset open-sourced
From the original article
1 min read at arxiv.orgLLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth.
By Amir Taubenfeld · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.