PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
PerSo benchmarks LLMs' cognitive and behavioral social intelligence on real Reddit persuasion dialogues.
PerSo is a benchmark of LLM social intelligence that jointly measures inferring mental states and choosing social actions. It uses real persuasion dialogues from Reddit's ChangeMyView community, with tasks built by an LLM-in-the-loop pipeline and checked by human annotation. Across diverse models, performance generally scales with size but differs by dimension. The strongest LLM slightly exceeds the best observed human on the behavioral task, while the best human remains ahead on the cognitive task.
- PerSo tests cognitive and behavioral social intelligence together.
- Tasks use real Reddit ChangeMyView persuasion dialogues.
- An LLM-in-the-loop pipeline builds tasks validated by humans.
- The best model slightly beats humans on behavior but trails on cognition.
- Scores generally rise with model size and vary by dimension.
From the original article
1 min read at arxiv.orgUnderstanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals.
By Ali Zamani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.