Before You Poll with LLMs: A Deliberative Diagnostic Framework
Deliberative diagnostic shows all five tested frontier LLMs misrepresent human belief shifts after arguments, with GPT-5.1 reversing on outgroup questions.
The Deliberative Polling Diagnostic Framework compares human and LLM persona belief shifts after identical informational interventions, using data from America in One Room (526 personas, 72 questions). All five frontier models tested failed uniquely: GPT-5.1 exhibited partisan reversal (80% on outgroup vs 26% on policy questions), Gemini 2.0 Flash, Claude Sonnet 4.5 and Llama 3.3 70B overshot at 5-7x human magnitude, and DeepSeek V3 showed near-zero change (rigidity). The authors term the underlying signature 'self-sycophancy', conformity to the model's internal persona stereotype rather than reasoning from provided information.
- All five frontier models failed dynamic belief-update fidelity in unique ways
- GPT-5.1 reversed on outgroup questions (80%) despite balanced information
- Gemini 2.0 Flash, Claude Sonnet 4.5, Llama 3.3 70B overshot shifts at 5-7x human magnitude
- DeepSeek V3 showed near-zero belief change (rigidity)
- Failures are identity-specific, termed 'self-sycophancy'
From the original article
1 min read at arxiv.orgCan LLMs reason through new information like humans, or do they merely retrieve cached opinions? This is critical for silicon sampling, where LLM personas simulate public opinion at scale. Current evaluations test only whether personas hold the right opinions -- a static snapshot.
By Ahmed Wali · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.