Small LLMs fail complement consistency in agent opinion simulations
Original title: Do "True" and "False" Add Up to One? Complement Consistency in LLM-Agent Opinion Simulations
Paired true and false reports from Qwen3-4B, Llama-3.2-3B and Gemma-3-4B do not sum to one, and complementary populations diverge within five rounds.
The paper tests whether rewriting a claim as its complement and replacing each probability p with 1−p yields complementary answers from LLM agents. On 2,304 states from a running simulation, Qwen3-4B paired reports averaged 1.18, and reports on 48 AVeriTeC claims also exceeded one. A 24-world benchmark showed the same failure in Qwen3-4B, Llama-3.2-3B and Gemma-3-4B. The gap was two to ten times larger than a same-meaning paraphrase and persisted under resampling and different peer sizes. Populations started from complementary states diverged positively after five rounds for every model.
- Models tested were Qwen3-4B, Llama-3.2-3B and Gemma-3-4B.
- Qwen paired reports summed to 1.18 on average over 2,304 agent states.
- The discrepancy was two to ten times a same-meaning paraphrase effect.
- Averaging an answer with its complement's paired answer restored consistency by construction.
- That average beat the better original wording at the final round for only one model.
- AI models
- Qwen3-4BLlama-3.2-3BGemma-3-4B
From the original article
1 min read at arxiv.orgIn LLM-agent opinion simulations, each agent reports the probability that a claim is true after seeing its own previous probability and the probabilities reported by its neighbors, and that number becomes its state in the next round.
By Anouar Imel · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.