Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models
Six frontier models play a two-agent log(N)-Questions game; Claude Opus 5 lags with 28/68 wins while the top five are near-tied.
The study evaluates six frontier models on a two-agent game where a questioner must identify one of N Wikipedia lead paragraphs in exactly log2 N yes/no questions, run over 408 games at $363 total API cost. Claude Opus 5 wins 28 of 68 games versus 45-56 for GLM-5.3, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, and Kimi K3. Pooled top-five win rates decline with set size (r=-0.973) and fit win = p^(log2 N) with per-round reliability p=0.928, and information per question correlates with win rate at r=+0.88.
- Losses split roughly equally between answer errors and discrimination failures
- Only two models extract a full bit per question by partitioning on document titles
- Reasoning-token expenditure varies 4.5x across models with little relation to success
- 32 of 34 inspected unanimous errors were 'No' answers on first-sentence facts
From the original article
1 min read at arxiv.orgWe evaluate six frontier language models on the two-agent $\log(N)$-Questions game. A questioner sees $N$ Wikipedia lead paragraphs and must identify a secretly chosen target using exactly $\log_2 N$ yes/no questions. An answerer sees only the target and the question, and replies with one word.
By Peter Potash · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.