Correlated reasoning splits self-consistency certificates into two nulls
Original title: Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
On AIME, a single-unit rule crossed on 24% of prompts at nominal epsilon 0.05; a prompt-level test did not.
A paper argues that language-model self-consistency certificates assuming independent rollouts fail when reasoning streams are correlated. Correlation splits a realized null about the drawn belief from a population null about average model behavior, and the standard betting e-process remains anytime-valid only for the realized null. On Claude Haiku 4.5, GSM8K shows a dormant gap, while AIME shows the single-unit rule crossing on 24% of prompts at nominal epsilon 0.05. A prompt-level e-process restores population validity, optimally with one rollout per prompt under a pure rollout budget.
Unconfirmed: Reported rho, theta, and crossing rates are estimates from the authors' simulations and rollouts.
- Claude Haiku 4.5 rollouts are used in the experiments.
- GSM8K estimates are rho 0.435 and theta +0.96.
- AIME estimates are rho 0.480 and theta -0.15.
- Within-unit effective sample size saturates at 1/rho.
- The prompt-level procedure records a crossing rate of 0.00 on AIME.
From the original article
1 min read at arxiv.orgSelf-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent.
By Terence D. Agbeyegbe · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.