Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Researchers separate LLM debate collapses from corrections across 6,925 MMLU-Pro multi-agent debates.
Researchers introduce an auditable protocol for homogeneous multi-agent LLM debate on multiple-choice questions, separating harmful collapses from useful corrections. Across 6,925 MMLU-Pro debates, the ledger records 253 collapses. A leave-one-model-out probe-gated freeze prevented 29 collapses but lost 108 corrections under equal weights. A compact eight-probe pre-debate screen showed a high unadjusted association with conditional-collapse risk (Spearman rho 0.893), though the authors do not treat it as calibrated prediction. Replay schemas, audits, and zero-API rebuild scripts are released.
- Protocol records collapse, correction, onset, and signed utility
- 253 collapses identified across 6,925 MMLU-Pro debates
- A freeze prevented 29 collapses but lost 108 corrections
- An 8-probe screen correlated with collapse risk but is not calibrated
- Many collapses localized to the first debate round
Full article215 words · extracted from huggingface.co · click to collapse
Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On 6,925 MMLU-Pro debates, the protocol identifies 253 collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents 29 collapses but loses 108 corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate 8-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high (G=7, Spearman rho=0.893, exact two-sided p=0.0123), but initial-majority accuracy is a close comparator (rho=0.821; family partial rho=0.767, p=0.0877), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model-scaffold rows can be compared under the same denominators and signed utility ledger.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.35279