Competence-Gated Pooling of Language Models and Priors for Event Forecasting
Paper proposes a competence gate pooling language model forecasts with external priors, improving Brier score from 0.0771 to 0.0732 across 2,357 binary questions.
The paper defines a language model's relative competence as its marginal value beyond an available external forecast, and derives conditions under Brier loss where model disagreement improves that forecast. A competence gate estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, it improves the external baseline from 0.0771 to 0.0732 Brier and beats global forecast combinations, though it defers to the market on ForecastBench. Across four Qwen models, verbal confidence failed to identify when the model outperformed the external forecast, while outcome-estimated competence supported better abstention.
- Competence gate pools LM forecasts with external priors by measured marginal value
- Improves Brier score from 0.0771 to 0.0732 on 2,357 resolved binary questions
- Gains hold under leakage controls and against a leakage-safe time-series prior
- Qwen verbal confidence unreliable for abstention versus outcome-estimated competence
Full article221 words · extracted from huggingface.co · click to collapse
In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence, defined as the model's marginal value beyond the available external forecast. Under Brier loss, we characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. We then introduce a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, the gate improves the main external baseline from 0.0771 to 0.0732 Brier and significantly outperforms global forecast combinations. The gain remains significant under leakage controls and against a leakage-safe time-series prior on the pooled structured set, with separate evidence on FRED. In contrast, the gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market. Across four Qwen models, verbal confidence does not reliably identify when the model outperforms the external forecast, while outcome-estimated competence supports better abstention decisions. These results provide a practical approach for selective model use based on measured marginal value.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.12101