Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
A protocol finds natural research-agent explanations do not yet show confirmed predictive credit on Tox21 and OpenML.
The paper defines predictive credit for explanations produced by research agents, comparing paired forecasts that share an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Across 336 prospective states, 12 Tox21 endpoints, and 24 OpenML tasks, the frozen v5 credit decision was inconclusive: the Tox21 ROC AUC interval-score harm test and the OpenML formation, equivalence, and repeatability rule were both unmet. Under DeepSeek V4 Pro, matched and donor cards cut secondary Tox21 drift by 64.5 and 59.1 percent, but a DeepSeek V4 Flash replay missed matched point-accuracy and interval-score equivalence. A researcher-authored mechanism control lowered point MAE by 2.60 percentage points, while natural-explanation credit remained unconfirmed.
- Measures whether research-agent explanations improve paired experimental forecasts.
- Covers 336 states, 12 Tox21 endpoints, and 24 OpenML tasks.
- Preregistered Tox21 and OpenML credit tests were unmet.
- A researcher-written mechanism control cut point MAE by 2.60 points.
- Natural-explanation predictive credit stayed unconfirmed.
Full article200 words · extracted from huggingface.co · click to collapse
Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet (D-M=-.0026, 95 percent interval [-.0174, .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.00314