Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Paper proposes prediction-powered smoothing (PP-S/PP-TS) and a design-based cross-validation score to improve disaggregated AI evaluation estimates when labeled data is scarce.
The paper treats an AI evaluation set as a finite population and develops Bayesian prediction-powered smoothing (PP-S), plus a taxonomy extension (PP-TS) that borrows strength across domains. It derives an approximately unbiased design-based cross-validation score for selecting among direct and smoothed estimators. On a curated benchmark with verifiable grading and human-graded deployed agent traffic, the estimators improve point and interval accuracy with near-nominal coverage at the same sampling budget.
- Prediction-powered smoothing improves domain-level estimates where labels are sparse
- PP-TS extension borrows strength across a reporting taxonomy
- New design-based CV score matches an independent validation sample in estimator selection
- Demonstrated on verifiable benchmark grading and human-graded deployed agent traffic
Full article203 words · extracted from huggingface.co · click to collapse
Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20758