Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Paper proposes prediction-powered smoothing (PP-S/PP-TS) and a design-based cross-validation score to improve disaggregated AI evaluation estimates when labeled data is scarce.
The paper treats an AI evaluation set as a finite population and develops Bayesian prediction-powered smoothing (PP-S), plus a taxonomy extension (PP-TS) that borrows strength across domains. It derives an approximately unbiased design-based cross-validation score for selecting among direct and smoothed estimators. On a curated benchmark with verifiable grading and human-graded deployed agent traffic, the estimators improve point and interval accuracy with near-nominal coverage at the same sampling budget.
- Prediction-powered smoothing improves domain-level estimates where labels are sparse
- PP-TS extension borrows strength across a reporting taxonomy
- New design-based CV score matches an independent validation sample in estimator selection
- Demonstrated on verifiable benchmark grading and human-graded deployed agent traffic
From the original article
1 min read at huggingface.co
Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean.
By Sho Kawano, Zehang Richard Li, Paul A. Parker · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.