ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
ProbabilityBench gives exact class probabilities so researchers can measure model calibration and Bayes-error estimates.
ProbabilityBench is a benchmark with exact instance-level class probabilities across simulated peer review, covert information transfer, and credit-risk domains. The authors define each data-generating process, compute exact class probabilities and Bayes error, and show that rendering cases as natural-language memos preserves those probabilities. Across 16 models, most systems on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. The benchmark also tests calibration metrics and Bayes-error estimators against known ground truth using KL divergence.
- ProbabilityBench supplies exact instance-level class probabilities in three simulated text domains.
- Natural-language rendering is proven to preserve the underlying generative probabilities.
- Across 16 models, decision models such as Jev and Perplexity Decider lead the cost-quality frontier.
Full article144 words · extracted from huggingface.co · click to collapse
Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.17016