A Ranking Approach for Measuring Calibration
Researchers propose rankECE, a ranking-based calibration error measure with theoretical guarantees that outperforms binned ECE approximations.
The paper introduces rankECE, an alternative to Expected Calibration Error (ECE) that measures miscalibration by comparing points with neighboring predicted-probability values. It addresses the impossibility of estimating ECE with guaranteed accuracy in assumption-free settings. Theoretical guarantees and empirical results establish rankECE as a better proxy for ECE than the binned approximations most commonly used in practice.
- ECE cannot be estimated with guaranteed accuracy without assumptions
- rankECE compares points with neighboring values of predicted probability
- rankECE outperforms binned ECE approximations theoretically and empirically
Full article139 words · extracted from arxiv.org · click to collapse
When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that $Y=1$) exactly matches the forecasted probability $f(X)$. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability $f(X)$. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.13100