A Ranking Approach for Measuring Calibration
Researchers propose rankECE, a ranking-based calibration error measure with theoretical guarantees that outperforms binned ECE approximations.
The paper introduces rankECE, an alternative to Expected Calibration Error (ECE) that measures miscalibration by comparing points with neighboring predicted-probability values. It addresses the impossibility of estimating ECE with guaranteed accuracy in assumption-free settings. Theoretical guarantees and empirical results establish rankECE as a better proxy for ECE than the binned approximations most commonly used in practice.
- ECE cannot be estimated with guaranteed accuracy without assumptions
- rankECE compares points with neighboring values of predicted probability
- rankECE outperforms binned ECE approximations theoretically and empirically
From the original article
1 min read at arxiv.orgWhen providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i. e. , the probability that $Y=1$) exactly matches the forecasted probability $f(X)$.
By Anirban Chatterjee · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.