30
A Ranking Approach for Measuring Calibration
Researchers propose rankECE, a ranking-based calibration error measure with theoretical guarantees that outperforms binned ECE approximations.
The paper introduces rankECE, an alternative to Expected Calibration Error (ECE) that measures miscalibration by comparing points with neighboring predicted-probability values. It addresses the impossibility of estimating ECE with guaranteed accuracy in assumption-free settings. Theoretical guarantees and empirical results establish rankECE as a better proxy for ECE than the binned approximations most commonly used in practice.
22
55
30
55
57
30
60
60
55
42
57
45