Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective
Paper shows conformal prediction set-size reduction tracks information gain and relates to Shannon mutual information.
The paper gives a decision-theoretic foundation for treating conformal prediction set size as an uncertainty measure. It defines generalized information measures from set size and coverage, and shows Shannon mutual information has an exact integral representation in those measures. In standard classification, set-size reduction from extra information is bounded by calibration-dependent members of that family and obeys a data processing inequality, up to finite-sample error. Experiments on 11 classification settings find set-size reduction and Shannon mutual information can rank features differently in greedy selection.
- Links conformal set size to a decision-theoretic generalization of entropy.
- Shannon mutual information has an exact integral form via these measures.
- Set-size reduction obeys a data processing inequality up to error terms.
- On 11 tasks, set size and mutual information can rank features differently.
From the original article
1 min read at arxiv.orgConformal prediction is a popular tool for uncertainty quantification that outputs prediction sets with finite-sample coverage guarantees. While prediction set size is commonly used as a heuristic measure of uncertainty, the information-theoretic basis for this interpretation remains poorly understood.
By Kevin Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.