Available Guardrails: Certifying Selective Prediction across ML Systems
Finite data often blocks certifying selective-prediction safety gates, and split planning recovers part of the lost coverage.
The paper treats selective prediction as a safety gate that must be certified at a target precision for each reporting unit, such as a tool, policy label, or patient subgroup. Exact binomial inversion and a dynamic program make the trade-off among safety, granularity, and served traffic computable. A truth-informed planner gains 0.157 mean coverage over support balancing, while a naive estimator recovers only 0.005; planning and selecting partitions on separate splits recovers 0.060, in 59 of 60 model effects. The same frontier appears for LLM tool-calling, content moderation, lesion classification, and recommendation.
- Selective predictors abstain unless a reporting unit is certified at target precision.
- Truth-informed planning gains 0.157 coverage; naive estimation recovers only 0.005.
- Split partition selection improves mean coverage by 0.060 in 59 of 60 effects.
- Reallocating the familywise error budget adds further certified coverage.
From the original article
1 min read at arxiv.orgA selective predictor acts as a safety gate: it returns an output only when the prediction appears sufficiently trustworthy. Deployments increasingly require this reliability to be certified at a target precision for every reporting unit of interest, such as a tool, policy label, or patient subgroup.
By Parivesh Priye · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.