Available Guardrails: Certifying Selective Prediction across ML Systems
Finite data often blocks certifying selective-prediction safety gates, and split planning recovers part of the lost coverage.
The paper treats selective prediction as a safety gate that must be certified at a target precision for each reporting unit, such as a tool, policy label, or patient subgroup. Exact binomial inversion and a dynamic program make the trade-off among safety, granularity, and served traffic computable. A truth-informed planner gains 0.157 mean coverage over support balancing, while a naive estimator recovers only 0.005; planning and selecting partitions on separate splits recovers 0.060, in 59 of 60 model effects. The same frontier appears for LLM tool-calling, content moderation, lesion classification, and recommendation.
- Selective predictors abstain unless a reporting unit is certified at target precision.
- Truth-informed planning gains 0.157 coverage; naive estimation recovers only 0.005.
- Split partition selection improves mean coverage by 0.060 in 59 of 60 effects.
- Reallocating the familywise error budget adds further certified coverage.
Full article263 words · extracted from arxiv.org · click to collapse
A selective predictor acts as a safety gate: it returns an output only when the prediction appears sufficiently trustworthy. Deployments increasingly require this reliability to be certified at a target precision for every reporting unit of interest, such as a tool, policy label, or patient subgroup. The main difficulty is often not whether a granted certificate is valid, but whether finite calibration data can produce one at all. As the gate becomes safer or more fine-grained, some units may receive too little evidence to certify. We make this notion of availability computable through classical exact-binomial inversion and formulate reporting-partition selection, under a fixed group order, as a dynamic program that exposes the trade-off among safety, granularity, and served traffic. The resulting frontier reveals a large population opportunity that finite-sample estimation nearly erases: a truth-informed planner gains $0.157$ mean coverage over support balancing, whereas a naive estimator recovers only $0.005$, making recovery from finite data the central challenge. Constructing candidate partitions on one planning split and selecting among them on another recovers part of this gap, improving mean coverage over support balancing by $0.060$, with the direction reproduced in $59$ of $60$ model effects across three intent-routing datasets and two architectures. A complementary validity-preserving lever, reallocating the familywise error budget across reporting units, recovers additional coverage both with population quantities and noisy estimates. The same frontier recurs, with predictor-specific ceilings, across LLM tool-calling, content moderation, lesion classification, and recommendation. Certified availability is therefore a plannable deployment resource that determines when a safety gate can be certified, at what granularity, and over how much traffic.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22048