RACER: Role-Aligned Competence Estimation for Human-AI Routing
RACER estimates an unseen expert's instance-level competence to route queries between a model and a human.
The paper proposes RACER, Role-Aligned Competence Estimation for Routing, for learning to defer to unseen human experts. It estimates the posterior-predictive probability that an expert is correct under candidate class roles, then combines that with the model posterior using kernel-pooling estimators that exclude absolute class-identity channels. The authors prove class-relabelling invariance and a plug-in regret bound. On synthetic PathMNIST and CIFAR-100 splits and on the VinDr-CXR and CheXpert radiologist benchmarks, RACER is competitive or best in budget-swept deferral.
- Estimates expert correctness with role-relative posterior probabilities
- Blocks absolute class-identity shortcuts that bias prior deferral encoders
- Best aggregate score on a CIFAR-100 unseen-expert split
- Competitive budget-swept deferral on VinDr-CXR and CheXpert
Full article236 words · extracted from arxiv.org · click to collapse
Learning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this problem to unseen experts using a small context set of expert behavior. Neural context encoders such as L2D-Pop can be query-dependent, but may learn routing shortcuts tied to absolute class coordinates. Identity-Free Deferral (IFD) removes such shortcuts through role-indexed classwise competence profiles, but its estimates are constant within each class and cannot capture instance-level expert specialization. We propose RACER---Role-Aligned Competence Estimation for Routing---a role-relative framework for estimating an unseen expert's competence from context. RACER estimates the posterior-predictive probability that the expert is correct on a query under each candidate class role, then combines these estimates with the model posterior to obtain the Bayes-relevant expert-correctness probability. Nonparametric and neural kernel-pooling estimators use candidate-role relations, shared aggregation, and symmetric summaries, excluding absolute class-identity channels. We prove coherent class-relabelling invariance, derive a Bayes-aligned deferral surrogate, and give a plug-in regret bound relating routing regret to classifier and competence-estimation error. On controlled synthetic benchmarks, including a PathMNIST histopathology context-scaling study with simulated experts, RACER benefits from additional context under hidden subtype dependence and gives the strongest aggregate performance on a separately sampled unseen-expert split in the CIFAR-100 synthetic experiments. On the radiologist and human--AI chest-radiography benchmarks (VinDr-CXR and CheXpert), the RACER family is competitive or best in budget-swept deferral, with calibration results varying across metrics and datasets.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.21953