RACER: Role-Aligned Competence Estimation for Human-AI Routing
RACER estimates an unseen expert's instance-level competence to route queries between a model and a human.
The paper proposes RACER, Role-Aligned Competence Estimation for Routing, for learning to defer to unseen human experts. It estimates the posterior-predictive probability that an expert is correct under candidate class roles, then combines that with the model posterior using kernel-pooling estimators that exclude absolute class-identity channels. The authors prove class-relabelling invariance and a plug-in regret bound. On synthetic PathMNIST and CIFAR-100 splits and on the VinDr-CXR and CheXpert radiologist benchmarks, RACER is competitive or best in budget-swept deferral.
- Estimates expert correctness with role-relative posterior probabilities
- Blocks absolute class-identity shortcuts that bias prior deferral encoders
- Best aggregate score on a CIFAR-100 unseen-expert split
- Competitive budget-swept deferral on VinDr-CXR and CheXpert
From the original article
1 min read at arxiv.orgLearning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this problem to unseen experts using a small context set of expert behavior. Neural context encoders such as L2D-Pop can be query-dependent, but may learn routing shortcuts tied to absolute class coordinates.
By Joshua Strong · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.