False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Safety-router benchmarks overstate routing gains by picking the best model on test data.
The paper shows LLM safety-routing evaluations that choose the best single-model comparator on evaluation data inflate routing benefits under distribution shift. On HELM Safety the selection cost is 0.003-0.030 of harm on random splits and 0.045-0.113 on held-out categories, and it rises seven- to ninefold on held-out AgentDojo suites. Scored against a baseline chosen without test labels, routing buys little, and a model-aware attacker lowers GPT-5.4's judged recognition of a late injection by 19.6 points.
- Test-set model selection biases safety routing scores under shift
- Held-out HELM Safety selection cost reaches 0.045-0.113 harm
- Honest scoring shows little routing gain on these benchmarks
- Attacker-aware prompts cut GPT-5.4 injection recognition by 19.6 points
Full article294 words · extracted from arxiv.org · click to collapse
Safety routers send each request to one of several models and are judged against the best single model. A major routing benchmark picks that comparator on the evaluation data. In the benchmark's own setting this is harmless, but under distribution shift it is not. On HELM Safety the selection cost is 0.003-0.030 of harm under random splits and 0.045-0.113 under held-out categories, comparable to the whole deficit attributed to routing, with its direction holding under either published judge alone. It rises seven- to ninefold on AgentDojo when suites are held out. Across seven safety corpora chosen by rules fixed in advance, three meet a registered interval test and four beat a later permutation null, and three of the four interval misses are corpora where some models have zero observed harm. Prior work proves the direction of this bias. We size it on harm and accuracy, show that it is larger under the held-out splits we measure, and bound it by optimism plus a shift-dependent regret. Scored honestly under shift, routing buys little on these benchmarks. In most pool cells the nested router serves the honest baseline's model, and on the nearly saturated AgentDojo corpus a perfect pre-dispatch router is worth at most two points of harm. We also find a model's expressed recognition of a late injection steerable. On held-out reruns an attacker who knows which model it faces lowers GPT-5.4's judged recognition by 19.6 points, confirmed by an independent label. In an offline counterfactual composition into a controller, the same attack raises or lowers estimated harm depending on the fallback model. Safety routing should be evaluated under shift, against a baseline chosen without the test labels, and recognition-based defences should be scored on harm against an attacker who chooses what the model sees.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.01535