Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Signed Rescue Routing improves LLM cascade efficiency by predicting when a larger model actually corrects a smaller one rather than uncertainty.
Signed Rescue Routing (SRR) is a budgeted cascade method that separately predicts rescues and regressions when escalating from a small to a large model, ranking requests by their signed difference. The authors prove this signed conditional gain is Bayes-optimal under a fixed escalation budget and add only a lightweight two-head router needing small-model output statistics at deployment. Evaluation with Qwen3-4B and Qwen3-8B on MMLU, HellaSwag, and ARC-Challenge shows better accuracy-compute tradeoffs than entropy routing and learned error predictors.
- Predicts rescue vs. harm separately instead of small-model uncertainty
- Signed conditional gain shown Bayes-optimal under fixed escalation budget
- Deployment requires only small-model output statistics plus a two-head router
- Tested with Qwen3-4B and Qwen3-8B on MMLU, HellaSwag, ARC-Challenge
- Products
- Signed Rescue Routing
From the original article
1 min read at arxiv.orgLarge language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong.
By Zheyuan Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.