Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention
RegimeAbstain uses structural retrieval features to abstain on predictable multi-hop failures, cutting confident wrong answers.
The paper argues multi-hop retrieval failures cluster in structurally predictable regimes and introduces RegimeAbstain, which computes a Retrieval Confidence Score from up to nine query and approximate-nearest-neighbor features without an extra LLM call. Across MuSiQue, 2WikiMultiHopQA, and HoVer, and both LLM-judge and dense-only retrieval, RCS is best or co-best on AUC-AC in five failure regimes with baseline CWAR from 14.5% to 62.1%. On MuSiQue with an LLM judge, abstention cuts CWAR from 39.5% to 20.6% at 50% coverage, with ECE of 0.035.
- RegimeAbstain scores retrieval confidence from up to nine ANN features without extra LLM calls.
- Evaluated on MuSiQue, 2WikiMultiHopQA, and HoVer under LLM-judge and dense-only retrieval.
- On MuSiQue, CWAR falls from 39.5% to 20.6% at 50% coverage.
- A MuSiQue-trained model transfers to 2WikiMultiHopQA with only 0.5 percentage-point AUC loss.
Full article230 words · extracted from arxiv.org · click to collapse
Multi-hop retrieval failures are not uniformly distributed across queries: they cluster in structurally predictable subpopulations. We prove two results formalizing this structure. First (CWAR Reducibility): confident-failure reduction is achievable if and only if retrieval features carry mutual information about success, a condition satisfied by LLM-judge pipelines but substantially weaker in dense-only settings, explaining the AUC-AC gap between regimes. Second (Feature Regime Complementarity): no single ANN score feature achieves best predictive performance across all failure regimes; the dominant feature differs between datasets (query length on MuSiQue, hop-1 concentration on HoVer), and a constructive witness pair shows each is necessary in one regime and non-contributory in the other. We instantiate these principles in RegimeAbstain, which computes a Retrieval Confidence Score (RCS), a logistic function of up to nine query-ANN structural features, all available without any additional LLM call, and uses it to implement a calibrated abstention policy. We define the Confident-Wrong-Answer Rate (CWAR) metric and evaluate across three multi-hop benchmarks (MuSiQue, 2WikiMultiHopQA, HoVer) and two retrieval architectures (LLM-judge and dense-only), covering five failure regimes with CWAR from 14.5% to 62.1%. RCS achieves best or co-best AUC-AC in all five conditions against eight confidence baselines. On MuSiQue (LLM-judge), RCS reduces CWAR from 39.5% to 20.6% at 50% coverage (47.8% relative reduction), with ECE=0.035. A model trained on MuSiQue transfers to 2WikiMultiHopQA with only -0.5pp AUC loss, confirming the domain-agnostic structure of regime features.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22056