Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention
RegimeAbstain uses structural retrieval features to abstain on predictable multi-hop failures, cutting confident wrong answers.
The paper argues multi-hop retrieval failures cluster in structurally predictable regimes and introduces RegimeAbstain, which computes a Retrieval Confidence Score from up to nine query and approximate-nearest-neighbor features without an extra LLM call. Across MuSiQue, 2WikiMultiHopQA, and HoVer, and both LLM-judge and dense-only retrieval, RCS is best or co-best on AUC-AC in five failure regimes with baseline CWAR from 14.5% to 62.1%. On MuSiQue with an LLM judge, abstention cuts CWAR from 39.5% to 20.6% at 50% coverage, with ECE of 0.035.
- RegimeAbstain scores retrieval confidence from up to nine ANN features without extra LLM calls.
- Evaluated on MuSiQue, 2WikiMultiHopQA, and HoVer under LLM-judge and dense-only retrieval.
- On MuSiQue, CWAR falls from 39.5% to 20.6% at 50% coverage.
- A MuSiQue-trained model transfers to 2WikiMultiHopQA with only 0.5 percentage-point AUC loss.
From the original article
1 min read at arxiv.orgMulti-hop retrieval failures are not uniformly distributed across queries: they cluster in structurally predictable subpopulations. We prove two results formalizing this structure.
By Andre Bacellar · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.