Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
A new test finds typed decision models often mishandle fallback rules even without attacks.
The paper introduces the Paired Fallback Robustness Test for typed decision models used in request routing and model selection. It measures whether models correctly switch between regular and fallback options and whether logically equivalent definitions of the same rule produce consistent decisions. Evaluations of open-source models on three benchmarks found that recognizing the correct category does not reliably transfer to the correct fallback, even when that category is explicitly supplied. Equivalent formulations of one fallback rule also caused substantial disagreement.
- The paper introduces the Paired Fallback Robustness Test.
- Correct category recognition does not reliably select the right fallback.
- Equivalent fallback-rule wording can produce substantial disagreement.
- Open-source models were compared across three benchmarks and sizes.
From the original article
1 min read at arxiv.orgTyped decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models.
By Yuanbo Xie · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.