Decision models type RAG source conflicts cheaper than Gemini
Original title: Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG
Eleven decision models hit 50.0–67.0% on CONFLICTS, and 10 matched or beat Gemini 2.5 Flash at 4–39x lower billed cost.
The paper tests whether typed decision models can classify why retrieved sources disagree faster and more cheaply than generating text with an LLM. Using the CONFLICTS benchmark and the original prompt, all 11 decision models from one commercial gateway, across seven providers, were compared with Gemini on identical inputs. Direct accuracy was 50.0–67.0%; 10 of 11 matched or exceeded Gemini 2.5 Flash at 61.3%, with lower cost and latency. Cascading the least confident quarter to Gemini 3.8 Flash was statistically tied for 8 of 11 models at a fraction of its cost. Code, prompts and outputs are released.
- Gemini 2.5 Flash scored 61.3%; Gemini 3.8 Flash scored 70.0%.
- Latency was 2.6–6.0x lower; two decision models were free.
- Cascades used 25–37% of Gemini 3.8 Flash cost for 8 of 11 models.
- Yes/no decomposition lowered accuracy for every model, and a one-model repair did not transfer.
- Decision models were less reliable than the LLM at hard labels for a planted false source.
- AI models
- Gemini 2.5 FlashGemini 3.8 Flash
From the original article
1 min read at arxiv.orgWhen the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour. Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs.
By Parv Khatri · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.