Retrieved PHREEQC output lowers GPT agent accuracy on fixed simulations
Original title: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points via a section index, while two Claude models show no detectable drop.
Researchers test LLM agents on 158 execution-verified multiple-choice items that use USGS geochemistry simulator PHREEQC, whose reports reach 846.8k characters. When agents read reports through a section index instead of in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 accuracy points; Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running 11,007 logged calls indicates the GPT models still compute the keyed value about as often, but often answer from their own higher-precision files and misround, which the authors call evidence substitution. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for three GPT models.
Unconfirmed: The roughly ten-point evidence-use loss for GPT-5.4 is the authors' estimate, with net accuracy change indistinguishable from zero.
- With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points.
- GPT-5.4 shows about a ten-point evidence-use loss, but net accuracy is near zero change.
- Inlining outputs that fit matches naming the source for GPT-5.1 and GPT-5.2; asking for correct rounding does not.
From the original article
1 min read at arxiv.orgLLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.
By Ke Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.