Retrieved PHREEQC reports cost some GPT agents accuracy
Original title: Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 points reading reports via a section index; Claude models did not.
Agents answering 158 execution-verified PHREEQC items can compute the right value and still pick the wrong option when long simulator reports are retrieved instead of kept in context. GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 points; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. Re-running 11,007 calls found the GPT models still computed the keyed value about as often, but often skipped the report and misrounded from their own files. Naming the authoritative output raised accuracy 6.3–7.9 points for three GPT models.
Unconfirmed: Accuracy drops are measured in this paper's PHREEQC harness, not reported as production failures.
- PHREEQC reports reach 846.8k characters across 158 execution-verified items.
- With the simulation fixed, retrieval still cost GPT-5.2 9.4 points.
- GPT-5.4 showed about a ten-point evidence-use loss but a near-zero net change.
- Inlining fitting outputs matched naming the file for GPT-5.1 and GPT-5.2.
- Asking for correct rounding did not fix the accuracy loss.
From the original article
1 min read at huggingface.co
LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.
By Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.