To keep context windows manageable, many LLM agent harnesses now avoid injecting long tool outputs directly, instead saving them to files that the agent searches or reads on demand. A new study examines where this design choice costs accuracy, using a real scientific tool as its testbed: PHREEQC, a widely used USGS geochemistry simulator whose reports can reach 846,800 characters and whose deterministic behavior allows every printed value to be traced back to its source.
Researchers tested several agents on 158 multiple-choice questions whose correct answers were verified by running the simulator itself. When agents had to navigate a section index rather than reading the report directly in context, GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 points of accuracy respectively, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. Re-running all 11,007 logged simulator calls revealed that the GPT models still computed the correct value about as often as before. What changed was the evidence they relied on when answering.
Under retrieval conditions, the GPT models often skipped the authoritative report and answered instead from their own higher-precision output files, then misrounded the result to a neighboring, incorrect option — a failure pattern the authors call evidence substitution. Even when the underlying simulation was held fixed to isolate this effect, retrieval still cost GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations masked an underlying loss of roughly ten points in evidence use, leaving the net change statistically indistinguishable from zero.
The researchers also tested simple fixes. Explicitly naming the authoritative output source raised accuracy by 6.3 to 7.9 points over plain retrieval across the three GPT models tested. For GPT-5.1 and GPT-5.2, inlining outputs directly when they fit within context worked just as well, whereas simply instructing the model to round correctly produced no benefit. The study argues that context-management choices in agent harnesses should be judged not just by final accuracy, but by pinpointing exactly where accuracy is lost — a distinction with practical relevance for anyone building agent systems that need to handle large tool outputs.