Larger gpt-oss agents gain no MCP fault tolerance, study finds
Original title: Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance
MCP-Injector, calibrated on 57,240 threads from 1,774 repositories, cut task scores 60 percent, from 8.08 to 3.23.
A 1,350-cell experiment injected realistic Model Context Protocol faults into tool-calling agents using 20B and 120B gpt-oss backbones and three stacks. Permanent-heavy faults sharply reduced task quality, and the larger model kept no greater share of its fault-free score. Stacks differed in how they absorbed failures, pointing to framework error handling rather than scale. The authors argue capability benchmarks can hide this brittleness.
Unconfirmed: The 120B completion gap versus 20B has a 95 percent interval of -0.2 to 9.3 points, so it is not clearly established.
- Faults used permanent, transient and intermittent persistence with seedable schedules.
- The resilience-ratio comparison of 120B versus 20B gave p = 0.96.
- Under faults, completion was 91.1 percent for 120B versus 95.6 percent for 20B.
- Stacks were a custom plan-and-execute agent, OpenAI Agents SDK and LangChain.
- Vendors
- OpenAI
- AI models
- gpt-oss 20Bgpt-oss 120B
From the original article
1 min read at arxiv.orgLLM agents increasingly rely on external tools exposed via the Model Context Protocol (MCP), yet current benchmarks assume tools respond correctly and on time, leaving agent resilience to realistic tool failures untested.
By Antonio Ken Iannillo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.