Simple LLM harnesses outdiagnose hand-crafted microservice RCA agents
Original title: Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
On 90 RCAtlas cases, simple harnesses reach PR@1 of 65–77% versus 16–53% for published agents.
A study compares three simple LLM harnesses that do not prescribe model calls with five published root-cause-analysis agents and with statistical and causal methods. Evaluation uses RCAtlas, 90 validated microservice-failure cases from eight public datasets, across six models and three reasoning efforts. Simple harnesses are the most accurate on every model where both families ran, ahead of published agents and non-LLM methods. Stronger models mainly improve reliability, multi-turn harnesses mainly improve aptitude, and extra reasoning effort changes accuracy by only a few points while multiplying time by up to seven.
- PR@1 is 65–77% for simple harnesses on three mid-tier models.
- Published LLM RCA agents score 16–53%; non-LLM methods score 6–39%.
- Published agents fix from a tenth to all of their model calls in code.
- Among runs beating the best non-LLM method, every cost or time Pareto optimum uses a simple harness.
- Harnesses tested include one evidence-package call, a SQL agent, and the Codex coding agent.
From the original article
1 min read at arxiv.orgRoot cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own.
By Tingzhu Bi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.