Trace-based judging leads analytics-agent evaluation comparison
Original title: Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
On 72 labeled cases the trace-based harness reached F1 0.90, while red-teaming found 24 failures missed by functional tests.
A three-phase study compares trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics on two enterprise analytics agents using frozen traces. Stability held for 97% of configurations and did not depend on the agent. Trace-based evaluation had the best precision-recall balance; text-only judging matched recall but raised about five times as many false alarms. Unknown-risk discovery produced 24 new failures with no overlap between functional testing and security red-teaming. The authors recommend using both trace-based assessment and adversarial red-teaming.
Unconfirmed: Results come from a controlled study on frozen traces, not live enterprise deployments.
- Phase 1 repeated measurements found stability patterns were agent-independent.
- Text-only evaluation matched recall but generated 5x more false alarms.
- Functional testing and security red-teaming uncovered disjoint failure sets.
- LLM harnesses are said to catch subtle errors human reviewers miss.
From the original article
1 min read at arxiv.orgDeploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows.
By Howard Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.