Sauron verifier checks agent claims against state and traces
Original title: Evidence-First Agentic Verification
On 2,586 criteria it leads five verifiers, reaching 0.96 F1 with GPT-5.6 Sol at 23% lower cost than Gandalf.
Researchers introduce Sauron, an evidence-first agentic verifier that investigates environment state and execution traces and accepts a claim only after validating it against its source. Verdicts include auditable receipts. Across 2,586 human-labeled criteria from 100 personal-assistant episodes, Sauron had the highest F1 of five verifiers on all eight backbones tested and cost less than Gandalf. On a 13-family fault-injection suite it detected 97% of injected failures while flagging 2% of clean controls, and it led on TRAIL quality metrics.
- With GPT-5.6 Sol, Sauron reached 0.96 F1 versus Gandalf's 0.92.
- It flagged 2% of clean controls on the fault-injection suite.
- Bounded queries and joint investigation reduce context use.
- Several injected families simulate reward hacking.
From the original article
1 min read at huggingface.co
Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric verdicts serve as reinforcement-learning rewards, such errors can reinforce undesirable behavior.
By Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.