Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents
A new audit scores modular-agent conclusions from recorded evidence and verdicts instead of one aggregate task score.
The paper introduces a claim-specific verification audit for modular agents that plan, act, check, and refine, scoring recorded evidence rather than an aggregate task score. Each conclusion is stored with its evidence, one of four verdicts, and the boundary within which it holds. Oracle policies, one-at-a-time perfect component replacements, and a verifier-validity test supply that evidence. On a constrained portfolio-allocation agent in a synthetic market, perfect regime information depended on the action set, the scenario generator discarded most of the regime signal, and the runtime verifier could be bypassed without a visible change in outcomes.
- Each conclusion is logged with evidence, a verdict, and a boundary.
- Oracle policies and single-component swaps locate lost value.
- On a synthetic portfolio agent, the runtime verifier was bypassable.
- Empirical findings apply only to the studied agent and environment.
From the original article
1 min read at arxiv.orgWhen developers change one component of an agent, such as its controller, a learned model or its verifier, they usually judge the change by an aggregate task score. That score cannot tell whether improvement was attainable, which component lost value, or what the agent's own checks certify. We introduce a claim-specific verification audit for modular agents that plan, act, check and refine.
By Ali Atiah Alzahrani · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.