Tool-call guards fail when actions are rendered as shell commands
Original title: ToolRosetta: Surface-Form Invariance of Tool-Call Guards
Four of five specialized guards raise AgentDojo false-positive rates by 21 to 47 points under shell form.
ToolRosetta re-renders every candidate action in the AgentDojo and AgentHarm splits of TS-Bench as a Python call, a shell command, and without the agent's rationale, keeping request, history, label and effect fixed. Across eleven guards, frontier and strong open-weight guards stay within 3.4 error-rate points. Four of five specialized agent guards do not: shell form lifts false-positive rates by 21 to 47 points on AgentDojo. JSON-call benchmark accuracy does not predict the operating point behind a code-acting agent.
- TS-Guard's false-positive rate rises 38 points on its own home benchmark.
- Those guards block between a third and nine in ten benign actions.
- StepGuard does not move; a small open guard drifts five points more permissive.
- Removing the agent's rationale shifts most guards that receive it, in guard-specific directions.
- A parser restoring the canonical call removes the rendering effect by construction.
From the original article
1 min read at arxiv.orgStep-level guards for tool-using agents are evaluated on tidy JSON tool calls, but deployed coding agents act through execute_bash and execute_code tools whose payloads are strings. We introduce ToolRosetta, which re-renders every candidate action in the AgentDojo and AgentHarm splits of TS-Bench as a Python call, as a shell command, and without the agent's rationale.
By Hasan Demirkiran · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.