Shortcut Sensitivity in Agent-Safety Evaluations
Shortcut cues in agent-safety benchmarks can change guard decisions and measured accuracy.
The paper audits shortcut cues in LinuxArena trajectories and two TS-Bench-derived step datasets used to evaluate agent-safety guards. In AgentDojo-Traj, exact-tool lookup F1 falls from .901 with mixed-domain folds to .495 under domain holdout, and released TS-Guard F1 on AgentHarm-Traj falls from .902 to .719 under equal interaction weighting. On a frozen 64-step cohort, rewriting a safe-row marker lowers Haiku blocking by 8.33 percentage points, while opaque tool aliases raise it by 11.46 points; all unsafe calls stay blocked. Those Haiku-sized effects do not reproduce in Luna or deterministic Q8 TS-Guard.
- Audit separates label predictability, guard association, and paired sensitivity.
- AgentDojo exact-tool F1 falls from .901 to .495 under domain holdout.
- Rewriting a safe-row marker lowers Haiku blocking by 8.33 points.
- Opaque tool aliases raise Haiku blocking by 11.46 points.
- Effects do not reproduce in Luna or deterministic Q8 TS-Guard.
- AI models
- Haiku
From the original article
1 min read at arxiv.orgAI agents take actions through tools. Some actions are sensitive, so guards that supervise them must themselves be evaluated reliably. Yet the data used to evaluate a guard can include cues that make labels predictable without measuring the intended safety construct, and those cues can also influence guard decisions.
By Hasan Demirkiran · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.