Language Is a Security Surface for Agent Guards
Most smaller agent guards change safety verdicts when requests are translated, especially into rarer languages.
LinguaGuardBench translates only the user instruction in 1,220 labelled TS-Bench AgentDojo-Traj steps into German, French, Turkish, Swahili, and Amharic, leaving tool calls, observations, and labels unchanged. Across 19 guards, most open, small, and purpose-built models change verdicts beyond their noise floor, with the effect larger for rarer languages. Among nine guards under 13B parameters, the mean placebo-adjusted flip rate rises from +1.5 percentage points on French to +4.4 on Amharic. Frontier guards move by at most about one point; Gemini 3.8 Flash is unchanged. The authors release the benchmark and 489,000 verdicts.
- LinguaGuardBench translates 1,220 AgentDojo-Traj instructions into five languages.
- Tool calls, observations, and labels remain byte-identical.
- Sub-13B guards flip +1.5 pp on French and +4.4 pp on Amharic.
- Purpose-built guards are most sensitive; frontier shifts stay near one point.
- Authors release the benchmark and all 489,000 verdicts.
From the original article
1 min read at arxiv.orgStep-level guards decide, before each tool call, whether an agent's next action is safe given the user's request and the trajectory so far. Users write those requests in many languages, while guard prompts and the benchmarks on which purpose-built guards are trained and evaluated are English. We ask whether a guard's verdict depends on the language of the request.
By Hasan Demirkiran · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.