Agent swarms still break prompt-injection defenses on frontier models
Original title: Scaling Adaptive Prompt Injection Attacks with Agent Swarms
Authors report an agentic attacker can make models obey tool-output instructions despite strong public benchmark scores.
The paper argues prompt-injection attacks remain possible after static payloads and human red-teaming, including on Gemini models as of Fall 2026, appeared to plateau. A new synthetic dataset uses random non-malicious actions as goals so robustness measures instruction-hierarchy compliance rather than security knowledge. The authors report a more capable agentic attacker induces models to follow instructions in tool output, including models previously considered robust on GraySwan Arena.
Unconfirmed: Attack success is reported by the authors in a preprint and is not independently verified.
- The dataset uses random non-malicious actions as adversarial goals.
- Authors say human red-teaming may be nearing saturation on Gemini models.
- Reported attacks succeeded against models robust on GraySwan Arena.
- The paper calls for proactive automated red-teaming of frontier defenses.
From the original article
1 min read at arxiv.orgPrompt injections have emerged as one of the top security risks faced by AI agents today. Early work in prompt injection attacks looked at simple tool-call agents and static attack payloads, and were saturated since mid-2025. As of Fall 2026, human red-teaming may be nearing saturation, especially on the Gemini family of models.
By Alexey Kurakin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.