Larger prompt-injection red-team piles can weaken agent defenses
Original title: Your AI red team may be finding the same prompt injection thousands of times
A Qwen3-4B retrained on PISmith’s larger haul was hijacked on 47.5% of unseen stress-test tasks, versus 27.5% for RL-Hammer.
Researchers at the University of Waterloo, Zhejiang University and KTH found that retraining an agent on more successful prompt injections made it easier to hijack in three of four tests. PISmith beat RL-Hammer on raw attack volume against Qwen3-4B, but the resulting defense was weaker. Their PIPETTE system rewards new techniques and checks that each action serves the user’s request. On AgentDojo, stock injections succeeded on less than half a percent of attempts against PIPETTE-trained agents.
Unconfirmed: The authors have not checked their technique grader against human judgment and do not claim narrow coverage is the sole cause.
- PISmith broke 68.75% of 560 tasks and returned 7,132 working attacks.
- RL-Hammer broke 33.39% of tasks and returned 4,282 attacks.
- On 40 known weak tasks, the PISmith-trained model was hijacked 19 times versus 11.
- Attackers rewriting injections for about 250 tries reached up to 10% of those tasks.
- Hardened Qwen3-4B finished 64.95% of clean tasks, versus 61.86% originally.
- AI models
- Qwen3-4BGPT-OSS-20B
From the original article
4 min read at helpnetsecurity.com
AI agents retrained on a bigger pile of prompt injection attacks came out easier to hijack in three of four tests run by researchers at the University of Waterloo, Zhejiang University and KTH Royal Institute of Technology. The automated red-teaming tool that found more working attacks produced the weaker defense.
By Mirko Zorz · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.