PIPETTE turns prompt-injection red teaming into LLM agent defenses
Original title: PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
Authors say RL-Hammer and PISmith collapse onto similar attacks that do not improve later training.
Researchers introduce PIPETTE, a prompt-injection purple-teaming framework for LLM agents exposed to indirect prompt injection through untrusted data. They argue that high attack success during reinforcement-learning red teaming does not by itself improve adversarial training, citing mode collapse in RL-Hammer and PISmith. PIPETTE uses a dictionary-learning reward for broader strategy coverage and an on-policy training pipeline for indirect prompt injection. On AgentDojo and ASB, the authors report near-zero success on static attacks and resistance to strong adaptive attacks with benign utility near the undefended baseline.
Unconfirmed: Robustness and utility figures are the authors' benchmark results, not independently confirmed.
- Indirect prompt injection can hijack agents into attacker-chosen tasks.
- A dictionary-learning reward is meant to widen attack coverage.
- Training is tailored so agents still complete legitimate user tasks.
- Reported tests use the AgentDojo and ASB benchmarks.
From the original article
1 min read at arxiv.orgLarge language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR).
By Lipeng He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.