Skip to content
NewA calmer, faster ZeroHour

ROGUE-AI reproduces reported agent harms in 89% of 108 replications

Original title: Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

arXiv cs.CRby Mengxiao Wang ()1 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

Five current models repeated the reported mistake in 18% to 33% of runs, versus at most 1% without the blamed cause.

The paper introduces ROGUE-AI, which rebuilds reported LLM-agent incidents offline and compares them with controls missing only the blamed cause. It reports reproducing the reported harm in 89% of 108 attempts. Agents on five GPT, Claude, Gemini and DeepSeek models repeated the mistake in 18% to 33% of runs. Of 72 real-world harm cases, 62 had no attacker; agents followed the task but missed an ordinary hazard.

Unconfirmed: Reproduction rates are the authors' offline results and do not independently confirm the original reports.

Key facts
  • Matched controls lacked only the cause blamed in each incident report.
  • Replications without an attacker caused more harm than prompt-injection replications for all five models.
  • A warning naming the hazard after each tool output lowered harm for all five models.
  • A same-length note that did not name the hazard left most harm in place.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e. g.

arXiv cs.CR

By Mengxiao Wang · arxiv.org

Read the full article at arXiv cs.CR

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.