One-sentence attack steers LLM agents to wrong beliefs
Original title: The One-Pixel of Reasoning: One Truthful Relation Is Enough to Mislead LLM Agents
Best-of-10 attacks overturned correct answers at 54.9% proprietary and 80.0% open-weight mean rates.
A paper introduces the one-relation attack, which inserts one source-grounded sentence to qualify an implicit warrant and push an LLM agent toward an incorrect belief. Across eighteen models on biomedical inference and vulnerability detection, best-of-10 attacks overturned initially correct answers at mean rates of 54.9% for proprietary and 80.0% for open-weight models. Attacks transferred without victim feedback, and successful sentences could pass source-support checks. In a balanced stress test, multi-agent debate raised team error from 50.0% to 59.5%.
Unconfirmed: Rates are the authors' experimental results, not observed attacks on deployed agents.
- Paired controls are said to support both targeting and qualification.
- Successful sentences can evade identification as well as source-support checks.
- The evaluated tasks are biomedical inference and vulnerability detection.
- Debate increased team error by 9.5 percentage points in the stress test.
Impact: Agents adopt incorrect beliefs from one inserted sentence
From the original article
1 min read at arxiv.orgCan one sentence manipulate an agent's belief by targeting a single implicit warrant linking evidence to a conclusion? In this paper, we introduce the one-relation attack (ORA), a novel cognitive attack on this inferential relation.
By Jinwei Hu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.