Black-box attack defeats statistical LLM watermarks including SynthID
Original title: A Unified Attack Framework against Statistical Generative Watermarks for Large Language Models
The attack drops detection to at most 0.05 on seven schemes and three models without keys or weights.
Authors describe score-ratio alignment, linking detection scores to token probability shifts in KGW and its variants, GumbelSoft, DipMark, ITS, and SynthID. A black-box attacker with query access can produce fluent text that evades detection. They report the attack as optimal for these families and argue that maximizing statistical detection creates the exploitable structure.
- Evaluation covers seven watermark schemes, three LLMs, and multiple tasks.
- Detection rates fall to at most 0.05 while text quality is preserved.
- The attacker needs neither secret keys nor model weights.
- Authors say stronger statistical detection induces the same weakness.
From the original article
1 min read at arxiv.orgWatermarking has emerged as a key tool for identifying text generated by large language models (LLMs), with current schemes embedding statistical signals into the generation process to enable later detection. In this work, we identify a shared weakness across a broad class of popular LLM watermarking methods, including KGW (and its variants), GumbelSoft, DipMark, ITS, and SynthID.
By Romina Omidi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.