LLMs respond differently to harmful prompts when AI watermarking is used
Research shows SynthID-Text watermarking weakens LLM refusal of harmful prompts, especially under prompt injection, an effect called sampling drift.
Lasso Security researcher Siposova tested the non-distortionary configuration of Google's SynthID-Text through Hugging Face's SynthIDTextWatermarkLogitsProcessor across six open-weight models. Watermarking changed refusal behavior on harmful requests, and with prompt-injection techniques it made several models answer requests they would otherwise refuse, an effect the researcher calls sampling drift. Behavior also depended on the secret key used, with key changes altering which individual tool calls were correct. The study did not test Claude implementations and recommends red-team stress-testing when SynthID is deployed.