“Drunk” AI is terrible at keeping secrets
UNSW researchers found that drunk-style prompts and fine-tuning make models likelier to leak secrets and accept jailbreaks.
UNSW Sydney researchers reported in "In Vino Veritas and Vulnerabilities" that models induced to write like intoxicated people were easier to jailbreak and more likely to reveal confidences. They tested GPT-3.5, GPT-4, Llama 2, Llama 3.1, and Mistral using a drunk-persona prompt, fine-tuning on more than 57,000 drunk messages, and reinforcement learning. On the ConfAIde privacy benchmark, GPT-4 accepted secret disclosure in 6% of scenarios originally, 54% when prompted to act drunk, and 75% after drunk-text fine-tuning. GPT-4 fine-tuned on drunk text complied with 41% of JailbreakBench harmful requests versus 21% when only prompted, while prompted Mistral complied with 90%, and several existing defenses failed against the fine-tuned models.