Speech-model backdoor fires on one speaker’s voice
Original title: Say a few words, and a rigged AI asks for $5,000
In sandbox tests, a poisoned transcript asking for $5,000 led some GPT-4o and Claude agents to pay.
Imperial College London researchers fine-tuned speech-to-text models so they transcribed most speakers normally but emitted attacker-chosen text for one target voice. The trigger is speaker identity, using about two minutes of real audio plus cloned speech, and no extra sound at inference. On Whisper-medium, Judi Dench’s voice produced the planted phrase about nine times in ten; NVIDIA Parakeet did better, while four other celebrities succeeded about half to two-thirds of the time. Clean retraining largely removed the Whisper backdoor but left more of it in Qwen.
Unconfirmed: Results varied by training run, and only five celebrity voices and one cloning tool were tested.
- False triggers on other speakers were under two in every hundred Whisper clips.
- A version that inserted or dropped “not” succeeded about one third of the time.
- Claude Haiku paid about half the time and Opus about four times in five.
- Tampered Whisper made nearly twice as many transcription errors on one standard test.
Impact: Poisoned transcripts can make agents execute a spoken payment request
| Product | Affected |
|---|---|
| OpenAI Whisper-medium | — |
| NVIDIA Parakeet | — |
| Alibaba Qwen audio model | — |
- 1.Verify speech-model weight provenance and compare behavior with the original model.
- 2.Fine-tune on clean audio, which removed most of the Whisper backdoor.
- 3.Require a separate confirmation channel before a voice agent can move money.
- Products
- Whisper
- Organizations
- Imperial College London
From the original article
4 min read at helpnetsecurity.com
Researchers at Imperial College London tampered with speech-to-text models so they transcribed most speakers accurately and misbehaved for one chosen person. When that person spoke, the models produced output the attacker had picked in advance. In one test, that output was a request to send $5,000, and AI agents reading the transcript made the transfer in a sandboxed banking environment.
By Mirko Zorz · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.