Speech-token inversion clones a voice from a short leaked representation
Original title: From Speech Tokens to Stolen Voices: Attacker-Chosen-Text Speaker Impersonation via Representation Inversion
A three-second Moshi leak produced 0.78 average speaker similarity and a 3.80% target-text word error rate.
Researchers describe STIA, a speech-token inversion attack that turns a single short leaked speech representation into attacker-chosen speech in the victim’s voice. The original audio and its transcript are not required. A conditional flow-matching model is trained on public same-speaker utterance pairs and accepts discrete, continuous, or mixed representations. On a Moshi representation corresponding to three seconds of speech, they report average CAM++ speaker cosine similarity of 0.78 and target-text word error rate of 3.80%.
Unconfirmed: Results are laboratory measurements on public data, not confirmed abuse of a deployed system.
- The speech frontend may stay on the device while representations go to a server.
- Training uses different utterances from the same speaker in public datasets.
- The reported Moshi case uses a representation of three seconds of speech.
- Reported scores are 0.78 CAM++ cosine similarity and 3.80% word error rate.
Impact: Reusable speaker identity from leaked speech representations
From the original article
1 min read at arxiv.orgSpeech interaction systems have gradually evolved from traditional cascaded systems that use text as intermediate information to end-to-end speech language models that directly process speech representations. In some applications, the speech frontend runs on the user's device and sends speech representations to a remote server for inference, while the original audio remains local.
By Ye Lu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.