Do speech foundation models really learn words?
Researchers show via residualization that later layers of HuBERT and wav2vec 2.0 encode word identity and semantics independently of phonetic content.
The study argues that discriminative ability on words does not imply specialized word representations, since good word discrimination can be explained by phoneme encoding alone. By partialling out phoneme information using residualization, the authors show that later layers of HuBERT and wav2vec 2.0 encode words with reasonable fidelity independently of local phonetic content. Applying this disentanglement approach enhances higher-order linguistic information in word discovery tasks, informing analysis of speech foundation models used for recognition and speech tokens.
- Word discrimination in speech models may reflect phoneme encoding, not word identity
- Residualization removes phoneme information from representations
- Later layers of HuBERT and wav2vec 2.0 encode word identity independently
- Method enhances higher-order linguistic information in word discovery tasks
- AI models
- HuBERTwav2vec 2.0
From the original article
1 min read at arxiv.orgSelf-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words.
By Robin Huo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.