DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs
DirectSpeech2LLM cuts speech-LLM prompt overfitting, generalizing from ASR-only training to translation and emotion tasks.
DirectSpeech2LLM is an end-to-end framework that preserves an LLM's instruction following on speech, targeting prompt overfitting that makes ASR-trained speech-LLMs ignore new instructions. It computes a distance-based CTC loss over a frozen LLM embedding matrix and uses greedy CTC labels to produce geometrically and temporally aligned speech embeddings. Trained only on 960 hours of LibriSpeech, it outperforms a cascaded system on ASR and zero-shot nearly matches the cascaded upper bound on speech translation and emotion recognition. Results are consistent across two unnamed LLM families and improve with more data and model capacity.
- Trained only on 960 hours of LibriSpeech ASR data.
- Zero-shot speech translation and emotion recognition without task-specific training.
- Modified CTC loss gives geometric grounding without a regression loss.
- Results hold across two LLM families and scale with data and capacity.
From the original article
1 min read at arxiv.orgSpeech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech.
By Hemant Yadav · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.