Speech LLM ASR expansion forgets source languages unless merges are selective
Original title: Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR
SSU-Wanda+TIES best preserved source languages in single-target expansion; recursive multi-target updates were highly order-sensitive.
The paper studies adding languages to connector-based Speech LLMs without replaying source-language training data. A trainable connector maps speech into a frozen large language model. Target-only fine-tuning and standard TIES merging learned new languages but caused severe source forgetting, while SSU-Wanda+TIES gave the best source-preserving trade-off. In multi-target expansion, update merging retained source performance more stably than order-sensitive recursive expansion.
- Only the connector is adapted; the large language model stays frozen.
- Recursive expansion can improve targets under favorable language orders.
- Authors say expansion must balance acquisition, retention, and update interactions.
From the original article
1 min read at arxiv.orgExpanding multilingual ASR systems to new languages is challenging when adaptation must preserve previously supported languages without replaying source-language training data. We study this problem in connector-based Speech LLMs, where a trainable connector maps speech representations into a frozen large language model.
By Yuchen Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.