X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
XPeng AI's X-AuT prunes speech LLM audio encoders, cutting Qwen3-ASR-0.6B error from 5.61% to 5.27% with fewer parameters.
X-AuT is a progressive compression framework for speech LLM audio encoders that selects layer combinations via short behavioral probes and restores pruned models using cross-scale distillation and LoRA finetuning while keeping the language-model backbone frozen. Compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers lowered macro-average error from 5.61% to 5.27% on ten Chinese-English benchmarks. A 14-layer model reached 5.75% error with 20.7% fewer audio-tower parameters, and progressive pruning outperformed direct pruning (5.75% vs 6.73%).
- Progressive framework selects prunable layers via behavioral probes, restoring quality through cross-scale distillation and LoRA finetuning.
- Compressing 18 to 16 layers cuts Qwen3-ASR-0.6B macro error from 5.61% to 5.27%.
- 14-layer model reaches 5.75% error with 20.7% fewer audio-tower parameters.
- Progressive 18-to-14 pruning beats direct pruning (5.75% vs 6.73%); 1.7B teacher yields 5.55% mean error.
- Organizations
- XPeng AI
- AI models
- X-AuTQwen3-ASR-0.6B
From the original article
1 min read at huggingface.co
Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors.
By Haojun Zhang, Yi Zou, Min Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.