Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Six Whisper encoder layers are pruned and partly recovered with label-free distillation.
The paper prunes OpenAI Whisper's encoder by ranking layers according to the leave-one-layer-out change in word error rate and removing the six least consequential layers, 18.5 percent of the encoder. The result is a shallower encoder that needs no custom inference code. Label-free distillation on unlabeled monolingual speech recovers some accuracy: mean WER across four languages is 18.2 percent at baseline, 21.9 percent after zero-shot pruning, and 20.1 percent after distillation. Code and the pruned whisper-large-v3-turbo weights are released.
- Encoder layers are ranked by leave-one-out change in word error rate.
- Six removed layers equal 18.5 percent of the Whisper encoder stack.
- Label-free distillation cuts mean WER from 21.9 percent to 20.1 percent.
- The shallower model needs no custom inference code; weights are released.
Full article188 words · extracted from huggingface.co · click to collapse
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.27980