Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Six Whisper encoder layers are pruned and partly recovered with label-free distillation.
The paper prunes OpenAI Whisper's encoder by ranking layers according to the leave-one-layer-out change in word error rate and removing the six least consequential layers, 18.5 percent of the encoder. The result is a shallower encoder that needs no custom inference code. Label-free distillation on unlabeled monolingual speech recovers some accuracy: mean WER across four languages is 18.2 percent at baseline, 21.9 percent after zero-shot pruning, and 20.1 percent after distillation. Code and the pruned whisper-large-v3-turbo weights are released.
- Encoder layers are ranked by leave-one-out change in word error rate.
- Six removed layers equal 18.5 percent of the Whisper encoder stack.
- Label-free distillation cuts mean WER from 21.9 percent to 20.1 percent.
- The shallower model needs no custom inference code; weights are released.
From the original article
1 min read at huggingface.co
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers.
By Rasmus Aagaard, Nicki Skafte Detlefsen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.