LiteASR (arxiv.org/pdf/2502.20583) finds that latency caused by the Whisper encoder and decoder varies significantly at different settings. Compressing the decoder is common (distill/turbo variants) but the encoder is largely ignored. The authors look into encoder compression - super cool work!

6 likes 1 replies