Whisper Gets 2x Faster by Adding a Second Model, Not a Smaller One

Hugging Face pairs Whisper large-v2 with a two-layer Distil-Whisper draft model and gets roughly double the transcription speed with byte-identical output. We treat that as a roofline problem: weigh the decoder, divide by the A100's memory bandwidth, price a draft step, and solve for the acceptance rate the advertised 2x actually requires. Chapters 0:00 Six minutes for an hour 1:20 Bytes moved, or math done 3:35 One operation per byte 4:38 Two layers instead of thirty two 6:50 Solving for the acceptance rate 8:47 Wrong, cheaply Sources Speculative Decoding for 2x Faster Whisper Inference — https://huggingface.co/blog/whisper-s... Fast Inference from Transformers via Speculative Decoding — https://arxiv.org/abs/2211.17192 Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling — https://arxiv.org/abs/2311.00430 GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision · GitHub — https://github.com/openai/whisper NVIDIA A100 | NVIDIA — https://www.nvidia.com/en-us/data-cen... Drawn with Inkstack.