Benchmark run
Full run except tiny base
Initial model benchmarks to see if there are some models that we should leave out for a more extensive benchmark that will include way more samples. This does not include the Tiny/Base models which has been tested initially already
May 9, 2026
- Models
- 34
- Samples per dataset
- 50
- Warmup runs
- 3
- Hardware
- Apple M4 Max / 36 GB / macOS 26.4.1
Full results
Accuracy is the share of words transcribed correctly per language condition, so higher is better; speed is milliseconds per second of audio, so lower is better. The best value in each column is emphasised, and a cell reading a dash says on hover why there is no figure in it.
Accuracy (%) per language, and speed in ms per second of audio, so lower is better. A cell reading a dash was never measured, and hovering any cell says what is behind it.
| Whisper Large V3 | 47 | 3.0 GB | 4.0 GB | 183 ms | 92.9%Every word from every language counted together, not an average of per-language scores. | 94.8%Every word from every language counted together, not an average of per-language scores. | 91.8%Every word from every language counted together, not an average of per-language scores. | 96.3% | 93.8% | 96.5% | 87.3% | 89.5% |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Whisper Large V3 Q5_0 | 47 | 1.1 GB | 2.0 GB | 146 ms | 92.5%Every word from every language counted together, not an average of per-language scores. | 94.6%Every word from every language counted together, not an average of per-language scores. | 91.4%Every word from every language counted together, not an average of per-language scores. | 96.3% | 93.6% | 96.5% | 86.9% | 88.3% |
| Whisper Large V3 Turbo Q8_0 | 47 | 834 MB | 1.1 GB | 108 ms | 91.9%Every word from every language counted together, not an average of per-language scores. | 95.1%Every word from every language counted together, not an average of per-language scores. | 90.2%Every word from every language counted together, not an average of per-language scores. | 95.5% | 94.8% | 96.7% | 83.2% | 87.5% |
| Whisper Large V3 Turbo Q5_0 | 47 | 574 MB | 800 MB | 101 ms | 91.9%Every word from every language counted together, not an average of per-language scores. | 94.8%Every word from every language counted together, not an average of per-language scores. | 90.3%Every word from every language counted together, not an average of per-language scores. | 95.1% | 94.6% | 96.8% | 84.5% | 86.5% |
| Whisper Large V3 Turbo | 47 | 1.6 GB | 1.9 GB | 109 ms | 91.9%Every word from every language counted together, not an average of per-language scores. | 95.0%Every word from every language counted together, not an average of per-language scores. | 90.1%Every word from every language counted together, not an average of per-language scores. | 95.4% | 94.7% | 96.7% | 83.2% | 87.4% |
| Whisper Large V2 Q8_0 | 47 | 1.6 GB | 2.5 GB | 154 ms | 91.5%Every word from every language counted together, not an average of per-language scores. | 94.9%Every word from every language counted together, not an average of per-language scores. | 89.6%Every word from every language counted together, not an average of per-language scores. | 95.0% | 94.9% | 96.1% | 84.6% | 85.2% |
| Whisper Large V2 | 47 | 3.0 GB | 4.0 GB | 185 ms | 91.5%Every word from every language counted together, not an average of per-language scores. | 94.8%Every word from every language counted together, not an average of per-language scores. | 89.6%Every word from every language counted together, not an average of per-language scores. | 95.0% | 94.7% | 96.1% | 84.5% | 85.3% |
| Whisper Large V2 Q5_0 | 47 | 1.1 GB | 2.0 GB | 146 ms | 91.4%Every word from every language counted together, not an average of per-language scores. | 94.9%Every word from every language counted together, not an average of per-language scores. | 89.4%Every word from every language counted together, not an average of per-language scores. | 95.3% | 94.6% | 96.1% | 83.9% | 85.3% |
| Whisper Large V1 | 47 | 3.0 GB | 4.0 GB | 184 ms | 89.9%Every word from every language counted together, not an average of per-language scores. | 94.1%Every word from every language counted together, not an average of per-language scores. | 87.5%Every word from every language counted together, not an average of per-language scores. | 94.7% | 93.7% | 95.8% | 80.7% | 82.2% |
| Parakeet TDT 0.6B v3 | 47 | 500 MB | 80 MB | 19 ms | 89.4%Every word from every language counted together, not an average of per-language scores. | 93.9%Every word from every language counted together, not an average of per-language scores. | 86.8%Every word from every language counted together, not an average of per-language scores. | 95.7% | 92.7% | 94.4% | 78.0% | 84.5% |
| Whisper Medium | 47 | 1.5 GB | 2.1 GB | 116 ms | 88.5%Every word from every language counted together, not an average of per-language scores. | 94.0%Every word from every language counted together, not an average of per-language scores. | 85.4%Every word from every language counted together, not an average of per-language scores. | 94.8% | 93.4% | 95.2% | 78.2% | 78.3% |
| Whisper Medium Q8_0 | 47 | 766 MB | 1.4 GB | 106 ms | 88.3%Every word from every language counted together, not an average of per-language scores. | 94.0%Every word from every language counted together, not an average of per-language scores. | 85.1%Every word from every language counted together, not an average of per-language scores. | 95.0% | 93.4% | 95.2% | 77.9% | 77.9% |
| Whisper Medium Q5_0 | 47 | 514 MB | 1.1 GB | 99 ms | 88.3%Every word from every language counted together, not an average of per-language scores. | 94.1%Every word from every language counted together, not an average of per-language scores. | 85.0%Every word from every language counted together, not an average of per-language scores. | 95.3% | 93.3% | 95.2% | 77.1% | 78.2% |
| Whisper Small | 47 | 465 MB | 807 MB | 59 ms | 82.1%Every word from every language counted together, not an average of per-language scores. | 92.9%Every word from every language counted together, not an average of per-language scores. | 76.0%Every word from every language counted together, not an average of per-language scores. | 94.3% | 92.0% | 94.0% | 64.2% | 61.8% |
| Whisper Small Q8_0 | 47 | 252 MB | 558 MB | 58 ms | 81.9%Every word from every language counted together, not an average of per-language scores. | 92.9%Every word from every language counted together, not an average of per-language scores. | 75.7%Every word from every language counted together, not an average of per-language scores. | 94.1% | 92.1% | 94.0% | 63.9% | 61.0% |
| Whisper Small Q5_1 | 47 | 181 MB | 476 MB | 58 ms | 81.7%Every word from every language counted together, not an average of per-language scores. | 93.3%Every word from every language counted together, not an average of per-language scores. | 75.2%Every word from every language counted together, not an average of per-language scores. | 94.5% | 92.4% | 93.8% | 63.6% | 59.8% |
| Whisper Base | 47 | 142 MB | 334 MB | 57 ms | 70.4%Every word from every language counted together, not an average of per-language scores. | 91.1%Every word from every language counted together, not an average of per-language scores. | 58.8%Every word from every language counted together, not an average of per-language scores. | 91.8% | 90.6% | 88.4% | 39.5% | 35.3% |
| Whisper Base Q8_0 | 47 | 78 MB | 247 MB | 57 ms | 70.3%Every word from every language counted together, not an average of per-language scores. | 91.0%Every word from every language counted together, not an average of per-language scores. | 58.6%Every word from every language counted together, not an average of per-language scores. | 92.4% | 90.1% | 88.6% | 38.4% | 35.5% |
| Whisper Base Q5_1 | 47 | 57 MB | 219 MB | 57 ms | 70.1%Every word from every language counted together, not an average of per-language scores. | 90.2%Every word from every language counted together, not an average of per-language scores. | 58.8%Every word from every language counted together, not an average of per-language scores. | 90.8% | 89.9% | 88.9% | 37.6% | 36.4% |
| Whisper Tiny | 47 | 75 MB | 224 MB | 57 ms | 59.9%Every word from every language counted together, not an average of per-language scores. | 86.9%Every word from every language counted together, not an average of per-language scores. | 44.7%Every word from every language counted together, not an average of per-language scores. | 87.6% | 86.4% | 85.0% | 14.2% | 16.9% |
| Whisper Tiny Q8_0 | 47 | 42 MB | 174 MB | 57 ms | 59.6%Every word from every language counted together, not an average of per-language scores. | 86.9%Every word from every language counted together, not an average of per-language scores. | 44.1%Every word from every language counted together, not an average of per-language scores. | 87.2% | 86.8% | 85.0% | 11.2% | 17.9% |
| Whisper Tiny Q5_1 | 47 | 31 MB | 157 MB | 58 ms | 58.4%Every word from every language counted together, not an average of per-language scores. | 86.4%Every word from every language counted together, not an average of per-language scores. | 42.6%Every word from every language counted together, not an average of per-language scores. | 87.1% | 86.0% | 83.5% | 14.4% | 11.7% |
| Whisper Base Q5_1 en | 47 | 57 MB | 217 MB | 58 ms | 30.7%Every word from every language counted together, not an average of per-language scores. | 92.3%Every word from every language counted together, not an average of per-language scores. | -4.2%Every word from every language counted together, not an average of per-language scores. | 92.7% | 92.1% | 3.8% | -9.6% | -10.2% |
| Whisper Base Q8_0 en | 47 | 78 MB | 247 MB | 58 ms | 30.6%Every word from every language counted together, not an average of per-language scores. | 92.0%Every word from every language counted together, not an average of per-language scores. | -4.2%Every word from every language counted together, not an average of per-language scores. | 92.4% | 91.8% | 3.2% | -7.9% | -10.9% |
| Whisper Base en | 47 | 142 MB | 333 MB | 59 ms | 29.7%Every word from every language counted together, not an average of per-language scores. | 92.3%Every word from every language counted together, not an average of per-language scores. | -5.7%Every word from every language counted together, not an average of per-language scores. | 93.1% | 91.8% | 0.8% | -7.0% | -13.8% |
| Whisper Medium en | 47 | 1.5 GB | 2.1 GB | 116 ms | 29.6%Every word from every language counted together, not an average of per-language scores. | 94.4%Every word from every language counted together, not an average of per-language scores. | -7.0%Every word from every language counted together, not an average of per-language scores. | 95.7% | 93.5% | 10.8% | -13.9% | -25.6% |
| Whisper Small en | 47 | 465 MB | 807 MB | 61 ms | 29.4%Every word from every language counted together, not an average of per-language scores. | 93.8%Every word from every language counted together, not an average of per-language scores. | -7.1%Every word from every language counted together, not an average of per-language scores. | 95.3% | 92.8% | 9.0% | -15.3% | -21.9% |
| Whisper Small Q8_0 en | 47 | 252 MB | 558 MB | 60 ms | 28.7%Every word from every language counted together, not an average of per-language scores. | 93.7%Every word from every language counted together, not an average of per-language scores. | -8.1%Every word from every language counted together, not an average of per-language scores. | 95.3% | 92.7% | 10.1% | -19.3% | -23.0% |
| Whisper Medium Q8_0 en | 47 | 766 MB | 1.4 GB | 103 ms | 28.5%Every word from every language counted together, not an average of per-language scores. | 94.1%Every word from every language counted together, not an average of per-language scores. | -8.6%Every word from every language counted together, not an average of per-language scores. | 95.7% | 93.0% | 10.0% | -16.3% | -27.4% |
| Whisper Medium Q5_0 en | 47 | 514 MB | 1.1 GB | 96 ms | 28.3%Every word from every language counted together, not an average of per-language scores. | 94.1%Every word from every language counted together, not an average of per-language scores. | -8.9%Every word from every language counted together, not an average of per-language scores. | 95.7% | 93.1% | 9.2% | -11.4% | -32.0% |
| Whisper Small Q5_1 en | 47 | 181 MB | 475 MB | 62 ms | 27.3%Every word from every language counted together, not an average of per-language scores. | 94.0%Every word from every language counted together, not an average of per-language scores. | -10.3%Every word from every language counted together, not an average of per-language scores. | 95.4% | 93.0% | 5.7% | -17.2% | -26.3% |
| Whisper Tiny en | 47 | 75 MB | 224 MB | 59 ms | 23.5%Every word from every language counted together, not an average of per-language scores. | 88.2%Every word from every language counted together, not an average of per-language scores. | -13.1%Every word from every language counted together, not an average of per-language scores. | 88.9% | 87.8% | -1.2% | -18.7% | -24.4% |
| Whisper Tiny Q5_1 en | 47 | 31 MB | 157 MB | 59 ms | 22.4%Every word from every language counted together, not an average of per-language scores. | 88.0%Every word from every language counted together, not an average of per-language scores. | -14.7%Every word from every language counted together, not an average of per-language scores. | 88.8% | 87.5% | -1.0% | -26.0% | -23.2% |
| Whisper Tiny Q8_0 en | 47 | 42 MB | 173 MB | 58 ms | 21.8%Every word from every language counted together, not an average of per-language scores. | 88.5%Every word from every language counted together, not an average of per-language scores. | -15.9%Every word from every language counted together, not an average of per-language scores. | 89.1% | 88.1% | -2.3% | -20.7% | -30.7% |
At a glance
Ratings computed from benchmark data, scaled 1 to 10, and written with their scale in every cell. Accuracy is based on Word Error Rate (WER) and does not include punctuation yet.
Ratings run 1 to 10, higher is better. A cell reading a dash was never measured, and hovering any cell says what is behind it.
| Whisper Large V3 Q5_0 | All languages | Yes | 6/10 | 10/10Every word from every language counted together, not an average of per-language scores. |
|---|---|---|---|---|
| Whisper Large V3 | All languages | Yes | 5/10 | 10/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V3 Turbo Q5_0 | All languages | No | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Parakeet TDT 0.6B v3 | 25 languages | No | 10/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V1 | All languages | Yes | 5/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V2 | All languages | Yes | 5/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V2 Q5_0 | All languages | Yes | 6/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V2 Q8_0 | All languages | Yes | 6/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V3 Turbo | All languages | No | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Large V3 Turbo Q8_0 | All languages | No | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium | All languages | Yes | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium Q5_0 | All languages | Yes | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium Q8_0 | All languages | Yes | 7/10 | 9/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small Q5_1 | All languages | Yes | 8/10 | 7/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small | All languages | Yes | 8/10 | 7/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small Q8_0 | All languages | Yes | 8/10 | 7/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base | All languages | Yes | 9/10 | 5/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base Q5_1 | All languages | Yes | 9/10 | 5/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base Q8_0 | All languages | Yes | 9/10 | 5/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny | All languages | Yes | 9/10 | 3/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny Q5_1 | All languages | Yes | 9/10 | 3/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny Q8_0 | All languages | Yes | 9/10 | 3/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base Q5_1 en | English only | No | 9/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Base Q8_0 en | English only | No | 9/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium en | English only | No | 7/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium Q5_0 en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Medium Q8_0 en | English only | No | 7/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small Q5_1 en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Small Q8_0 en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny Q5_1 en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
| Whisper Tiny Q8_0 en | English only | No | 8/10 | 1/10Every word from every language counted together, not an average of per-language scores. |
Charts
The plots generated alongside this run. Select a chart to open it full size.
