Benchmarks

Every benchmark run we have conducted, comparing speech-to-text models by accuracy, speed, and resource usage.

All models at a glance

Ratings computed from combined benchmark data across all runs, scaled 1 to 10. Accuracy is based on Word Error Rate (WER) and does not include punctuation yet.

Name ⇅	Samples ⇅	Disk ⇅	Avg RAM ⇅	Lang ⇅	Translate ⇅	Speed ⇅	Accuracy ↓
Large V3	50	3.0 GB	4.0 GB	all	✔	5	10
Large V3 Turbo q5_0	200	574 MB	800 MB	all	✘	7	9
Large V3 q5_0	200	1.1 GB	2.0 GB	all	✔	6	9
Parakeet TDT v3	200	500 MB	80 MB	25	✘	10	9
Large V1	50	3.0 GB	4.0 GB	all	✔	5	9
Large V2	50	3.0 GB	4.0 GB	all	✔	5	9
Large V2 q5_0	50	1.1 GB	2.0 GB	all	✔	6	9
Large V2 q8_0	50	1.6 GB	2.5 GB	all	✔	6	9
Large V3 Turbo	50	1.6 GB	1.9 GB	all	✘	7	9
Large V3 Turbo q8_0	50	834 MB	1.1 GB	all	✘	7	9
Medium full	50	1.5 GB	2.1 GB	all	✔	7	9
Medium q5_0	50	514 MB	1.1 GB	all	✔	7	9
Medium q8_0	50	766 MB	1.4 GB	all	✔	7	9
Small q5_1	200	181 MB	476 MB	all	✔	8	7
Small full	50	465 MB	807 MB	all	✔	8	7
Small q8_0	50	252 MB	558 MB	all	✔	8	7
Base full	50	142 MB	334 MB	all	✔	9	5
Base q5_1	50	57 MB	219 MB	all	✔	9	5
Base q8_0	50	78 MB	247 MB	all	✔	9	5
Tiny full	50	75 MB	224 MB	all	✔	9	3
Tiny q8_0	50	42 MB	174 MB	all	✔	9	3
Tiny q5_1	50	31 MB	157 MB	all	✔	9	2
Base full en	50	142 MB	333 MB	en	✘	8	1 (en: 9)
Base q5_1 en	50	57 MB	217 MB	en	✘	9	1 (en: 9)
Base q8_0 en	50	78 MB	247 MB	en	✘	9	1 (en: 9)
Medium full en	50	1.5 GB	2.1 GB	en	✘	7	1 (en: 10)
Medium q5_0 en	50	514 MB	1.1 GB	en	✘	8	1 (en: 10)
Medium q8_0 en	50	766 MB	1.4 GB	en	✘	7	1 (en: 10)
Small full en	50	465 MB	807 MB	en	✘	8	1 (en: 10)
Small q5_1 en	50	181 MB	475 MB	en	✘	8	1 (en: 10)
Small q8_0 en	50	252 MB	558 MB	en	✘	8	1 (en: 10)
Tiny full en	50	75 MB	224 MB	en	✘	8	1 (en: 9)
Tiny q5_1 en	50	31 MB	157 MB	en	✘	8	1 (en: 9)
Tiny q8_0 en	50	42 MB	173 MB	en	✘	8	1 (en: 9)

All models compared

Bar chart comparing all model accuracy across benchmark conditions. — Accuracy by model and test condition

Bar chart comparing transcription speed across all models and conditions. — Speed comparison across conditions

Bar chart showing average accuracy per model across all benchmarks. — Average accuracy by group

All benchmark runs

May 9, 202634 models50 samples
Initial model benchmarks to see if there are some models that we should leave out for a more extensive benchmark that will include way more samples. This does not include the Tiny/Base models which has been tested initially already
May 9, 202612 models50 samples
Triage of Tiny and Base model families (all quantization and language variants) to evaluate whether these smaller models are worth further benchmarking.
FeaturedMay 8, 20264 models200 samples
Comparison of Codictate's curated speech models (Small q5_1, Large V3 Turbo q5_0, Large V3 q5_0, Parakeet) to determine the best-performing default model.

All models at a glance

All models compared

All benchmark runs

Initial model benchmarks to see if there are some models that we should leave out for a more extensive benchmark that will include way more samples. This does not include the Tiny/Base models which has been tested initially already

Triage of Tiny and Base model families (all quantization and language variants) to evaluate whether these smaller models are worth further benchmarking.

Comparison of Codictate's curated speech models (Small q5_1, Large V3 Turbo q5_0, Large V3 q5_0, Parakeet) to determine the best-performing default model.