Evaluation report · November 2026
Measured on phone audio, not on demos.
Three public Turkish test sets, 500 spoken addresses and 1,273 synthesis sentences, scored the same way for every system. Lower word error rate is better.
3.42%
WER on read speech
FLEURS Turkish test set, 743 clips. Lowest in the comparison.
Fig. 2
1.68%
WER on spoken addresses
500 addresses, 8 kHz, with place hints.
Table 3
19.5×
faster than real time
Files through the Türkiye API, fast mode.
Table 3
3.38%
TTS word error rate
1,273 sentences of the open mihuai/turkish-tts-eval set.
Fig. 3
FLEURS tr_tr test · 743 clips
mihu-umay-v1
production API
openai/whisper-large-v3
multilingual, large
turkmedstt/whisper-large-v3-turkish-general
Turkish fine-tune of whisper-large-v3
balbal-v2-manas
Turkish, commercial use under conditions
openai/whisper-large-v3-turbo
multilingual, turbo
balbal-v2
Turkish, commercial use under conditions
Qwen/Qwen3-ASR-1.7B
multilingual, 1.7B
PatientDesk/Duyu-small
Turkish, small
Figure 2. Word error rate in %, lower is better, Turkish, medium scoring rule (number and format variants accepted). mihu-umay-v1 ranks first of 8 on read speech. Bars past the axis are cut and marked →.
Text-to-speech intelligibility
mihuai/turkish-tts-evalFigure 3. Each system reads the same 1,273 call-centre sentences of the open mihuai/turkish-tts-eval dataset; an independent recogniser transcribes the audio. Word error rate in %, lower means the speech is easier to understand. Seven Turkish TTS systems; every chart and table is on the benchmarks page.
Table 2. Naturalness of the same text-to-speech audio. Predicted quality scores, higher is better; best in each column in bold.
| System | UTMOSv2 | SCOREQ | DNSMOS Pro |
|---|---|---|---|
| mihu-ses-v1 | 3.31 | 4.10 | 3.64 |
| Antalia-2 Mini | 2.78 | 3.36 | 3.63 |
| Trendyol-TTS | 2.88 | 4.25 | 3.91 |
| EMA Lightning 1.1.2 | 2.68 | 3.54 | 3.25 |
| Anka TTS 0.1.6 | 2.66 | 3.55 | 3.93 |
| FreyaTTS-small | 2.45 | 2.57 | 3.18 |
mihu-ses-v1 has the highest UTMOSv2 score overall and in all 13 sentence categories of the set. The three predictors do not agree with each other and are not a substitute for a human listening test.
Two Turkish TTS test sets
call centre vs everyday speech
Our reference set for contact-centre speech, modelled on real customer conversations: amounts, account and ID numbers, addresses, e-mail and web addresses, spelled-out names and mixed formats. Hard by design.
An everyday conversational set: short sentences of 3 to 13 words about daily life, questions and exclamations, with few numbers or codes. Easier, so every system scores lower here.
Word error rate on freya-tr-eval, %
Same scoring as Figure 3. On the conversational set the leading systems are within a fraction of a percent of each other; the call-centre set is where systems separate.
Figure 4. The call-centre set measures what a contact centre needs read correctly; the everyday set measures plain conversational speech. Lower word error rate is better.
Table 3. Live service figures, Türkiye region, 3 November 2026.
| Measurement | Setting | Result |
|---|---|---|
| File transcription speed, 743 clips | fast mode | 19.5× real time |
| File transcription speed, 743 clips | normal mode | 8.5× real time |
| Failed, missing or empty results | 743 clips | 0 |
| Address word error rate, 500 addresses, 8 kHz | without → with place hints | 2.22 % → 1.68 % |
| Address sentences fully correct, 8 kHz | without → with place hints | 92.6 % → 97.2 % |
| Neighbourhood heard right, 8 kHz | without → with place hints | 95.6 % → 98.2 % |
Method. Medium scoring rule throughout. Phone: a public Turkish telephone test set, used for testing only. Media: only the 502 MediaSpeech clips held out from mihu training count. Speed is end-to-end through the public API, network included. Address audio is synthetic.
Want your system in these tables?
Send us API access or the model and we will run it on the same test sets with the same scoring, show you the results first, and publish them here, whatever they are.