Evaluation report · November 2026
Benchmarks
Speech recognition and speech synthesis for live calls, put to the test: Umay and SES in 7 languages, every system scored the same way.
Umay V1 · Ultra-low-latency Multilingual ASR, Yielding streams
Word error rate · Turkish · read speech
WER % · FLEURS tr_tr test set, 743 clips · log scale · lower is better
SES V1 · Streaming Expressive Speech
How well a listener understands each Turkish voice, how natural it sounds and how it handles numbers, addresses and codes.
Word error rate of the spoken output
Turkish, WER % · each system reads the 1,273 sentences of turkish-tts-eval, Whisper large-v3 transcribes them at 8 kHz (phone quality) · log scale · lower is better
How the SES number is reached: each of its 8 Turkish voices read all 1,273 sentences and was scored on its own; the chart shows the best one, Alp (3.38 %), the default voice of the live API. All voices: Alp 3.38 % · Bengü 3.70 % · Ece 3.91 % · Oğuz 3.98 % · Batu 4.07 % · Yağmur 5.19 % · Aysu 6.29 % · Kaya 7.42 %. The other tabs use the same voice. The test set is built from what call centres say: addresses, amounts, dates, codes, abbreviations and everyday speech; e-mail and web addresses are scored separately. Test sentences: turkish-tts-eval tests.
SES · six more languages
SES against open-source models in English, German, Spanish, French, Italian and Russian, with its best voice in each language.
Word error rate · English
100 sentences · Whisper large-v3 at 8 kHz · log scale · lower is better
How the SES number is reached: each of its English voices read the same 100 sentences and was scored on its own; the chart shows the best one, Oliver (1.29 %). All voices: Oliver 1.29 % · James 1.53 % · Emily 1.57 % · John 1.57 %.
SES · speed
Measured on CPU, the same image as the on-premises delivery.
Time to the first sound
SES on CPU, streaming · request to the first audio bytes, median · lower is better
Audio streams while the sentence is still being generated, so the caller hears the answer almost at once.
Method
How the numbers were measured.
| Speech To Text scoring | Word error rate after normalising numbers and formats (both “15” and “on beş” count as right). |
| Text To Speech scoring | Each system reads the test sentences; Whisper large-v3 transcribes the audio at 8 kHz, the quality of a phone line; word error rate against the source text. |
| Naturalness | Three models that each predict a listener's mean opinion score (1–5): UTMOSv2, SCOREQ, DNSMOS Pro. |
| Speech To Text, six more languages | FLEURS test sets (647–908 clips per language), the same clips through a simulated phone line, and Sierra real phone calls (English 817, Spanish 792). Word error rate after normalising numbers and formats. |
| Test sets | turkish-tts-eval (1,273 sentences), Freya-TR-Eval, FLEURS tr_tr, MediaSpeech TR, Turkish phone calls; 100 sentences per language for the other languages. |
| Date | November 2026. |
Want your system in these charts? Send API access or the model to research@mihu.ai and we will run it on the same sets.
Check the numbers
Everything you need to repeat a test is public, except our scoring script, which we send on request.
- 01Take the test sentences
Turkish text to speech: turkish-tts-eval (1,273 sentences). Speech to text: the public FLEURS and MediaSpeech test sets.
- 02Run Umay or SES
Get an API key and send the same audio or text, as described in the docs. Run the other systems on the same input.
- 03Score every system the same way
Speech to text: word error rate against the reference. Text to speech: transcribe the audio with Whisper large-v3 at 8 kHz, then word error rate against the source text. Naturalness: UTMOSv2, SCOREQ, DNSMOS Pro.
- 04Compare with ours
Normalising numbers and formats changes the result by a point or more. Ask research@mihu.ai for our scoring script so your numbers are comparable with this page.
Best test: your own calls. Benchmarks are a starting point; try Umay and SES on recordings from your own lines.