Navigating the Morphological Labyrinth: A Structural Benchmark for Turkish ASR

• In the world of artificial intelligence, "state-of-the-art" (SOTA) model performance is typically measured on massive English-centric datasets. However, languages are not structural copies of one another. When an architecture that works perfectly in high-resource languages encounters a completely different morphological structure, how does it behave?
• Turkish, with its agglutinative structure and rich vowel harmony rules, provides a unique testing ground for ASR (Automatic Speech Recognition) systems. The fact that a single root word can carry meanings equivalent to an entire sentence by taking dozens of suffixes challenges standard Word Error Rate (WER) metrics.
• In this article, we examine the foundations of our "Turkish ASR Leaderboard" project, where we not only create a scoreboard but also analyze the geometric and structural challenges of Turkish through modern ASR architectures (Whisper, Conformer, etc.). Our goal extends beyond showing which model is "the best"—we aim to visualize and understand how well models can follow Turkish's “semantic backbone.”

Figure 1: Turkish Morphological Chain - From Sound Wave to Meaning

The Core Idea: Turkish's Structural Challenge and ASR

The fundamental task of ASR is to convert acoustic signals into text. However, this conversion is not merely about mapping sounds to letters; it's about decoding the logical structure underlying the language.

In Indo-European languages, words are generally isolated and fixed. In Altaic languages like Turkish, the word is a dynamic building block. For example, an extreme case like "görüştüremediklerimizdenmişsinizcesine" may appear as a single word, but for an ASR model, it's a long morphological chain that must be decoded.

This situation creates two fundamental challenges for ASR models:

Vocabulary Explosion: The number of possible word forms is theoretically infinite, rendering classic n-gram-based language models insufficient.

Acoustic Ambiguity of Short Suffixes: Short suffixes that completely change meaning (e.g., the negation suffix -me/-ma) can easily be lost acoustically or misunderstood in rapid speech.

Figure 2: Morphological Chain Example - "Görüştüremediklerimizden"

A single root word can expand with multiple suffixes to create complex semantic structures. For example, "görüştüremediklerimizden" breaks down into: gör (root) → -üş → -tür → -e → -me → -dik → -ler → -imiz → -den.

Measuring a Turkish ASR model's performance is therefore not just an acoustic test but also a morphological endurance test.

Methodology: Building the Benchmark Arena

Just as the RDP (Ramer-Douglas-Peucker) algorithm searches for the "essence" of a curve, we wanted to measure how well ASR models capture the "essence" of Turkish speech. We established a fair and comprehensive structure when creating our leaderboard:

Datasets: Diversity is Essential

Testing models on a single type of data is misleading. Therefore, we created a pool representing different acoustic conditions:

Clean/Read Speech: (e.g., Common Voice Turkish) Controlled environment, clear pronunciation.

Noisy/Spontaneous Speech: (e.g., FLEURS, Media Recordings) Background noise, interruptions, fast and natural speech flow. This shows "real-world" performance.

Diverse Domains: (e.g., Mozilla DeepSpeech Corpus) Different speaking styles and content types.

For our leaderboard, we use three primary test sets:

• Common Voice (CV): Clean, read speech
• FLEURS: Spontaneous, diverse speakers
• Mozilla DeepSpeech Corpus (MDC): Real-world recordings

Models: Architectural Diversity

We included different approaches in the leaderboard:

Transformer-Based Giants: Whisper Medium, Whisper Small (Large-scale, multilingual)

Compact Architectures: Wav2Vec2, XLS-R variants (Faster, fine-tune focused)

CTC-Based Models: MMS-1B (Massively multilingual)

Multimodal Decoders: Voxtral-based models with LoRA adapters

Fine-tuned Turkish Models: Our community includes models specifically fine-tuned for Turkish:

• y0mur/voxtral-turkish-commonvoice-ft: Fine-tuned on CV17 Turkish (26.5k samples)
• y0mur/voxtral-turkish-fleurs-ft: Fine-tuned on FLEURS Turkish (948 samples)
• alikanakar/whisper-synthesized-turkish-4-hour: Fine-tuned on 4 synthesized datasets (210k samples)
• cahya/wav2vec2-base-turkish: Trained on Turkish CommonVoice
• mpoyraz/wav2vec2-xls-r-300m-cv8-turkish: Fine-tuned on CommonVoice v8

These models demonstrate how Turkish-specific optimization can improve performance on morphologically complex structures. We plan to continue expanding this leaderboard with models optimized for different Turkish dialects, accents, and domain-specific scenarios (medical, legal, technical documents).

Metrics: Beyond WER

Standard Word Error Rate (WER) is used, but due to Turkish's structure, Character Error Rate (CER) is also tracked as a critical indicator. Sometimes, even if the model incorrectly separates a word (writing separately what should be written together), CER shows us how close the model is phonetically.

We calculate:

• WER (Word Error Rate): Primary ranking metric
• CER (Character Error Rate): Shows phonetic accuracy
• Average WER: Across all three test sets for fair comparison
• RTF (Real-Time Factor): Inference speed measurement

Results and Observations: The Story the Numbers Tell

Figure 4: Evaluation Pipeline - Turkish ASR Leaderboard

Results show that, as expected, larger models are more acoustically successful. However, the truly interesting findings are the "characteristic behaviors" revealed in error analysis.

Current Leaderboard Standings

Based on our latest evaluation, here are the top performing models:

Figure 4: Turkish ASR Leaderboard - Model Performance Comparison (Average WER %)

Geometric Deviations in Error Analysis

Successful ASR models must be able to capture "semantic turns" (correct root and suffix distinctions) within acoustic noise.

Root vs. Suffix Errors: Some models correctly capture the root word (acoustic success) but tend to make errors in the inflectional suffixes at the end. This shows that the model cannot follow Turkish's "tail" structure.

Hallucinations: Especially large Transformer models tend to fabricate words that are acoustically non-existent but grammatically "plausible" in noisy sections. These are moments when the model disconnects from acoustic data and relies too heavily on its internal language model.

Model Type Patterns:

• Encoder-Decoder models (Whisper): Better context understanding, lower hallucination
• CTC models (Wav2Vec2, MMS): Faster inference, struggle with suffix chains
• Multimodal Decoders (Voxtral): High parameter efficiency with LoRA, variable performance based on training data

Our Mind

This work has shown us once again that developing ASR for a language is not just about collecting more data but understanding the structural "signature" of that language.

Just as the reference article discusses the "geometric nature of embedding space," Turkish also has a "morphological geometry." Suffixes come in a specific order and according to rules. A successful ASR model is one that has internalized these rules. The leaderboard is a report card of this internalization.

The success of fine-tuned models demonstrates the importance of language-specific optimization. Models like alikanakar/whisper-synthesized-turkish and Turkish-optimized Wav2Vec2 variants show better understanding of Turkish's vowel harmony rules, suffix chains, and phonetic characteristics.

In the future, focusing not only on WER scores but also on metrics like "Morphological Error Rate" that separately evaluate roots and suffixes will take Turkish ASR to the next level. We will continue to expand this leaderboard by developing models specific to different Turkish dialects, accent diversity, and domain-specific scenarios (medical, legal, technical documents).

KEY TAKEAWAYS

• Morphological Depth: Turkish ASR is not just an acoustic problem but a deeply morphological one.
• Metric Insufficiency: Standard WER does not fully reflect success in agglutinative languages; CER and morphological analyses are essential.
• Hallucination Risk: Large models can deviate from acoustic reality while trying to "predict" the language structure.
• Fine-tuning Impact: Turkish-specific fine-tuned models consistently outperform generic multilingual models on local datasets.
• Dynamic Map: The leaderboard is not a static result but a living map showing models' interaction with language structure.
• Geometric Analogy: Like semantic flow in embedding space, Turkish's morphological structure also carries a geometric backbone.
• Real-Time Factor: Model efficiency varies significantly—CTC models are ~10-100x faster than Transformer-based models.

CONCLUSION

This article has demonstrated step-by-step how critical morphological structure is in evaluating Turkish ASR systems. The Turkish ASR Leaderboard is not just a performance comparison but a dynamic tool that makes visible the effect of language structure on model behavior. Understanding how modern ASR architectures cope with Turkish's structural complexity provides important insights for future model development processes.

The leaderboard is continuously updated and welcomes community contributions. We encourage researchers and developers to submit their Turkish ASR models for evaluation at: https://huggingface.co/spaces/y0mur/turkish-asr-leaderboard

References

1. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In International conference on machine learning (pp. 28492-28518). PMLR.

2. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33, 12449-12460.

3. Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., ... & Auli, M. (2021). XLS-R: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296.

4. Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., ... & Auli, M. (2024). Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97), 1-52.

5.  Liu, A. H., Ehrenberg, A., Lo, A., Denoix, C., Barreau, C., Lample, G., ... & Tang, Y. (2025). Voxtral. arXiv preprint arXiv:2507.13264.