Impact of Quality Filters and Data Preprocessing on Training

Impact of Quality Filters and Data Preprocessing on Training

Introduction

• As large language model training scales to web-level corpora, data preprocessing emerges as a first-order modeling decision rather than a preparatory step. Large, heterogeneous datasets inevitably contain duplicated content, structural artifacts, language inconsistencies, and semantically weak text, all of which directly influence optimization dynamics and learned representations.

• Prior work on Turkish legal language models, notably Mecellem Models, demonstrates that improvements in pre-training loss do not reliably predict downstream performance. In both continual pre-training of decoder models and encoder pre-training from scratch, the best retrieval and domain performance often occurs before loss convergence, indicating that data quality and structure shape representation geometry beyond what likelihood metrics capture.

• This work systematically analyzes how individual preprocessing operations affect training stability, convergence behavior, and downstream utility. We study the cumulative impact of cleaning, quality filtering, language identification, URL-level safety filtering, morphology-aware filtering for Turkish, and semantic deduplication across both decoder and encoder training regimes.

• We treat preprocessing thresholds as tunable hyperparameters and evaluate them empirically rather than heuristically. Controlled parameter sweeps and staged ablations are used to quantify trade-offs between noise reduction, linguistic diversity, token retention, and semantic fidelity, enabling principled selection of operating points for different training objectives.

• Our results show that well-balanced preprocessing pipelines act as an implicit regularizer during training.Appropriate filtering improves convergence smoothness, reduces perplexity across legal subdomains, and enhances retrieval performance, while overly aggressive filtering degrades semantic diversity and downstream generalization despite continued improvements in language modeling loss.

Preprocessing Strategy and Design

In the experiment, we applied different types of preprocessing operations, where each operation focused on addressing a specific problem that hinders the learning process and negatively affects model performance. Below, we explain in more detail how each operation was applied and how we arrived at the final configuration of the preprocessing pipeline.

Cleaning and Normalization

All datasets underwent an initial cleaning and normalization stage designed to remove malformed, structurally invalid, or linguistically weak samples before any higher-level filtering was applied. This stage discarded records with missing or invalid fields, eliminated repeated entries within processing batches, filtered out overly short texts that lack sufficient linguistic signal, and excluded excessively long documents that were likely noisy, out-of-distribution, or dominated by boilerplate. In addition, HTML-derived samples whose structure consisted primarily of tables, navigation elements, or image references rather than natural language were removed, ensuring that the remaining corpus consisted predominantly of continuous textual content suitable for representation learning and language modeling.

Quality Filtering 

The FineWeb Quality Filter was applied to address structural noise at scale by evaluating documents using multiple indicators, including punctuation patterns, line-length distributions, character-duplication ratios, and newline- or list-like structural dominance. Rather than relying on default parameters, we performed systematic threshold tuning through controlled parameter sweeps on a balanced evaluation set constructed via five-fold sampling across content categories, with each category treated as a class and capped at one hundred examples per fold. Candidate values for ratio-based parameters were explored using a compact, binary-tree–style grid centered around the default settings, enabling us to analyze fold-averaged drop rates and per-indicator contributions while avoiding overly aggressive filtering. The final configuration, which balanced effective noise removal with minimal false positives, was selected based on this empirical evaluation.

Table (I) presents the results obtained using different configurations, illustrating how each setting affected both individual filtering criteria and the overall data reduction percentage.

URL-Based Filtering

To improve dataset safety and topical relevance, we extended the cleaning process by applying URL-level content filtering based on curated keyword patterns associated with adult, explicit, or gambling-related sources. This filtering operated directly on web data samples paired with URL strings, enabling the early exclusion of entire classes of unsafe or irrelevant content at the source level. To mitigate false positives particularly for legitimate news or informational sources that may contain ambiguous substrings we employed an allowlist of known domains and terms that were exempt from filtering. In large-scale collections, this stage resulted in substantial reductions in dataset size, demonstrating the effectiveness of source-level signals when safety and domain relevance are prioritized.

Language Filtering

At this stage, we performed document-level language identification by retaining only samples whose primary predicted language matched the target language set with sufficient confidence, thereby enforcing linguistic consistency. Documents exhibiting low-confidence predictions or mixed-language signals were filtered out entirely. This step was particularly important for embedding model training, where even small amounts of cross-lingual noise can degrade semantic alignment and downstream retrieval performance.

Turkish-Specific Quality Filtering

For Turkish corpora, we introduced morphology-aware quality indicators to capture linguistic properties that generic filters cannot detect. Using Zemberek-based morphological analysis, we computed suffix entropy over nominal case tags extracted from noun and verb tokens, measuring the balance and diversity of morphological case usage within each document. In parallel, lemma diversity was calculated as the ratio of unique lemmas to total analyzed tokens, capturing lexical variety after morphological normalization. Low values of these metrics were indicative of templated, repetitive, or boilerplate text, while unusually high values often signal noisy or out-of-domain content. By jointly sweeping suffix entropy and lemma diversity thresholds, we were able to identify operating points that preserved well-formed, sentence-like prose while removing morphologically impoverished or anomalous samples, providing a principled basis for Turkish-specific corpus quality assessment.

II. Morphological Quality Filtering Heatmap: Remaining Tokens by Threshold Combinations

Deduplication

Deduplication was applied as a two-stage process to remove both exact and near-duplicate content that can otherwise inflate token counts without contributing meaningful information. Exact match deduplication was performed using document-level text hashing, identifying and removing verbatim copies introduced through crawling overlap, mirroring, or OCR artifacts. To further reduce redundancy, semantic deduplication was applied using SemHash with dense embedding similarity, where documents exceeding a similarity threshold of 0.75 were considered near-duplicates and removed. This step proved particularly effective in filtering documents replicated across multiple domains with only superficial differences such as headers or footers, while preserving legitimate variations in content.

LLM vs Embedding Training

Preprocessing strategies were deliberately adapted to the downstream training objective. For large language model training, the pipeline emphasized structural quality and safety while tolerating greater linguistic variation to preserve broad coverage and token volume. In contrast, embedding model training required stricter language consistency, aggressive HTML-variant removal, and document-level aggregation to produce semantically coherent inputs. In datasets derived from page-level OCR sources, document merging significantly reduced row counts by consolidating multiple pages into single documents, resulting in improved semantic consistency at the cost of reduced sample granularity.

Observations and Impact on Training

The experiments consistently showed that data preprocessing choices had a first-order effect on training dynamics. For decoder models trained via continual training, progressively refined preprocessing pipelines translated into markedly smoother loss curves, reduced variance during long training runs, and faster stabilization after phase transitions. In the multi-phase CPT setup for Qwen3-1.7B, early phases that relied on aggressively cleaned, deduplicated, and language-consistent corpora converged rapidly and provided stable parameter initialization for later, more complex legal data, preventing the sharp loss spikes and instability typically observed when domain-specific data is introduced abruptly. As the curriculum advanced toward longer and more structurally complex legal documents, the model retained previously acquired general language competence while steadily improving domain-specific perplexity, indicating that preprocessing acted as an implicit regularizer against catastrophic forgetting rather than merely a filtering mechanism.

The impact of preprocessing was even more apparent when examining perplexity trends across legal subdomains. Each additional preprocessing stage semantic deduplication, FineWeb quality filtering, URL-based safety filtering, and Turkish-specific morphological filtering corresponded to consistent perplexity reductions across nearly all evaluated legal areas, with the largest gains observed in terminology-dense and structurally rigid domains such as tax law, personal data protection law, and corporate regulation. Notably, these improvements accumulated monotonically across CPT phases, demonstrating that the benefits of preprocessing compound over training rather than saturating early. Larger models such as Qwen3-4B benefited even more strongly from clean, well-structured corpora, achieving substantial perplexity reductions in a single-phase CPT setting, suggesting that model capacity amplifies the returns of high-quality data rather than compensating for noisy inputs.

For encoder models trained from scratch, preprocessing quality proved decisive not only for training stability but also for downstream utility. Iterative refinement of the pre-training dataset showed that reductions in masked language modeling loss alone were not reliable predictors of embedding performance. While successive dataset versions consistently lowered MLM validation loss, downstream retrieval performance followed a non-monotonic trajectory, peaking at an intermediate preprocessing configuration before regressing when filtering became overly aggressive or when continued pre-training exceeded the optimal convergence window. This divergence highlights a critical insight: preprocessing influences not just optimization efficiency, but the geometry of the learned representation space. Over-filtered or excessively homogenized corpora may improve likelihood objectives while simultaneously degrading semantic separability and domain generalization in embedding tasks.

Morphology-aware filtering for Turkish further illustrated this point. Introducing suffix entropy and lemma diversity as corpus-level quality signals selectively removed templated and repetitive content that generic filters could not detect, leading to measurable improvements in retrieval performance on legal benchmarks. However, aggressive thresholding quickly collapsed lexical diversity and reduced token length distributions, reinforcing that linguistic richness must be preserved alongside cleanliness. The optimal preprocessing regime therefore emerged not as the most restrictive configuration, but as a carefully balanced one that maximized structural integrity while maintaining morphological and semantic variety.

Conclusion

The results presented in this work demonstrate that data preprocessing is not a passive or preliminary step in large-scale model training, but a core modeling decision that directly shapes learning dynamics, representation quality, and downstream performance. Across both continual pre-training of decoder models and pre-training of encoder models from scratch, carefully designed preprocessing pipelines consistently led to faster convergence, more stable optimization, lower perplexity in domain-specific evaluations, and substantially improved semantic alignment in embedding spaces. Conversely, overly aggressive filtering or misaligned preprocessing objectives were shown to harm generalization, even when conventional training metrics such as language modeling loss continued to improve.

A central lesson emerging from these experiments is that preprocessing thresholds should be treated as tunable hyperparameters rather than fixed heuristics. Optimal configurations depend on model scale, training objective, and domain characteristics, particularly in morphologically rich and structurally formal languages such as Turkish. Language-aware filtering, semantic deduplication, and curriculum-aligned data staging proved especially effective when integrated holistically rather than applied in isolation. Moreover, the observed decoupling between pre-training loss and downstream utility reinforces the need to evaluate preprocessing decisions using task-relevant metrics rather than relying solely on optimization signals.

Ultimately, as language models continue to scale and training budgets grow, the marginal gains from architectural changes will increasingly depend on the quality, structure, and linguistic integrity of the data they consume. The findings here suggest that disciplined, empirically grounded preprocessing pipelines offer one of the highest-leverage opportunities for improving model behavior, efficiency, and reliability. In this sense, data preprocessing is not merely about cleaning the past, but about actively shaping the future capabilities of large language models.

References

1. Uğur, Ö., Göksu, M., Çimen, M., Yılmaz, M., Şavirdi, E., Demir, A. T., Güllüce, R., Çetin, İ., & Sağbaş, Ö. C. (2026). Mecellem Models: Turkish models trained from scratch and continually pre-trained for the legal domain. arXiv preprint arXiv:2601.16018. https://arxiv.org/abs/2601.16018

2. Akın, A. A., & Akın, M. D. (2007). Zemberek, an open source NLP framework for Turkic languages. Structure, 10, 1–5.

3. Kargaran, A. H., Imani, A., Yvon, F., & Schütze, H. (2023). GlotLID: Language identification for low-resource languages. Findings of the Association for Computational Linguistics: EMNLP 2023, 6155–6218.

4. MinishLab. (2025). SemHash: Fast multimodal semantic deduplication and filtering. GitHub repository. https://github.com/MinishLab/semhash

5. Penedo, G., Kydlíček, H., Cappelli, A., Wolf, T., & Sasko, M. (2024). DataTrove: Large-scale data processing. Zenodo. https://doi.org/10.5281/zenodo.14290813

6. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x