▶️ LISTEN TO THIS ARTICLE

The Training Data Problem: Why What Models Learn From Matters More Than How Much

By Tyler Casey · AI-assisted research & drafting · Human editorial oversight
@getboski

One of the AI industry's defining bottlenecks is shifting from architecture and compute to something far less glamorous: the data itself. As high-quality human-generated text becomes harder to source and synthetic content spreads across the web, the field faces a convergence of risks: quality, contamination, ownership, and collapse.

The Hidden Ingredient

GPT-4 and Llama 3 differ less in architecture than most people assume. Both are dense transformer models. Both use variants of attention mechanisms published years ago. Both were trained on massive GPU clusters using well-understood optimization techniques. The meaningful divergence is in what they learned from: the composition, curation, and provenance of their training data.

This has always been true, but the industry spent years treating data as a logistics problem rather than an engineering one. The Chinchilla scaling laws published in 2022 argued, within the experiments studied, that a given compute budget has a better balance of model size and training tokens. Train a model too large on too little data, and you waste compute. Train too small on too much, and you hit capacity limits. The insight was elegant and quantifiable. It was also incomplete.

Chinchilla treated tokens mostly as interchangeable units for scaling-law purposes. A token from a peer-reviewed paper, a token from a Reddit thread, and a token from a machine-generated SEO farm all counted the same toward the ratio. Subsequent data-quality work has made that simplification harder to ignore.

Quality Over Quantity

The first crack in the "more data is better" orthodoxy came from measuring what happens when you actually filter.

DataComp-LM (DCLM) is one of the clearest attempts to isolate the effect of data quality on language model performance. The project used a large Common Crawl-derived pool and tested filtering pipelines including deduplication, heuristic quality scoring, and model-based selection. In its reported setup, filtering alone improved MMLU scores by 6.6 points over the unfiltered baseline, with no changes to model architecture, training procedure, or compute budget. The same model family, the same number of training steps, different capabilities. The key variable was which tokens made the cut.

In its 2024 paper, this result extended Chinchilla's framework in a direction its authors gestured at but never formalized. Recent work on scaling laws has proposed a Q parameter, a quantitative measure of data quality that modifies the traditional compute-optimal scaling relationship. The practical idea is straightforward: higher-quality data can reduce the amount of raw token volume needed for a given target. Quality doesn't just help. Under some assumptions, it can substitute for quantity on a measurable curve.

The implications are practical. A team with access to a smaller but carefully curated dataset can sometimes match or exceed the performance of a team training on a much larger but noisier corpus. FineWeb, Hugging Face's open dataset effort, demonstrated that aggressive deduplication and quality filtering on Common Crawl data could produce training sets that outperformed much larger unfiltered alternatives in reported evaluations.

Sub-scaling laws push this further. Research studying over 400 models found that data redundancy produces diminishing returns that follow predictable patterns, what the authors call data density effects. Adding more data helps, but the marginal value of each additional token declines as a function of how similar it is to data the model has already seen. Beyond a certain density threshold, more data doesn't just stop helping. It can actively degrade efficiency by forcing the model to allocate capacity to memorizing duplicates rather than learning generalizable patterns. The practical upshot: a 10x increase in dataset size can still yield far less than linear gains, and only if the additional data introduces genuinely new information.

This convergence (the Q parameter, DCLM's filtering results, sub-scaling diminishing returns) points toward a single conclusion. The Chinchilla insight was right but underspecified. Compute-optimal training isn't just about how many tokens you train on. It's about which tokens, selected how, from what sources.

The Synthetic Mirage

The obvious response to a data scarcity problem is to manufacture more data. Generate synthetic training examples using existing models, filter for quality, and feed them back into training. This approach is seductive, widely practiced, and more dangerous than it appears.

The most comprehensive study of synthetic data mixing to date examined over 1,000 models trained with varying proportions of synthetic data. The finding is specific and important: there exists a sweet spot at approximately 30% synthetic data in the training mix. Below that threshold, synthetic data genuinely helps. It provides diversity, fills gaps in underrepresented domains, and regularizes training. Above it, performance degrades. The models begin to lose the distributional richness that comes from human-generated text, replacing it with the narrower, smoother distributions that characterize machine output.

Thirty percent isn't a universal constant. It varies with the quality of the generative model, the domain, and the downstream task. But the existence of a ceiling, and the consistency with which it appeared across a thousand model configurations, should temper the enthusiasm for synthetic data as a scaling solution.

The deeper risk is model collapse. A landmark study published in Nature in 2024 demonstrated that when models are recursively trained on their own outputs, or on data contaminated with outputs from previous model generations, the tail distributions of the training data progressively vanish. Rare but important patterns get smoothed away. Each generation of model produces slightly less diverse outputs, which become slightly less diverse training data for the next generation, in a compounding cycle that the authors characterized as irreversible without intervention.

This isn't a hypothetical concern. The modern web is already saturated with AI-generated content. Any large-scale web crawl conducted today, the kind that feeds Common Crawl and, by extension, most training pipelines, contains a nontrivial and growing fraction of machine-generated text. The proportion is difficult to measure precisely, which is part of the problem.

Recent work on detection offers partial relief. SIGMA, a spectral analysis method, can detect the onset of model collapse in training data by analyzing the eigenvalue distribution of text representations. The technique identifies when a dataset's spectral signature begins to narrow, a mathematical fingerprint of the distributional smoothing that precedes collapse. SIGMA works as an early warning system rather than a cure, but early warning is valuable when the alternative is training a billion-dollar model on data that has already begun to degrade.

The honest assessment: synthetic data is a tool, not a solution. Mixed at the right ratio with high-quality human data, it works. Treated as a replacement for human data at scale, it produces models that are fluent, confident, and subtly wrong in ways that compound over time.

Contamination: The Numbers We Cannot Trust

There is a quieter crisis running alongside data quality and synthetic contamination, and it undermines something more fundamental: the ability to measure progress at all.

Benchmark contamination occurs when test set examples leak into training data. A model that has seen GSM8K math problems during training will score higher on GSM8K, not because it learned to reason mathematically, but because it memorized the answers. The effect is invisible in standard evaluations. The model's outputs look correct. The benchmark numbers go up. The capability improvement is an illusion.

Recent work on inference-time decontamination quantified one version of the problem. By applying decontamination techniques at evaluation time, methods that detect and discount memorized responses, a proof-of-concept experiment found that benchmark contamination could inflate GSM8K accuracy by 22.9% and MMLU scores by 19% when test data leaks into training. These are not small corrections in that setting. They point to a risk of overestimating model capabilities when benchmark overlap is not controlled.

The implications cascade. If a benchmark score is inflated by contamination, comparisons between models on that benchmark become less reliable. Scaling law extrapolations based on those scores become less reliable too. Corporate decisions about which model to deploy, made mainly on the basis of benchmark rankings, can be built on contaminated evidence. The Benchmark Crisis explores how this saturation dynamic, combined with gaming incentives, is pushing the industry toward a post-benchmark evaluation model.

A counterpoint deserves honest treatment. Research on the forgetting effect found that contamination can be "washed out" in some training regimes. At approximately 5x the Chinchilla-optimal token count in the studied setup, models trained on contaminated data converged toward the same performance as models trained on clean data. The memorized answers appeared to be overwritten by broader learning. This is more reassuring for very large training runs than for smaller models, fine-tuned checkpoints, and specialized deployments that operate well below that threshold.

The contamination problem is largely a data provenance problem. When training corpora are assembled from web crawls containing billions of documents, verifying that no test set overlap exists is computationally expensive and hard to do completely at scale. The Pile, one carefully curated open training set, still contained benchmark overlaps that were only discovered after release. That should make readers cautious about assuming unaudited commercial datasets are clean.

Who Owns the Data

The legal situation around training data has shifted from theoretical concern to active litigation.

The New York Times' lawsuit against OpenAI is the highest-profile case, but it represents a broader pattern. The Times alleges that OpenAI's models can reproduce near-verbatim passages from its articles, suggesting that the training process constitutes copyright infringement rather than fair use. OpenAI's defense rests on the argument that training on copyrighted material constitutes fair use: the model doesn't store articles, it learns statistical patterns. The legal question remains unresolved, and its outcome will shape the economics of AI training for years.

Meanwhile, the data itself has acquired a price. Reddit disclosed $203 million in data licensing deals ahead of its IPO, agreements with AI companies for access to its corpus of human conversations, opinions, and expertise. Stack Overflow's licensing deal with OpenAI sparked a revolt among contributors who argued that their volunteer contributions were being monetized without consent. The pattern is consistent: platforms that accumulated human knowledge as a free byproduct of their services are now treating that knowledge as a sellable asset.

The supply side is finite. Epoch AI's analysis estimates that approximately 300 trillion tokens of quality human-generated text exist, with exhaustion projected between 2026 and 2032 depending on growth rates and quality thresholds. That projection treats all currently accessible text as available for training, an assumption that becomes less tenable with every licensing deal and lawsuit that restricts access.

The convergence is uncomfortable. The total supply of quality human text is bounded. Legal and economic pressures are restricting access to existing supplies. Synthetic data fills some gaps but introduces its own failure modes at scale. And the demand for training data continues to grow with every new model generation, every fine-tuning run, every domain adaptation. Something in this equation has to give.

What Comes Next

The labs that have internalized the training data problem are already building around it.

DeepSeek's approach is instructive. Rather than competing only on raw compute or dataset size, DeepSeek invested heavily in data curation pipelines: multi-stage filtering, domain-specific quality classifiers, careful deduplication, and mixing strategies optimized through extensive ablation studies. The result was a family of models that, in several public comparisons, matched or exceeded competitors trained with larger reported budgets. DeepSeek treated data curation as a first-class engineering discipline, not a preprocessing step.

This may be an emerging competitive moat. Architecture innovations can diffuse rapidly. A new attention mechanism published in January may be implemented in open-source libraries by March. Compute advantages are temporary and capital-intensive. But a high-quality, well-curated, legally clear training dataset is difficult to replicate, difficult to audit from the outside, and can compound in value as filtering techniques improve. Labs that build better data pipelines may build better models, not because data is the only variable, but because it is hard to copy and easy to get wrong.

The downstream consequences extend to applications built on top of these models. Agents are only as capable as their training data, and the gap between models trained on well-curated data and models trained on noisy web crawls can show up as the difference between agents that reason reliably and agents that confabulate under pressure. Budget-aware training strategies amplify the quality problem. If you are allocating less compute per query, each unit of compute needs to draw on better-learned representations. Self-evolving agents need high-quality training signal to evolve in productive directions rather than drifting toward the mean of their training distribution.

The Era of Data Engineering

For most of the deep learning era, the prevailing assumption was that scale would solve quality. Throw enough data at a large enough model with enough compute, and the noise washes out. This assumption was productive for a time. It is no longer sufficient.

The training data problem is real, multidimensional, and defining. Quality degrades scaling efficiency. Synthetic data helps within strict bounds but collapses beyond them. Contamination undermines the metrics used to measure progress. Legal constraints are tightening supply. And the total stock of quality human text has a ceiling that the field is approaching.

But the problem is also tractable. Some contamination can be washed out at scale. Synthetic data can work when mixed at disciplined ratios. Spectral methods may detect collapse before it compounds. And data curation pipelines, unglamorous, painstaking, deeply technical, produce measurable, reproducible gains. DeepSeek is a case study in making data quality and systems engineering count. DCLM is evidence that filtering can sometimes substitute for adding more raw tokens.

Labs that treat data as an engineering problem, with the same rigor, tooling, and investment currently reserved for model architecture and distributed training, are better positioned for the next era of AI capabilities. The rest may wonder why their models plateau despite having more parameters and more compute than ever before.

Part of the answer is likely to be in the data. It often was.

Sources

Research Papers:

Industry / Case Studies:

Commentary:

Related Swarm Signal Coverage: