Model Collapse and Data Pollution: The Long-Term Risks of Training on Synthetic and Adversarial Data

NeuralStack | MS – AI Security Engineering · Reading time: approx. 10 minutes
TL;DR: Two distinct failure modes threaten the integrity of future AI systems. Model collapse is a statistical degeneration that occurs when models learn from their own outputs. Data poisoning is a deliberate attack in which an adversary plants content in the training corpus. Both exploit the same structural weakness: the open web is no longer a trustworthy, human-authored data source, and training pipelines have not yet adapted to that fact.
1. The Problem: A Corpus That Is Eating Itself
Large language models were built on a simple premise: scrape the public internet, filter it, train on it. That premise held while the internet was overwhelmingly human-written. It no longer does. Generated text, images and code now circulate at scale, and every future training run that crawls the web will ingest some fraction of it.
At the same time, high-quality human-written text is a finite resource. Labs therefore turn to synthetic data deliberately for distillation, instruction tuning, reasoning traces and domain-specific augmentation. Synthetic data is cheap, scalable and often hard to distinguish from human writing.
This creates two long-term risks that are frequently conflated but should be analyzed separately:
| Model collapse | Data poisoning | |
|---|---|---|
| Cause | Recursive training on model-generated data | Deliberate injection of malicious content |
| Actor | None (emergent, statistical) | Adversary |
| Symptom | Loss of diversity, forgotten rare knowledge, degraded factuality | Backdoors, targeted misbehavior, steered narratives |
| Detectability | Gradual, diffuse | Potentially trigger-specific and hidden |
| Primary defense | Data composition and provenance | Data vetting, auditing, behavioral testing |
From a security perspective, the dangerous scenario is their combination: an ecosystem polluted by benign synthetic content is precisely the environment in which malicious content is hardest to spot.
2. Model Collapse: What the Research Actually Shows
2.1 The mechanism
Model collapse describes the degenerative process in which generative models, trained recursively on synthetic data from earlier generations, progressively lose information about the true data distribution. The first casualty is the tail: rare events, minority viewpoints, unusual phrasing and edge cases. Because a model samples from its learned distribution, rare events are under-sampled; the next generation sees even fewer of them, and the effect compounds. In the limit, output variance shrinks and the model converges on repetitive, low-diversity content.
The canonical result is the study by Shumailov et al., later published in Nature. In the original experiments, language models fine-tuned repeatedly on their own outputs degraded generation by generation; after nine rounds, the output was gibberish. The authors stress that the effect is not specific to language models and applies to generative models more broadly.
2.2 Early versus late collapse
The literature distinguishes an early stage, in which the model loses tail information while aggregate metrics still look healthy, from a late stage with visible, severe degradation. This is the part that matters operationally: early collapse is invisible to standard benchmarks. A model can retain its average-case performance while quietly becoming worse at precisely the rare, high-stakes inputs that security and safety teams care about.
2.3 Fluency survives; facts do not
Recent work on knowledge collapse (Keisha et al., 2025) sharpens this point. Under recursive synthetic training, the authors identify three distinct stages and show that fluency can persist while factual accuracy erodes. They also find that the onset of collapse depends on the instruction format, which they describe as a conditional, prompt-dependent form of instruction-following collapse. Their proposed mitigation, domain-specific synthetic training, delayed accuracy decay by a factor of 15 compared to general synthetic training. A caveat is warranted: the experiments used a small model (Gemma 3 1B), so extrapolation to frontier scale should be treated with care.
The implication for practitioners is uncomfortable. A collapsing model does not fail loudly. It continues to sound confident, and that is exactly the failure mode that is hardest to catch in production.
2.4 Is collapse inevitable? No, but the regime matters
The popular narrative that "AI will inevitably eat itself" is too broad. The key distinction is replace versus accumulate:
- Replace: each generation is trained only on the previous generation's output, discarding the original data. Collapse reliably occurs.
- Accumulate: synthetic data from each generation is added to the original real data. Gerstgrasser et al. (2024) show that this avoids collapse, with test error bounded rather than diverging.
- Augment with a small synthetic fraction: Bertrand et al. (2024) indicate that collapse can be avoided when the proportion of synthetic data is small.
There are, however, important counterpoints:
- Strong model collapse. Dohmatob et al. (2024) show, in a theoretical high-dimensional setting, that even a small fixed fraction of synthetic data whose labels deviate from the true distribution can impose a floor on achievable test error, irrespective of how much real data is added.
- Over-reliance on synthetic data in an accumulation setting can still yield a final model that performs worse than one trained on real data alone (Kazdan et al., 2024).
- Theoretical work on recursive parametric training (Xu et al.) concludes that progressively increasing the sample size at each step is necessary to prevent collapse under mild conditions.
The honest summary: collapse is real, regime-dependent and avoidable, but only with deliberate data governance. Pipelines that quietly drift into the replace regime, for instance by re-crawling a web that is increasingly synthetic, are exposed.
Surveys of the literature also point to maintaining a non-trivial share of real data in each generation as a widely recommended safeguard. Some recent preprints claim that remarkably little real data suffices; I treat such claims as promising but preliminary until independently replicated.
2.5 The distributional equity dimension
Because collapse erodes the tails first, the groups most affected are those already underrepresented: low-resource languages, minority dialects, niche domains. Model quality there degrades first and least visibly. For organizations deploying AI in multilingual or safety-critical settings, this is a fairness issue as much as a quality issue.
3. Data Poisoning: The Adversarial Counterpart
Model collapse needs no attacker. Data poisoning needs only one.
3.1 The scaling assumption that no longer holds
For years, a comforting assumption prevailed: because frontier training sets are enormous, an attacker would need to control a meaningful percentage of the data to influence a model. That would make poisoning of large models impractical.
A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute (October 2025) challenges this directly. In the largest poisoning investigation to date, the researchers found that as few as 250 malicious documents were enough to implant a backdoor in models ranging from 600M to 13B parameters. Roughly 0.00016% of the training tokens were affected. 100 documents were not sufficient to reliably backdoor any model, while 250 or more succeeded across model scales. Larger models, trained on more than 20 times as much clean data, were no harder to compromise.
The central finding: attack success depends on the absolute number of poisoned documents, not on their percentage of the training data.
3.2 Scope and caveats
Precision matters here, and the authors are explicit about it:
- The tested backdoor was deliberately narrow: a trigger phrase causing the model to emit gibberish, a denial-of-service style behavior that is unlikely to pose significant risk in frontier models.
- Models were pretrained at Chinchilla-optimal data volumes, up to 13B parameters.
- It remains an open question whether the same dynamics hold for much larger models or for more complex, higher-stakes behaviors.
The result should therefore be read as a warning about feasibility, not as a demonstration of a catastrophic exploit. Anthropic's own framing is that defenders retain an advantage here, since datasets and trained models can be re-examined, provided defenses are designed around small, constant numbers of malicious samples rather than percentages.
3.3 Attack surface
The practical concern is the open web. Anyone can publish content that may later be absorbed into a training corpus. Two broad patterns are described in the literature:
- Backdoor attacks: specific trigger tokens cause hidden behaviors that are dormant on ordinary inputs.
- Topic steering: biased or false content is seeded so that the model reproduces it as fact, with no explicit trigger at all.
The second is arguably the more insidious. It looks, statistically, much like ordinary noise.
4. Where the Two Risks Meet
Consider the combined picture:
- Dilution of provenance. As synthetic content grows, it becomes harder to attribute any given document to a human author, an honest model or an adversary.
- Laundering through generations. A poisoned or biased claim, once reproduced by a model and republished, re-enters the corpus as apparently independent corroboration.
- Masked symptoms. Collapse-induced degradation and poisoning-induced misbehavior can both present as "model quality issues", complicating incident attribution.
- Tail overlap. The rare, low-frequency regions of the distribution that collapse erodes are also where trigger-based backdoors live. Evaluation that ignores the tails is blind to both.
5. Defensive Practices
The following measures reflect the convergence of the research above. None is a silver bullet; they work as layers.
Data composition
- Accumulate, do not replace. Preserve the original, verified human data as a permanent anchor.
- Treat the real-to-synthetic ratio as a monitored hyperparameter, tracked across training runs.
- Diversify generators. Synthetic data produced by multiple models reportedly yields broader output distributions than single-source generation.
Provenance and governance
- Version synthetic generations. Record which model produced which samples, so contamination can be traced.
- Maintain a curated, frozen reference corpus of pre-synthetic-era human data for evaluation and anchoring.
- Scrutinise third-party and crawled sources by origin, not just by content quality.
Security controls against poisoning
- Plan for absolute counts. Detection thresholds defined as a percentage of the corpus are inadequate; a few hundred documents can matter.
- Scan for anomalous associations, such as rare token sequences consistently co-occurring with unusual continuations.
- Deduplicate and quality-filter before training, and re-audit datasets after the fact, since defenders can re-inspect both data and models.
Evaluation and red teaming
- Measure tails, not just averages. Track distributional diversity and rare-case performance across successive training iterations.
- Test for factuality separately from fluency, given that the two can decouple.
- Red-team for triggers. Probe models with candidate trigger patterns and compare behaviour against controls, for example through perplexity gaps between triggered and clean prompts.
- Evaluate low-resource and minority-domain slices explicitly, since degradation appears there first.
6. Open Questions
Several issues remain unresolved and are worth following:
- Do poisoning dynamics persist at frontier scale and for high-stakes behaviours, including through fine-tuning and post-training stages?
- How can synthetic content be reliably detected or watermarked at web scale, and will adversaries simply strip the markers?
- What is the safe lower bound of real data in practice, as opposed to in idealised theoretical settings?
- How should provenance standards be established across labs, data vendors and the open web?
7. Conclusion
The long-term risk to AI systems is not that models will inevitably consume themselves, nor that a single crafted document will take down a frontier model. The risk is subtler: a training ecosystem whose data lineage is no longer known, in which statistical decay and deliberate manipulation are both difficult to see.
The encouraging aspect of the research is that the failure conditions are reasonably well characterised. Replace-style recursion collapses; accumulation with retained real data does not. Poisoning is feasible at small absolute counts, but datasets and models remain auditable. In other words, these are engineering and governance problems, and they respond to engineering and governance: provenance tracking, deliberate data composition, tail-aware evaluation, and adversarial testing.
For AI security practitioners, the takeaway is to treat training data as part of the attack surface, and to treat its provenance as a security property.
Manuela Schrittwieser – specializing in AI Security Engineering & Technical Writing / Co-authored by Claude AI
About the author
- Programmer, Creative and Tech Nerd. I build with code, write about AI and Software Development and explore the future of intelligent Agents.
Comments
Join the discussionNo comments yet. Be the first to comment.





