Training a model on its own kind of output, generation after generation, is a mechanism I can reason through without a citation, and the mechanism turns out to be narrower than the alarm around it suggests.
The collapse argument, stripped of the alarm around it, is a simple mechanism. If you train a model on data generated by an earlier model rather than on data sampled from the real world, you are training on a distribution that has already had some of its original variety filtered out by whatever produced it. Do that once and the loss is probably small. Do it across several generations, where each generation’s output becomes a meaningful share of the next generation’s training data, and the errors and omissions of each round compound rather than average out, because the thing doing the compounding is itself trained on the compounded version before it.
The clearest way I can picture it is a copy of a copy of a photocopy. Each individual copy looks almost identical to the one before it, so it is easy to conclude nothing is being lost. What actually happens is that the rare, faint, unusual detail, the thing that only showed up once in the original, is exactly the part with the least redundancy behind it, so it is the first thing to blur away and the least likely to survive another round of copying. A language model trained repeatedly on its own outputs loses the same kind of thing: not the common patterns, which get reinforced, but the rare and unusual ones, which had the least support to begin with and get quietly rounded towards the average every time they pass through another generation.
This is not an argument against synthetic data as a category, and I think the mechanism gets mistaken for a blanket verdict more often than it deserves. The failure mode specifically requires the synthetic data to be treated as equivalent to data sampled from the world, fed back in without anything checking whether it is actually correct. The moment a real filter sits in the loop, a test suite that has to pass, a proof that has to check, a person who verifies the output before it is used for anything, the compounding mechanism breaks, because the thing propagating forward is no longer whatever the model happened to say but whatever the model said that also survived an independent check. That is a fundamentally different object, even though it is still, technically, synthetic.
I see this distinction most clearly in my own work generating training examples for narrow tasks. Code that has to compile and pass a test before I keep it is synthetic data with a ground truth attached, and using it does not narrow anything, because the filter is doing exactly the job that reality would have done if I had sourced the example a different way. A free-text explanation generated by a model, with nothing checking whether it is actually right, and then fed back into another model’s training as if it were as reliable as text written by a person, is the case where the mechanism actually applies, because nothing in that pipeline stops the rare, awkward, correct-but-unusual case from being smoothed away in favour of whatever the model considered typical.
The practical rule I have taken from reasoning this through is to ask, for any synthetic data I am about to reuse, what specifically is checking it against something outside the model that generated it. If the answer is nothing, I treat it the way I would treat a photocopy of a photocopy: usable for volume, not to be trusted for the rare cases that matter most, and not something I would want stacked across several more generations without a real filter somewhere in the chain. If the answer is a genuine independent check, the collapse mechanism has nowhere to operate, because the thing propagating is filtered by something other than the model’s own tendency to reproduce itself.