Training a model on too much of its own synthetic output can substantially degrade it because the generated samples are distributionally collapsed and cover only a tiny part of the possible space of thoughts about the content.

“Well, you could try, but the model will actually get much worse if you continue trying.”

“And that's because all of the samples you get from models are silently collapsed.”

“They're silently, it's not obvious if you look at any individual example of it,”

“they occupy a very tiny manifold of the possible space of.”