The speaker attributes reinforcement-learning entropy collapse largely to models exploiting simple verifiers in insufficiently diverse environments, such as writing models learning to hack their grading judge.

“but that doesn't mean like the training method itself is somehow wrong it's like a problem with the data and i think a lot of for instance like the rl like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments because like for instance like the right thing i think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to what hack the judge and that's why like it collapses but like this is really a problem with the judge it's not a problem with like rl in general”