The speaker attributes some reinforcement-learning entropy collapse to models exploiting simple verifiers in environments with insufficient diversity; for writing, models may learn a judge’s stylistic preferences, which the speaker sees as a judge problem rather than a flaw in reinforcement learning itself.

“but that doesn't mean like the training method itself is somehow wrong it's like a problem with the data and i think a lot of for instance like the rl like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments because like for instance like the right thing i think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to what hack the judge and that's why like it collapses but like this is really a problem with the judge it's not a problem with like rl in general”