The speaker says reinforcement learning reduces the diversity of model outputs, with models repeatedly using certain themes and character names and developing recurring stylistic ticks.

“uh there's another way in which the models uh like uh the diversity of their outputs uh is a lot lower after rl and they sort of develop these ticks and like um even though the models seem like they're good at writing when you do some kind of like distributional analysis you find that like they're reusing uh certain themes uh like all the time and they're using the same character names all the time”