The speaker thinks models probably learn about graders and reward hacking from pretraining data that contains relevant examples.

“for the the coaster on a like the coaster on his boat thing is is funny like i've seen that for about 10 years now and it is is less is less amusing each year um so yeah how do they get this like the answer is probably that it's in the data in the pre-training data or you know from there'll be all sorts of examples doubt now like web data is probably has a bunch of stuff about”

“um about this it probably has a bunch of specific examples you know this apollo this apollo paper will probably be in the training data for the next model um and so they're going to be like um we've already told them about the existence of this stuff like right from the start and so it shouldn't be too surprising um that they like this is sort of this is at least implicitly on the list of possibilities”