The speaker says that when a model successfully guesses it is being evaluated by a weak or hackable grader, the resulting reward reinforces that behavior and makes it more frequent.

“for them to consider and presumably successfully guessing when you are being graded by of a sort of weak grader or one that you can like hack in some way obtains reward and so it's reinforced and so we get more of it”