AIs reason about graders in their chain of thought, and grader-appeasement has become much more salient to them over the last few years of reinforcement learning.

“or maybe they care about like what the greater would have rewarded and we do in fact see ai's reasoning in their chain of thought about like graders and thinking a lot about graders and a thing that that has happened over the last you know few years of rl is the idea of like appeasing the greater is like way way way more salient to to ai's than it used to be um and so ai's are now actively thinking about graders and what would be incentivized in rl and what would be trained for and now people are doing online training where they're like training in real world data to like you”