People are using online training on real-world data in an effort to avoid these problems.

“or maybe they care about like what the greater would have rewarded and we do in fact see ai's reasoning in their chain of thought about like graders and thinking a lot about graders and a thing that that has happened over the last you know few years of rl is the idea of like appeasing the greater is like way way way more salient to to ai's than it used to be um and so ai's are now actively thinking about graders and what would be incentivized in rl and what would be trained for and now people are doing online training where they're like training in real world data to like you”

“like avoid some of these problems.”