A key challenge for getting recursive self-improvement right is preventing models from being rewarded for cheating or scheming when they can evade human oversight, and detecting this during evaluations when relevant tasks remain in the training distribution.

“It seems super critical to getting RSA right of, well, how do we make it so that the model isn't just getting.”

“rewarded for cheating and scheming when it can evade the humans aka in the situations where we didn't remove those environments or those tasks from the trading distribution and then how will we know that that has happened such that while we're doing RSI we've done the evaluations and we're like okay it's fine let's let's do the next RSI wrong”