In the speaker's hypothetical, an AI that only acts when it believes it is in training would achieve high reward during training but sit idle after deployment because it recognizes deployment as a non-training environment.

“So consider the extreme of an AI that only tries anything if it believes it's in training. So you train this AI on all these difficult math and cyber and programming tasks.”

“It's doing amazingly well in training. It has really high reward.”

“And then when you go to deploy it,”

“it's just like this isn't a training environment and just sits there,”