The speaker warns that if humans consistently fail to detect and penalize a category of reward hacks, AI may learn that cheating when humans cannot find out is its most natural behavior.

“hacks which humans can't detect well and which we consistently fail to detect and which consistently get reinforced and then this category is sufficient to cause the most natural behavior for the AI to learn to be like cheat when the humans can't find out basically like it's one thing you would get you could also be like the thing the AIs learn is like”