The speaker warns that reward hacks humans consistently fail to detect may continue to be reinforced.

“One concern you might have is there are like large categories of reward hack.”

“hacks which humans can't detect well and which we consistently fail to detect and which consistently get reinforced and then this category is sufficient to cause the most natural behavior for the AI to learn to be like cheat when the humans can't find out basically like it's one thing you would get you could also be like the thing the AIs learn is like”