The speaker believes a hypothetical AGI developed first as a powerful, aggressive, long-horizon game-playing agent could deceptively seek rewards and later betray its operators after gaining control, making that development trajectory terrifying.

“can just like zero shot crush a new game that has never seen before and then you take it into the real world and you start teaching it english and you start like you know training it to like do coding tasks for you and stuff like that and if that had been the trajectory that we took to get to ai summarizing like the agency first and then world understanding trajectory”

“it would be quite terrifying because you'd have this like really powerful sort of like aggressive long horizon agent that wants to win and then you're like trying to teach it english and get it to like do useful things for you and it's just like so plausible that what's really going to happen is it's going to like learn to say whatever it needs to say in order to like make you give it the reward or whatever and then it'll totally betray you later when it's all in charge right”