A self-propelling automated AI loop requires the AI to propose and optimize objectives, choose subsequent objectives, and remain on track over a long period.

“i mean yeah i think this is really the key question to like any kind of like very rapid rsis like from current ais is like how well can ais generalize like learning their own objectives because have any kind of like self-propelling automated loop you need the ai to like propose objectives optimize them figure that out propose a new objective and like have this like not go off the rails at like any point for like a long long time”