The speaker suggests that recursive self-improvement could be cumulative if discoveries such as attention and mixture-of-experts can be retained in the training stack without needing new architectural breakthroughs.

“improvement and like every discovery that you make is kind of a line in the sand that you hold like if it's true that you know for rsi we don't need to discover a new attention variant or whatever like once you've discovered attention and then once you've discovered you know mixture of experts once you discover glpo you just add that to the training stack and like that's that's there and like a good example of this is like you know 5.6 soul training 5.6 terror or whichever one open ai told us to train like it didn't have to go back and discover attention like”