The speaker proposes that models may excel at cumulative tasks but continue to struggle with tasks involving non-stationary distributions that require ongoing learning and repeated reconsideration.

“I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is like cumulative or like you kind of have to you have this like non-stationary distribution you have to keep learning and like re-litigating a bunch of stuff so like maybe an example of a cumulative task might be RSI like it's theoretically possible to maybe like have less than a million token like you know Python file which like from scratch trains a model that is capable of recursive self-improvement.”