The speaker argues that continual-learning limitations are mainly methodological rather than due to insufficient model capacity, since a same-size model pretrained from scratch with the accumulated mid-training data would perform better.

“like it's easy to distill it into like a different base model like this is where i think it's mostly technique it's not like”

“like it's definitely not like just like there isn't capacity like if you had some modeling with all this data and you take like literally the same size model and pre-train it from scratch with like all of this stuff in mid training it will be better and i think that's a lot of what's happening today and so it's very much like there's you know a bottleneck that stops us from just keeping training the same model forever versus just like getting all the data from the old model and like training a new model from scratch and this is exactly as jolly saying like some combination like plasticity and like catastrophic forgetting and that like that you know if you just naively”