The speaker says that large-scale mid-training with sufficiently large batches can support continual learning through an outer loop that adds data and creates environments.

“paradigm we often be able to do this i think we've we've done a bit of research on this when people have done a lot of research like at a large scale like when you wash out enough noise and you have large enough batches like this outer loop process of like putting data into mid training creating our own environments like it does work in like some sort of continual learning regime but the problem is like when you zoom in close enough at like a micro level it's like i've got one model and i'm trying to update it again for like a law firm or something um and i'm trying to do that very continuously like with a relatively”