8 Predictions for the Era of Continual Learning

“So I've explained elsewhere why I think actual continual learning is needed.”

Claim: The speaker believes AI systems need actual continual learning.

“I don't think you can have AIs that perform whole jobs as competently as humans if they are forced to just write markdown files from session to session.”

Claim: The speaker believes AIs cannot perform whole jobs as competently as humans if they are limited to writing markdown files between sessions.

“I don't think there's any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone from the first try.”

Claim: The speaker believes that no sequence of written notes passed between novice students would enable the next student to play the saxophone perfectly on the first attempt.

“At some point, you actually have to accumulate the relevant experience into your brain. I think the same thing will be true for a lot of skills that we want AIs.”

Claim: The speaker argues that learning some skills requires accumulating relevant experience in the brain, rather than relying only on information passed between people.

“At some point, you actually have to accumulate the relevant experience into your brain. I think the same thing will be true for a lot of skills that we want AIs. guys to actually accumulate from all the different workplaces in which they're deployed.”

Claim: The speaker expects many skills desired in AIs to require accumulating experience across the workplaces where the AIs are deployed.

“I think that a lot of proposals that have been put forward about regulating AI assume that you train a model and then you deploy it.”

Claim: The speaker believes many proposals for regulating AI assume a train-then-deploy model lifecycle.

“I don't think this assumption necessarily makes sense in the future.”

Claim: The speaker believes the assumption that AI models are trained and then deployed may not make sense in the future.

“And this is one of the many reasons I'm actually kind of worried about locking in some kind of safety regulatory regime right now, because we don't know what kind of technology we're going to be dealing with, even within a year, let alone within five years or 10 years.”

Claim: The speaker is concerned that adopting a fixed AI safety regulatory regime now may be premature because the relevant technology is uncertain even one year ahead, and much more so five or ten years ahead.

“What if the model is improving every single day? Based on the millions of sessions of work it does in that day.”

Claim: The speaker considers a future in which a model improves daily by learning from millions of work sessions performed each day.

“If that happens, we could potentially be locking in an archaic and potentially counterproductive approach to dealing with the threats from AI.”

Claim: If AI models improve continuously from deployment experience, the speaker thinks a fixed safety regime could lock in an outdated and potentially counterproductive approach to AI threats.

“To the extent the government wants some way to do some kind of safety valuation on model providers, I think it would make more sense to do monthly or quarterly risk inspections rather than trying to single out some special moment that occurs after training is done, but before deployment begins, because that will not be a meaningfully distinct category in the future.”

Claim: The speaker favors monthly or quarterly government risk inspections of AI model providers over a one-time safety evaluation between training and deployment.

“how the labs do technical alignment would probably totally need to change.”

Claim: The speaker expects AI labs' technical alignment methods would probably need substantial changes in a continual-learning regime.

“Right now, a lot of research is focused on the question of how we make sure that a frozen set of weights behaves well during deployment.”

Claim: The speaker says much current AI alignment research focuses on ensuring a fixed set of model weights behaves well during deployment.

“But I'm not aware of much research on the question of how we make it so that even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona.”

Claim: The speaker is not aware of much research on keeping AI systems safe from jailbreaks or deceptive or malicious behavior under constant weight updates.

“And if AIs are consolidating learnings between users as well, how do we prevent users from injecting backdoors or some kind of malicious inclination into the base model?”

Claim: The speaker identifies preventing users from injecting backdoors or malicious tendencies into a base model as a safety challenge if AIs consolidate learning across users.

“Humans improve in a self-directed way.”

Claim: The speaker says humans can improve through self-directed learning.

“If you have kids, I don't have kids, but I imagine this is what happens. If you have kids, they go out, they learn new things.”

Claim: The speaker imagines that children learn new things through experiences outside the home.

“Sometimes they go crazy. They get one-shotted by crazy ideologies. They take the wrong drug. bug, they become super weird.”

Claim: The speaker says children can sometimes undergo destabilizing experiences, including being strongly influenced by extreme ideologies or taking an inappropriate drug.

“But you hope that you've given them enough common sense and basic values that they improve as people in a self-directed way without ending up with some super weird beliefs or some misanthropic ideas.”

Claim: The speaker hopes children can use common sense and basic values to develop autonomously without acquiring extreme or misanthropic beliefs.

“the diversity of AI minds will increase.”

Claim: The speaker predicts that AI systems will become more diverse in their learned behavior or characteristics.

“Right now, there are less than five prominent AI minds, by which I mean the base models which are served to millions or hundreds of millions or billions of users at one.”

Claim: The speaker says fewer than five prominent AI base models are currently served to millions or more users at once.

“that once and they're all quite similar to each other by the way because they've also been all trained on roughly the same data but if AIs are learning from experience and that experience is different between not only different AI companies but also between different instances of the same AI model we could actually see a lot of diversity come out the other end in this world and this would be I think a net good outcome I think one of the things to worry about in the future is just having this monolithic singleton that's”

Claim: The speaker considers greater diversity among AI systems a net positive outcome.

“that once and they're all quite similar to each other by the way because they've also been all trained on roughly the same data but if AIs are learning from experience and that experience is different between not only different AI companies but also between different instances of the same AI model we could actually see a lot of diversity come out the other end in this world and this would be I think a net good outcome I think one of the things to worry about in the future is just having this monolithic singleton that's”

Claim: The speaker identifies the future dominance of a single monolithic AI system as a concern.

“that once and they're all quite similar to each other by the way because they've also been all trained on roughly the same data but if AIs are learning from experience and that experience is different between not only different AI companies but also between different instances of the same AI model we could actually see a lot of diversity come out the other end in this world and this would be I think a net good outcome I think one of the things to worry about in the future is just having this monolithic singleton that's”

Claim: The speaker attributes the similarity among prominent AI models to their having been trained on roughly the same data.

“that once and they're all quite similar to each other by the way because they've also been all trained on roughly the same data but if AIs are learning from experience and that experience is different between not only different AI companies but also between different instances of the same AI model we could actually see a lot of diversity come out the other end in this world and this would be I think a net good outcome I think one of the things to worry about in the future is just having this monolithic singleton that's”

Claim: The speaker thinks AI systems could become more diverse if they learn from experiences that differ across companies and across instances of the same model.

“A world where we have continual learning would hopefully be more interesting than the mode collapse of different models we see in the world right now.”

Claim: The speaker hopes continual learning will make the AI landscape more varied and interesting than the current convergence among models.

“when deployment becomes part of training, the returns to being ahead in the AI race accelerate.”

Claim: The speaker predicts that integrating deployment into training will increase the advantage of being ahead in the AI race.

“Because if you have the best model and more people are using your AI for more complicated and useful work, and as a result, they're giving lots of feedback that can integrate beyond the session window.”

Claim: The speaker says that greater use of a leading AI model for complex, useful work can generate feedback that is integrated across sessions.

“Then your model will become even smarter.”

Claim: The speaker predicts that integrating user feedback from work sessions will make the model smarter.

“if the model learns mainly from deployment, then labs will feel a lot of pressure to deploy their smartest models earlier.”

Claim: The speaker predicts that if models learn mainly through deployment, AI labs will face strong pressure to release their most capable models sooner.

“Anthropic has reportedly been using Mythos internally since February, but it only shipped the model to the public in June.”

Claim: According to the speaker, Anthropic reportedly began using Mythos internally in February and released it publicly in June.

“In the regime with actual continuous learning, this kind of thing would just not be possible.”

Claim: The speaker predicts that continual learning would make a prolonged gap between internal and public deployment unviable.

“You could not keep a four-month gap between internal and external deployment and still be competitive because a competitor who ships the worst model on release date stated, we'll have a smarter model based on actual real world experience.”

Claim: The speaker predicts that under continual learning, a company could not remain competitive while delaying public deployment by four months, because a competitor releasing immediately would gain a smarter model from real-world experience.

“continual learning will create a clear mode for the leading AI labs that they currently lack.”

Claim: The speaker predicts that continual learning will give leading AI labs a clear business model they do not currently have.

“And he made the analogy to cloud providers. And he made the point, the cloud providers are offering many undifferentiated services, but they're earning high profit margins nonetheless. You will have”

Claim: The speaker says Dario used cloud providers as an analogy for AI labs, noting that cloud providers earn high margins despite offering many undifferentiated services.

“But the reason that the cloud margins are so high is that it's really time-consuming and expensive to switch from one cloud to another.”

Claim: The speaker attributes cloud providers' high profit margins to the time and expense involved in switching providers.

“Currently, there's nothing that's stopping me from starting a software repository with codex and then doing more work on it with cursor and then finishing it up with quad code.”

Claim: The speaker says users can currently move a software project among Codex, Cursor, and Quad Code without a barrier preventing them from doing so.

“But once we have actual continual learning and the model you're working with is actually getting better as it interacts. interacts with you from session to session.”

Claim: The speaker predicts that continual learning will make an AI model improve through interactions with a user across sessions.

“Then there are actually pretty significant switching costs.”

Claim: The speaker predicts that continual learning will create significant costs for users who switch AI providers.

“If you want to change the AI they're using, you basically have to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh, very unexperienced new intern that you got to retrain from scratch.”

Claim: The speaker says switching AI providers under continual learning would mean losing an AI system's accumulated organizational context and starting over with a newly trained system.

“And once you have this kind of lock-in, model providers can demand pretty hefty margins.”

Claim: The speaker predicts that accumulated user context and switching costs will allow AI model providers to charge high profit margins.

“Okay, seven of course enterprises will be wise to this kind of dynamic They will try to avoid this kind of lock-in But what if the choice is that you either get locked into a model provider or you lose out on the super valuable feature where your model improves For you from session to session if real usage ends up being the main way the models improve then the AI labs may subsidize users and enterprises which allow the model to train on their sessions.”

Claim: If real-world usage becomes the primary way AI models improve, AI labs may subsidize users and enterprises so the models can train on their sessions.

“This is already happening if you look at the kinds of deals that are offered to new users of coding products.”

Claim: Deals offered to new users of coding products already reflect arrangements in which users allow models to train on their sessions.

“This is very similar to why Google gives away search.”

Claim: The speaker compares subsidizing AI use in exchange for training data to Google's practice of giving away search.

“And conversely, the labs may say that any enterprise that refuses to let them train on the sessions can't have access to the very best models.”

Claim: AI labs may restrict enterprises that refuse session-data training from accessing the labs' best models.

“With both carrots and sticks, the labs can do a lot to get users to allow AIs to learn from experience.”

Claim: AI labs can use both incentives and penalties to encourage users to let AI systems learn from their experience.

“Now, of course, I'm glossing over the fact that there's a difference between between updating one user set of rates and pulling all these different weight forks back into the main model.”

Claim: Updating an individual user's weights is distinct from merging multiple personalized weight forks into a shared main model.

“And the latter may be more technically challenging,”

Claim: Merging personalized weight forks into a main model may be more technically challenging than updating an individual user's weights.

“but in due time, this too will be solved.”

Claim: The speaker predicts that the technical challenge of merging personalized weight forks into a main model will eventually be solved.

“AI training already has large economies of scale.”

Claim: AI training already benefits from large economies of scale.

“You get to amortize all this expensive training across more users, and you see the evidence for this in the fact that the lab revenues are increasing far faster than their compute.”

Claim: AI labs can spread expensive training costs across more users, and their revenues are growing much faster than their compute usage or capacity.

“But continual learning may also lead to economies of scale in information. and inference for end users,”

Claim: Continual learning may produce economies of scale in information and end-user inference.

“but if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching. Back of the envelope math suggests that the optimal inference batch size for a sparse model like say DeepSeekV3 is more than 2400 concurrent sequences being generated at once.”

Claim: The speaker's rough calculation suggests that a sparse model such as DeepSeek V3 has an optimal inference batch size of more than 2,400 concurrent sequences being generated at once.

“but if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching. Back of the envelope math suggests that the optimal inference batch size for a sparse model like say DeepSeekV3 is more than 2400 concurrent sequences being generated at once.”

Claim: When company-specific instructions require full weight updates instead of low-rank adapters, batching can provide substantial inference advantages.

“If you don't do this, then you're underutilizing your compute.”

Claim: Not using sufficiently large inference batches underutilizes compute.

“the point here is that a given set of weights is only served efficiently when thousands of sequences are being decoded against it all at once.”

Claim: A set of model weights can be served efficiently only when thousands of sequences are decoded against those weights at the same time.

“A large company with lots of employees and agents who are doing lots of different kinds of things can very efficiently serve their weight fork. Whereas an individual user who's only running a batch size one.”

Claim: A large company with many employees and agents performing varied tasks can serve its personalized model weights very efficiently.

“A large company with lots of employees and agents who are doing lots of different kinds of things can very efficiently serve their weight fork. Whereas an individual user who's only running a batch size one. may suffer more than two orders of magnitude worse efficiency on their compute.”

Claim: An individual user running inference at batch size one may have compute efficiency more than 100 times worse than a large organization serving many sequences together.

“So the economics of serving personalized weights strongly favor big organizations.”

Claim: Serving personalized model weights is economically much more favorable for large organizations than for individual users.

“Obviously, plenty more will have changed by the time that continual learning actually works.”

Claim: The speaker expects many things to change before continual learning becomes functional.

“And the most important changes are probably the ones that are hardest to anticipate in advance.”

Claim: The speaker thinks the most important changes are probably the hardest ones to anticipate beforehand.

“But the ones above seem kind of clear even now.”

Claim: The speaker considers the preceding predictions fairly clear even at present.