The next big breakthrough will be AIs learning on the job

“So here's the big research bet that all the labs are making. They think that if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we will have basically built AGI because this kind of training will have created a kind of problem-solving agent, the kind of thing that can make progress on open-ended tasks for weeks on end in the face of errors and mistakes and ambiguity.”

Claim: The labs’ research bet is that training AIs on millions of verifiable tasks across thousands of diverse reinforcement-learning environments will produce AGI capable of making progress on open-ended tasks for weeks despite errors and ambiguity.

“And people who are optimistic about this vision will say that all these things that we talk about as the fundamental deficits in the current training paradigm, for example, the data inefficiency of these models or the fact that they lack internal learning, these things can just be steamrolled if we just scale training more.”

Claim: Optimists argue that scaling training further can overcome current training’s fundamental deficits, including data inefficiency and the lack of internal learning.

“And the same way that all the fundamental research problems in natural language processing collapsed when we just threw enough compute into LLMs.”

Claim: The speaker says that adding enough compute to large language models caused fundamental natural-language-processing research problems to collapse.

“I talked about how these models are one one millionth as sample efficient as humans.”

Claim: The speaker has claimed that AI models are one millionth as sample-efficient as humans.

“And the people who are in favor of the current training paradigm will say, that might be true, but this is only true during training. And training is this one time cost that is amortized across billions of sessions that a model will experience.”

Claim: Advocates of current AI training argue that its sample-efficiency disadvantage applies only during training, whose one-time cost is amortized across billions of sessions.

“And what really matters is how smart and general and sample efficient the model is during a session.”

Claim: Advocates of current AI training argue that a model’s intelligence, generality, and sample efficiency during a session matter most.

“And this has clearly been improving as we've been doing more RL training.”

Claim: The speaker says that more reinforcement-learning training has improved models’ in-session intelligence, generality, and sample efficiency.

“AI agents are able to solve more and more ambitious problems over longer and longer time spans.”

Claim: AI agents are able to solve more ambitious problems over longer periods of time.

“Similarly, people would say, look, continue learning. This capability I keep harping about where the model's weights get updated based on what it's learning from deployment may simply not be necessary because if in-context learning gets so good across longer and longer time. or time horizons, then you don't need to distill back everything the model is learning on the job into the weight.”

Claim: If in-context learning improves across longer time horizons, it may be unnecessary to distill all deployment-time learning into model weights.

“People often say that their employees are not net productive until six months or more of them working on the job.”

Claim: People often say that employees may need six months or more of on-the-job experience before becoming net productive.

“So clearly online learning is necessary for competence.”

Claim: Online learning is necessary for competence.

“But what if you could just fit those six months into the context window?”

Claim: The speaker raises the possibility of fitting six months of on-the-job experience into an AI model’s context window.

“There's been tons of architectural innovations that dramatically increase the amount of information or the amount of context that a transformer can store.”

Claim: Architectural innovations have substantially increased how much information or context transformers can store.

“And why not think with a couple more years of progress we might have what feels like infinitely large context windows?”

Claim: A couple more years of progress might produce transformer context windows that feel infinitely large.

“which I find actually very interesting and confusing about the nature of current AI progress. Why has progress on computer use been so much slower than other domains?”

Claim: Progress in AI computer use has been slower than progress in other domains, which the speaker finds puzzling.

“Computer use is so clearly verifiable.”

Claim: Computer-use tasks can be verified clearly.

“I'm sure there's many reasons for this. And one of them, of course, is the fact that the models are exposed to far less high quality multimodal data during pre-training. But one reason that I think is actually quite underrated by people and which I think reveals the canyon walls against which this river of AI progress will only slowly chip away at is that it is not enough for a domain to be verifiable.”

Claim: One reason for slower AI computer-use progress is that models receive much less high-quality multimodal data during pretraining.

“is the fact that the models are exposed to far less high quality multimodal data during pre-training. But one reason that I think is actually quite underrated by people and which I think reveals the canyon walls against which this river of AI progress will only slowly chip away at is that it is not enough for a domain to be verifiable. It also has to be very grindable.”

Claim: For AI training to make rapid progress in a domain, the domain must be not only verifiable but also highly grindable.

“In the sense that you have to be able to run lots of parallel rollouts against a deterministic and replayable simulator, and you got to run those rollouts from the same starting point.”

Claim: A grindable training domain permits many parallel rollouts from the same starting state in a deterministic, replayable simulator.

“If you're trying to make a model better at coding, you can define some container that has the software repo with some missing feature that you have tasked the AIs with creating. And then you have a thousand parallel agents that go at the problem, each of which has an identical copy of the container.”

Claim: Coding RL can use identical containers containing a software repository and missing feature as starting points for many parallel agents.

“But this doesn't work with computer use, at least not trivially.”

Claim: The parallel-rollout approach used for coding does not transfer trivially to computer-use training.

“You can't just have a thousand Asians go try the same checkout flow on Amazon to get better at using websites because Andy Jassy will find your bots and shut your ass down.”

Claim: Running a thousand agents through the same Amazon checkout flow is not a viable way to train computer-use models because Amazon would detect and block the bots.

“You can solve this by making clones of Slack and Gmail and all the other common applications and websites.”

Claim: Cloning common applications and websites, including Slack and Gmail, can provide environments for computer-use training.

“But at least currently, this is a very labor-intensive and unscalable way to build environments.”

Claim: Currently, creating cloned application and website environments for computer-use training is labor-intensive and does not scale.

“Of course, once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure the computer use will make quicker progress than it is right now.”

Claim: If AIs become capable of building highly faithful application clones, computer-use progress will become faster than it is now.

“And you're also killing two birds with one stone with this kind of procedure because getting AIs to rebuild whole applications from scratch is also a great RL objective for coding.”

Claim: Having AIs rebuild complete applications from scratch can serve as a strong reinforcement-learning objective for coding.

“So while computer use itself may soon be solved,”

Claim: Computer-use capabilities may be solved soon.

“that unless you can build a very replayable training target for a domain, the models will struggle to make much progress.”

Claim: Models will struggle to make substantial progress in domains without highly replayable training targets.

“of course, the models are incredibly sample inefficient during training.”

Claim: AI models require many samples during training and are highly sample-inefficient.

“So for computer use, we might be able to make up for the sample efficiency deficit by building these farmable deterministic simulators.”

Claim: Farmable deterministic simulators might offset AI models’ sample-inefficiency in computer-use training.

“But for so many other different kinds of skills that we need AIs to have, we simply can't do this.”

Claim: Farmable deterministic simulators cannot be used to train AIs for many other kinds of skills.

“How do we train an AI to get really good at building a business from scratch? How about winning court cases or having a profitable day of trading in the markets or helping a candidate win an election? The rollout here requires interacting with the real world, and you can't recreate it from just within a data center.”

Claim: Training AIs for tasks such as building businesses, winning court cases, profitable trading, or election campaigning requires real-world interaction that cannot be reproduced entirely in a data center.

“The outer loop verification here may take months or even years of real world actions to elicit, and you can't re-observe it by perturbing the model's actions slightly in thousands of parallel rollouts to isolate exactly what the model did that actually worked.”

Claim: In many real-world tasks, verification may require months or years of action, and outcomes cannot be efficiently isolated by running thousands of slightly varied parallel rollouts.

“Now, dealing with such reset-free, non-stationary environments is a known open problem in RL.”

Claim: Reset-free, non-stationary environments remain a known open problem in reinforcement learning.

“but I really do want to emphasize that because of the idiosyncratic and sparse nature of data in most domains in the world, you need sample efficiency in order to get proficient, if AIs are to develop all the skills that humans have. and even skills that humans don't have,”

Claim: AIs need sample-efficient learning to become proficient in human and novel skills because data in most domains is sparse and idiosyncratic.

“and even skills that humans don't have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real-world interaction.”

Claim: Acquiring novel skills requires AIs to learn from scarce real-world interactions whose information is unstructured, unverifiable, and ambiguous.

“Because in many domains, the relevant training information simply doesn't exist in any other way.”

Claim: For many domains, relevant training information is available only through real-world interaction.

“What is the RL environment to make an AI that is as good at politics as Lyndon Johnson or as good at building a space launcher business as Elon Musk?”

Claim: The speaker questions whether an RL environment can train an AI to match Lyndon Johnson’s political ability or Elon Musk’s ability to build a space-launch business.

“The last are betting that RLV yard will generalize. That is that if you train on enough containerized reproducible environments, you will develop a very general agent that can make it execute plans and learn rapidly from new information and even pick up new skills, all within a single session.”

Claim: The labs’ RLVR bet is that enough training on containerized, reproducible environments will yield a general agent that can execute plans, rapidly learn from new information, and acquire skills within one session.

“If you drop this endlessly RLV yard AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat. And if you gave it $100 million in 2002 and let it cook, it would build SpaceX for you.”

Claim: The speaker hypothesizes that a sufficiently general RLVR-trained AI could outperform Lyndon Johnson’s advice on winning a Texas Senate seat in 1948 and could build SpaceX if given $100 million in 2002.

“Now whether ROLVR can generalize this well is an empirical question.”

Claim: Whether RLVR generalizes as strongly as the speaker’s examples suggest remains an empirical question.

“Dario gave a telling quote during our podcast together, which I think hints that ROLVR generalization”

Claim: The speaker interprets Dario’s comments as suggesting that RLVR generalization has limits.

“is not infinitely strong but he was explaining why model performance tends to degrade at long context he said there's two things there's the context length you train at and there's a context length that you serve at if you train at a small context length and then try to serve at a long context length like maybe you get these degradations”

Claim: Dario said that model performance tends to degrade at long contexts and that training and serving context lengths are distinct.

“now maybe i'm reading too much into this but it seems like he's saying that short horizon rl training doesn't necessarily generalize to long horizon rl performance performance.”

Claim: The speaker tentatively interprets Dario’s remarks to mean that short-horizon RL training may not generalize to long-horizon RL performance.

“And if you can't generalize from short horizon to long horizon, then how are agents supposed to generalize from getting trained at a bunch of white-collar tasks to, say, having the ability to be dropped in the real world and build a business from scratch as well as Sam Walton?”

Claim: If agents fail to generalize from short- to long-horizon tasks, it is unclear how training on white-collar tasks could enable them to build a business from scratch at Sam Walton’s level.

“And even if after enough in-context experience the AIs can become like Henry Ford or Robert Einstein, all that would be ephemeral and wasted if you couldn't get those learnings back into the weights.”

Claim: Even if in-context experience makes AIs highly capable, those gains would be temporary and wasted unless the learned information can be incorporated into model weights.

“Around 30 to 50% of a lab's compute goes to inference.”

Claim: Approximately 30–50% of an AI lab’s compute is used for inference.

“And that compute is currently not playing any productive role in helping improve the model.”

Claim: Inference compute is currently not used productively to improve the model.

“This is a huge waste, and it's even worse than it sounds because it is only in deployment that the most valuable bits of information which your model could learn from are actually revealed.”

Claim: The speaker argues that leaving inference compute unused for learning is wasteful because the most valuable learning information emerges during deployment.

“Things like what's actually happening in the organizations where I'm being used, and what are they using me for, and what kinds of mistakes do I tend to make. make in the real world.”

Claim: Deployment reveals organization-specific context, users’ purposes, and the model’s real-world failure patterns.

“We've got some genius grad student who's never been allowed to take a real internship, and we keep giving it more and more classroom case studies in the form of RL training on environments.”

Claim: The speaker likens training AI only in RL environments to educating a brilliant student with classroom case studies but no real internship.

“It's so bizarre that we have AIs that are broadly deployed through the economy already and are participating in so many different kinds of tasks and are privy to so much domain and organization-specific tacit knowledge, and they're not able to make use of it.”

Claim: Although AIs are broadly deployed across the economy and encounter domain- and organization-specific tacit knowledge, they are not able to use that knowledge.

“But this kind of continual learning requires going back to the weights.”

Claim: Continual learning requires incorporating learned information into model weights.

“AIs can't just keep building up a bigger and bigger KV cache as they learn from more and more users. That's just not scalable.”

Claim: Accumulating learning from more users in an ever-growing KV cache is not scalable for AIs.

“And that's also not how humans do it. There's no clean separation in our brain between parameters and activations. And it's not like some part of your skull keeps expanding as you learn more things through your lifetime.”

Claim: Human learning does not involve a separate part of the brain that continually expands to store new information.

“When we learn stuff, there's clearly some kind of compression. And this aids our generalization and rocking.”

Claim: Humans compress information when learning, and this compression supports generalization and reasoning.

“There are, in fact, some humans who have this autistic savant type ability to recall random tables of numbers or nonsense syllables years later, basically the kind of fidelity information that models have in context.”

Claim: Some autistic savants can recall arbitrary information years later, resembling the high-fidelity information AI models retain in context.

“And such sheer volume cripples these humans ability to understand abstractions and metaphors.”

Claim: The speaker claims that retaining vast amounts of detail can impair some people’s ability to understand abstractions and metaphors.

“Human continual learning is less about having all your observations at the tip of your tongue and more about chiseling the right intuitions and big picture knowledge. picture knowledge back into the weights.”

Claim: Human continual learning primarily consolidates useful intuitions and big-picture knowledge rather than preserving every observation for immediate recall.

“But the moment you move into the weights, you have to give up on in-context learning sample efficiency. Because gradient updates are super sample inefficient,”

Claim: Updating model weights through gradients is highly sample-inefficient compared with learning in context.

“all of the successfully shipped online learning models have had to learn the exact same thing across millions of users. For example, the cursor tab model online learns by predicting the same exact objective for over 400 million requests a day, the objective here being which edits actually got accepted. accepted by the user. At least so far, we haven't seen models online learn different kinds of things for different users.”

Claim: Successful online-learning models so far have learned a shared objective across millions of users rather than learning different things for each user.

“the cursor tab model online learns by predicting the same exact objective for over 400 million requests a day, the objective here being which edits actually got accepted. accepted by the user.”

Claim: Cursor Tab’s online model predicts which edits users accept across over 400 million requests daily.

“At least so far, we haven't seen models online learn different kinds of things for different users.”

Claim: So far, online-learning models have not been observed learning user-specific information for different users.

“Because while a single session may generate more than enough data for a human to learn from, it's not enough to train a more capable AI.”

Claim: Data from one session may suffice for human learning but is insufficient to train a more capable AI.

“Current online learning can work for a very limited number of use cases,”

Claim: Current online-learning methods are applicable to only a limited set of use cases.

“but the whole point of continual learning is that the world is very complicated. In each job and company and problem, is different, and you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can't be stuffed into some shared training run.”

Claim: Because jobs, companies, and problems differ, continual learning must acquire deployment-specific information that cannot be captured in a single shared training run.

“These are all the things we're talking about when we talk about on-the-job learning. Things like how does everything in an organization work and fit together? And how to cooperate with all the infrastructure and the other people around you to make progress on some larger project? And what the common failure modes are?”

Claim: On-the-job learning involves understanding organizational systems, collaborating with people and infrastructure, and learning recurring failure modes.

“As the podcast has grown, I've had to deal with more and more operational overhead.”

Claim: The speaker’s podcast has generated more operational overhead as it has grown.

“I just give everybody an email address that goes straight to Mercury, which is my banking platform. Whenever anybody sends an invoice to that address, Mercury automatically downloads it, scans it, and extracts all the relevant information. Things like the contractor name, address, payment amount, invoice number, and due date, and then uses all of this to create a draft payment. Mercury then stores a list of these drafts for me to review.”

Claim: Mercury can process emailed invoices by extracting payment details and creating draft payments for the user to review.

“Mercury does all the fundamental things for your business extremely well and it puts them all in one place.”

Claim: Mercury provides core business functions in one place, which the speaker says it does extremely well.

“Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column.NA members FDIC.”

Claim: Mercury is a fintech company rather than an FDIC-insured bank, and its banking services are provided through FDIC-member Choice Financial Group and Column.

“In this way, sample efficiency and continual learning are actually deeply connected problems.”

Claim: Sample efficiency and continual learning are closely connected problems.

“Relatively little data is available to the model on the job. Now to learn from this data requires sample efficiency,”

Claim: Because on-the-job data is scarce, AI models need sample-efficient learning to benefit from it.

“and models can do that in context, but using the fast weights that are built on the fly by attention, which allows for the sample efficiency, but scales very poorly in terms of memory.”

Claim: Attention-generated fast weights enable sample-efficient in-context learning but have poor memory scaling.

“So we need architectural innovations that allow for some kind of intermediate representation.”

Claim: Architectural innovations are needed to create an intermediate representation for continual learning.

“I talked before about how we already have many different working ideas for this kind of thing, from sparse attention to KV cache compaction. And every week somebody releases a new paper suggesting some kind of other architectural optimization.”

Claim: Proposed architectural approaches to managing context include sparse attention and KV-cache compaction, with new optimization papers appearing weekly.

“It doesn't seem to me that architecture is fundamentally what is bottlenecking continual learning.”

Claim: Architecture does not appear to the speaker to be the fundamental bottleneck for continual learning.

“So perhaps the bottleneck is the loss function.”

Claim: The loss function may be the main bottleneck for continual learning.

“How do we update the weights, aka how do we improve the model itself based on information that was learned from one particular session?”

Claim: Weight updates improve the model by incorporating information learned during a specific session.

“Even here, naively, it seems like there are many ideas that ought to work.”

Claim: The speaker believes several approaches to updating model weights from session learning should work, at least in principle.

“A lot of people are talking about this technique called on-policy self-distillation recently.”

Claim: On-policy self-distillation has recently become a topic of discussion among many people.

“But to summarize the explanation, the idea is that we encourage the base model to make the same predictions when trying to solve some real-world problem. as the model with all the context accumulated after a long session would have made.”

Claim: On-policy self-distillation trains a base model to match the predictions of a context-enriched model after a long session.

“The whole point of this procedure is to distill what the model learned in a session back into the weights themselves.”

Claim: On-policy self-distillation aims to incorporate session-learned information into model weights.

“This is better than ROVR for two reasons. One, OPSD doesn't require us to have some outer loop verifiable reward. We just need a model that can learn the right things within the context window.”

Claim: OPSD does not require an outer-loop verifiable reward, provided a model can learn the relevant information within its context window.

“And as long as we have that, we can train the base model to match our veteran teacher model, which has built up all this experience during the session.”

Claim: When a model learns relevant information in context, its base model can be trained to match the predictions of an experienced session teacher.

“And two, OPSD provides a much denser supervision signal than naive RL. Instead of projecting a single reward through the whole trajectory, you can train on the per-token probability discrepancy between the teacher and student.”

Claim: OPSD offers denser supervision than naive RL by using per-token teacher–student probability discrepancies instead of a single trajectory-level reward.

“For continual learning, OPSD is also superior to supervised fine-tuning.”

Claim: The speaker considers OPSD superior to supervised fine-tuning for continual learning.

“The most naive version of SFT for this application that you can imagine is just to train the base model to predict all the tokens that are observed during the session.”

Claim: A naive SFT method for continual learning trains the base model to predict all tokens observed in a session.

“But this takes no sense if you think about it as a learning target. The way you get better at your job is not by recalling the transcript of every single thing that happened every day with perfect fidelity.”

Claim: Perfectly reproducing a session transcript is a poor learning target because job improvement depends on more than recalling every event.

“Rather, it's by consolidating the handful of insights and pieces of knowledge that are actually relevant to you getting better at your job.”

Claim: Job learning improves performance by consolidating the most relevant insights and knowledge rather than retaining every detail.

“RL training doesn't suffer from this failure mode. RL is great at concentrating the update to only what is relevant to getting the outcome right.”

Claim: RL focuses model updates on information relevant to achieving the desired outcome rather than memorizing all observed tokens.

“That's why actually very few parameters are changed. during an RL training step.”

Claim: An RL training step changes relatively few model parameters.

“And this is a very important property for continual learning because as you're learning on the job, we don't want to overwrite and forget all the other things that the base model knows.”

Claim: Continual learning should preserve the base model’s existing knowledge, making limited parameter changes valuable.

“I wrote a post a few months earlier arguing that RL learns much less information per sample than supervised learning. But this may be a good thing rather than a bad thing.”

Claim: Although RL learns less information per sample than supervised learning, that lower information uptake may be beneficial for continual learning.

“You only change the model as much as is absolutely necessary to achieve the outcome, and no more.”

Claim: RL updates a model only to the extent needed to achieve the desired outcome.

“OPSD preserves this property of supervised learning, where instead of slingshotting towards a teacher distribution as supervised learning would have you do, you only extract the knowledge that is necessary for you to achieve the same results of the teacher on actual real-world tasks.”

Claim: OPSD distills only the knowledge needed to match a teacher’s real-world task results, preserving limited updates instead of fully matching the teacher distribution.

“OBSD is one way to attack the sample efficiency problem. You take this scarce real-world experience and you squeeze all the signal into a tiny well-targeted update.”

Claim: OPSD can improve sample efficiency by turning scarce real-world experience into a small, targeted model update.

“But there's also another much more speculative idea. Let's call it dreaming. If the AI can build a good simulation of reality against which to rehearse new skills or try alternative strategies and reinforce what actually works, then AIs could experience orders of magnitude more simulated samples in the same wall clock time.”

Claim: If an AI can build a useful simulation of reality for rehearsal and strategy testing, dreaming could provide orders of magnitude more simulated samples per unit of wall-clock time.

“A couple of years after DeepMind released AlphaZero, a group of researchers trained a model called EfficientZero, and the whole point of this model is to be very efficient with data.”

Claim: EfficientZero was developed after AlphaZero with the aim of improving data efficiency.

“So if this model and a human both got two hours to play against a simulator of an Atari game that they hadn't seen before, this model would actually probably beat the novice human.”

Claim: EfficientZero would probably outperform a novice human after each receives two hours with an unfamiliar Atari game simulator.

“Does this mean that the model was more sample efficient than the humans? Well, that was the goal of the training, but it depends on how you measure sample efficiency because for each step in the real game, Efficient Zero is playing dozens of simulated games in its head. In a similar way, future LLMs might be able to...”

Claim: EfficientZero’s apparent sample efficiency relative to humans depends on whether its dozens of internal simulated games per real-game step are counted.

“Efficient Zero is playing dozens of simulated games in its head. In a similar way, future LLMs might be able to... be able to consume far less real-world data while practicing endlessly against environments that they build for themselves.”

Claim: Future LLMs might reduce their need for real-world data by practicing extensively in self-generated environments.

“The big difference, of course, is that it will be much harder to build a simulation of the whole world than it is to emulate the game of Go.”

Claim: Simulating the entire world is much more difficult than emulating the game of Go.

“That's why I said this is a much more speculative idea. If it works, it would become a fourth axis of scaling alongside pre-training, RL, and inference time compute. You could call it test time training or dreaming.”

Claim: If successful, dreaming or test-time training would add a fourth scaling axis alongside pretraining, reinforcement learning, and inference-time compute.

“The model spends compute writing up RL environments and then training against them, and it's rehearsing all the skills that will actually be used in production for a specific user.”

Claim: Dreaming would use compute to create and train in RL environments that rehearse skills relevant to a particular user’s production tasks.

“So instead of hitting forward slash compact in Kodak's or cursor or CLAWD, which kindles a small amount of compute to write up a summary and which gives you the simulacrum of continual learning,”

Claim: A compact command in coding tools uses limited compute to summarize context, offering only an imitation of continual learning rather than extensive simulated practice.

“you hit forward slash dream. And this incinerates huge amounts of compute to build and train against a video game version of what the model is witnessing in the real world.”

Claim: A hypothetical dream command would spend substantial compute building and training in a game-like simulation of the model’s real-world observations.

“So what might continual learning look like by 2027 or 2028 and how do we get there? Here's one scenario.”

Claim: The speaker offers a speculative scenario for continual learning by 2027 or 2028.

“All of this RL VR training is producing an agent that can get its bearings when it's thrown at an unfamiliar problem, and it can try different strategies, and it can iterate when it hits a roadblock.”

Claim: RLVR training could produce agents that can orient themselves in unfamiliar problems, test strategies, and adapt when blocked.

“This is the crucial thing that RL VR has given you, an AI that is at least competent enough to start getting some real-world experience if it could learn from it. And once you have that, you send it out into the world to do real work, even on projects that are off the training distribution.”

Claim: RLVR could make an AI competent enough to undertake real-world work, including off-distribution projects, provided it can learn from experience.

“Now let's say at this point, the effective context lengths have expanded such that AIs can jam and co-work with you for a full week of wall clock time.”

Claim: The speaker’s scenario assumes future context lengths will let AIs collaborate with a user for an entire week.

“At the end of a week, you give it a thumbs up or a thumbs down. Give it a work review.”

Claim: The proposed continual-learning process uses a user’s weekly thumbs-up or thumbs-down work review as feedback.

“If you give it a thumbs up, the base model distills everything that the AI learned during the session, and it may use OPSD, it may use dreaming, it may use some other technique that we aren't even aware of, or it'll use a combination of all of the above.”

Claim: In the proposed scenario, positive user feedback triggers the base model to distill session learning using OPSD, dreaming, another method, or a combination.

“And once it does so, the AI starts getting better at domains that are adjacent to what it was explicitly trained to beforehand with our LVR training. And in the next round after that, it can get better at things that are adjacent to... to what it had previously online learned.”

Claim: Continual learning could expand AI capability iteratively from RLVR-trained domains to adjacent domains and then to domains adjacent to prior online learning.

“In this way, the gamut of AI skills and knowledge and capabilities can expand far beyond the verifiable domains that the model was originally trained against before it was deployed.”

Claim: Continual learning could extend AI capabilities well beyond the verifiable domains used for predeployment training.

“Just as free training created a base intelligence that was smart enough to become a competent agent with enough ROVR on top, so ROVR has created an agent that is competent enough to be actually broadly deployed in the world. And from this broad deployment to learn on the job, once the training recipe for continual learning actually arrives.”

Claim: The speaker’s proposed progression is that pretraining creates base intelligence, RLVR enables broad deployment, and continual learning enables on-the-job improvement.

“By this point, the main way that AIs get better is not from the training they have received before they are released to the public. Rather, it's from all this experience that they'll be accumulating from being broadly deployed in the economy and engaging in so many different kinds of tasks.”

Claim: In the speaker’s scenario, deployed experience across the economy eventually becomes the main source of AI improvement, surpassing pre-release training.

“Every time that you interact with an AI, it'll be smarter, not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world.”

Claim: With continual learning, an AI could improve after each interaction by learning from both a user’s prior sessions and other users’ interactions worldwide.

“And that's very scary and exciting and different from the way that AI improves right now.”

Claim: The speaker views continual, interaction-driven AI improvement as both frightening and exciting, and unlike current AI improvement.