AI researchers debate how close we are to recursive self-improvement

“like there's been a classic thing almost like marvex paradox right where like we see like you know we think of the ai being like if it can do this it's going to be amazing right like if it can solve these hard math problems if it can win a chess blah blah and then it solves these things and then it's like not that impactful obviously it's somewhat impactful but like not everything it's like if somehow that continues and like there's never like the true like spark of generalization that occurs i think that could lead to like the ai just being like extremely good at kind of everything that people like put into a benchmark to incorporate into an environment but like there's still some persistent like sim to real which is somehow blocking everything i think this is kind of unlikely i think we do actually see this kind of generalization even from RL in practice already but”

Claim: If AI systems do not achieve a true spark of generalization, they could become extremely good at benchmarked tasks while persistent simulation-to-reality problems limit their broader impact.

“to incorporate into an environment but like there's still some persistent like sim to real which is somehow blocking everything i think this is kind of unlikely i think we do actually see this kind of generalization even from RL in practice already but”

Claim: The speaker considers it unlikely that AI will remain limited to benchmark performance by persistent simulation-to-reality problems, because reinforcement learning already shows some generalization in practice.

“like if it is just like ridiculously hard to like generalize meta learning plus like we don't solve container learning it's just like super hard and impossible like this would be my like default scenario in that case”

Claim: The speaker's default explanation for the absence of radically transformative superintelligence is that generalizing through meta-learning remains extremely hard and continual learning is not solved.

“Humans have a lot of advantages over models now and each time a new model comes out, it'll sort of, it'll catch up in some of these areas.”

Claim: Humans currently have many advantages over AI models, although each new model is expected to catch up in some of those areas.

“But like you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment or the models can't check themselves well enough.”

Claim: AI-assisted work is bottlenecked by models' weaker capabilities, poorer judgment, and insufficient ability to check their own work.

“People think where a new model comes out and people are blown away and they're like, this is it, this is AGI, but then they use it a bit and then it starts to feel dumb after a month or so.”

Claim: A recurring cycle occurs in which people initially call new AI models AGI after being impressed, but after about a month of use the models begin to feel unintelligent.

“So that cycle just might keep going and it's hard to predict how many times it's going to repeat.”

Claim: The cycle of initial excitement about new AI models followed by disappointment might continue, but its number of repetitions is difficult to predict.

“And like right now, you don't get explosive growth in capabilities because you still get bottlenecked enough when you're trying to do research and engineering that.”

Claim: AI capabilities are not currently growing explosively because research and engineering work remains bottlenecked.

“that even if the model can write way more code than a person it doesn't make you like a hundred times more productive but yes so maybe maybe there are just more of these cycles than we would than we would expect”

Claim: A model's ability to write far more code than a person does not necessarily make that person 100 times more productive.

“that even if the model can write way more code than a person it doesn't make you like a hundred times more productive but yes so maybe maybe there are just more of these cycles than we would than we would expect”

Claim: The cycle of excitement about new AI models followed by disappointment may repeat more times than people expect.

“i think people imagine that even once like once you have a an agent which is better than all humans at AI research even if it's like 0.1 better than all humans then the fact that you can run like you know hundreds of thousands if not millions of these in parallel you can run them much faster like chips going to speed up that's going to outweigh every other like bottleneck and like you're eventually just going to like hit this like very fast takeoff with recursive self-improvement I could imagine that if we continue along the trajectory that we're currently on”

Claim: If the current AI trajectory continues, an agent only slightly better than all humans at AI research might be replicated hundreds of thousands or millions of times and run faster, overcoming other bottlenecks and producing rapid recursive self-improvement.

“with that paradigm where you know it's basically just like self-attention rl scaling up rl environments the i guess like if you think about what happened with moore's law right like we had this very like nice straight line and that held for a really really long time but there were so many like discrete like discontinuities and innovations that had to happen to keep that scaling law going and the same thing has kind of happened with llms like we had this pre-training like scaling law and then that was kind of like you know hitting the dimension”

Claim: Moore's law maintained a seemingly straight scaling trajectory for a long time, but sustaining it required many discrete innovations and discontinuities.

“with that paradigm where you know it's basically just like self-attention rl scaling up rl environments the i guess like if you think about what happened with moore's law right like we had this very like nice straight line and that held for a really really long time but there were so many like discrete like discontinuities and innovations that had to happen to keep that scaling law going and the same thing has kind of happened with llms like we had this pre-training like scaling law and then that was kind of like you know hitting the dimension diminishing returns and then we came up with like RL and solved that and then we got this new like you know diminishing returns curve to hit that made it keep looking like a straight line going up and so like if it requires another one of those discontinuities to solve like I'm not sure that like the current method of like training elements with these RL environments even like RSI targeted RL environments would be able to discover that discontinuity and if not like we're probably going to hit this like asymptotic like curve where like”

Claim: After LLM pretraining encountered diminishing returns, reinforcement learning helped extend progress, which then encountered another diminishing-returns curve.

“diminishing returns and then we came up with like RL and solved that and then we got this new like you know diminishing returns curve to hit that made it keep looking like a straight line going up and so like if it requires another one of those discontinuities to solve like I'm not sure that like the current method of like training elements with these RL environments even like RSI targeted RL environments would be able to discover that discontinuity and if not like we're probably going to hit this like asymptotic like curve where like”

Claim: If sustaining AI progress requires a new discontinuity that current methods cannot discover, progress will probably approach an asymptotic curve.

“diminishing returns and then we came up with like RL and solved that and then we got this new like you know diminishing returns curve to hit that made it keep looking like a straight line going up and so like if it requires another one of those discontinuities to solve like I'm not sure that like the current method of like training elements with these RL environments even like RSI targeted RL environments would be able to discover that discontinuity and if not like we're probably going to hit this like asymptotic like curve where like”

Claim: The speaker is uncertain whether current training methods based on reinforcement-learning environments, including RSI-targeted environments, can discover the next innovation needed to sustain AI progress.

“we have the answer that we kind of have the ability to implement it but like maybe there's the distinguish we should distinguish between a discontinuity which adds to the current paradigm again it's like cumulative like there's something beyond the rl that we have to discover maybe they're capable of like you know connecting the dots in that straight line or like like again how far off the global optimum are we do we have to We have to go back and throw out like, you know, gradient descent and like neural nets in general.”

Claim: The speaker distinguishes innovations that cumulatively extend the current AI paradigm from a fundamental shift that might require abandoning gradient descent and neural networks.

“And I don't think like if you continue to scale up the current paradigm and LLM, no matter how many LLMs you're running are capable of necessarily discovering that if it's too far away.”

Claim: Scaling the current LLM paradigm, even across many models, may not discover an alternative that is too far from the current approach.

“And I don't think like if you continue to scale up the current paradigm and LLM, no matter how many LLMs you're running are capable of necessarily discovering that if it's too far away.”

Claim: The speaker doubts that scaling the current LLM paradigm, even across many models, will necessarily let AI discover a sufficiently distant new paradigm.

“The only hope really is if deep learning just can't get us to an AI, which is at least. can dominate human research and human development including the human ability to come up with new paradigms and so forth or like i don't know maybe human maybe humans would also never have discovered the the next learning architecture but um to the extent human sort of discovered it eventually but it just seems like i don't know if if you just look at the progress this happens in 2012 till now”

Claim: The speaker identifies as the only hope that deep learning cannot produce AI capable of dominating human research and development, including the creation of new paradigms.

“and you just continue that on i mean i know if that's been powered by huge amounts of compute scaling and so forth but um uh it would be weird if like it just didn't get to the point where it could like dominate humans at least in r&d especially over the next few years there's going to be ryan greenbaum was on the podcast recently and he made this point that you could imagine as ais get more and more capable and are capable of making progress on simulations which incentivize getting better at not only AI R&D but generally at science so this is the thing that all the labs targeting many startups are targeting or another intuition pump is if you look at the ELO score of chess bots since the 80s there's just like a very linear increase in ELO over time but there's this huge discontinuity as they cross the human range of human experts always win against AIs to like human experts never win against AIs”

Claim: The speaker suggests that increasingly capable AI systems may advance simulations that incentivize progress in AI research and development and science more generally.

“and you just continue that on i mean i know if that's been powered by huge amounts of compute scaling and so forth but um uh it would be weird if like it just didn't get to the point where it could like dominate humans at least in r&d especially over the next few years there's going to be ryan greenbaum was on the podcast recently and he made this point that you could imagine as ais get more and more capable and are capable of making progress on”

Claim: The speaker expects that continuing AI progress since 2012, supported by substantial compute scaling, would likely lead AI to dominate humans in research and development, particularly over the next few years.

“simulations which incentivize getting better at not only AI R&D but generally at science so this is the thing that all the labs targeting many startups are targeting or another intuition pump is if you look at the ELO score of chess bots since the 80s there's just like a very linear increase in ELO over time but there's this huge discontinuity as they cross the human range of human experts always win against AIs to like human experts never win against AIs”

Claim: The speaker describes chess-bot ELO as increasing roughly linearly from the 1980s, then undergoing a sharp transition around human-expert performance, after which human experts no longer won against AIs.

“So far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world, but that is because they're slowly rising in ELO relative to humans.”

Claim: The speaker attributes AI’s limited economic impact so far to its gradual increase in ELO relative to human performance.

“yeah i mean i agree it would be very i mean the only way for this to not happen is if like as you said somehow asymptotes like just before basically because we're already pretty close in my opinion to like where we'll start crossing like the human elo score and so we'll need to asymptote before that and like that's the only way you know in this scenario you post where like somehow we're sitting here in 2035 and like everything”

Claim: The speaker believes AI is already close to human-level ELO and says the discussed scenario would fail to occur only if AI capability growth asymptoted just before crossing that level.

“everything is normal for this to happen i think yeah i mean the only other way is like there's like some dramatic like regulation on ai it's like this is kind of what i see is like the most likely way for this scenario to happen actually rather than the technical thing”

Claim: The speaker considers dramatic AI regulation a possible—and more likely than a technical limitation—reason the discussed scenario might not occur.

“yeah i i think there's different kinds of research there's like research where it's like the auto research style where the objective is already specified very cleanly you're optimizing that objective and i think everyone is picturing like if we continue along this path of like you know making pre-training loss go down making our environment together what and those go up that's going to lead to like improvement but like yeah maybe what Ryan is talking about is like this much more open-ended type of science which is required for like paradigm shifts where we can't specify the objective and the AIs are definitely not able to specify that objective either like we have to be really really careful about how we specify objectives for any of these things”

Claim: The speaker distinguishes auto-research with a clearly specified objective from open-ended science for paradigm shifts, where neither humans nor AI can specify the objective.

“together what and those go up that's going to lead to like improvement but like yeah maybe what Ryan is talking about is like this much more open-ended type of science which is required for like paradigm shifts where we can't specify the objective and the AIs are definitely not able to specify that objective either like we have to be really really careful about how we specify objectives for any of these things”

Claim: The speaker argues that objectives for AI systems must be specified with great care.

“um i actually john i'm curious for your like sort of uh wisdom of the ages of um or wisdom of like being in the trenches way back when but presumably a big breakthrough was realizing that next token prediction is the like you wouldn't have thought that nanos nano gpt speed run is the thing to be optimizing for in 2014 but now that we have come to this new paradigm that's you would think to do a speed run on that and have you guys get really good at that but maybe there's like a next”

Claim: The speaker proposes that recognizing next-token prediction as an optimization target may have been a major AI breakthrough.

“inner loop to optimize that the AIs wouldn't anticipate and there's an outer loop of like revenue or something that eventually should be strong but it's a very slow outer loop.”

Claim: The speaker suggests AI development may involve an unanticipated inner optimization loop and a slower outer loop, such as revenue, that should eventually become strong.

“Yeah, in fact, I remember in the early open AI days having the intuition that actually just do like minimizing log loss wasn't going to get you to intelligence because like the important bits are accounting for. There was such a small fraction of the loss that like it was going to be overwhelmed by noise.”

Claim: The speaker recalls believing in OpenAI’s early days that minimizing log loss alone would not yield intelligence because important information made up too little of the loss and would be overwhelmed by noise.

“So just training a language model on next token prediction just wasn't going to learn the interesting things you wanted to learn.”

Claim: The speaker says next-token-prediction-only language-model training would not learn the desired interesting capabilities.

“And we needed to craft better objectives that would put more emphasis on the important things.”

Claim: The speaker says AI training needed better objectives that assign greater weight to important information or capabilities.

“And like you can make all sorts of arguments for this and you could say, oh, humans probably don't learn how to like we don't learn how to model everything in our environment. We can't most like people can't create a photorealistic reproduction of some kind of scene they've looked at. um there must be we must need a better objective but then it turned out that it just worked anyway”

Claim: The speaker says next-token prediction proved effective despite prior arguments that a better learning objective was necessary.

“And as you were pointing out, the inner loop, even in current AI research of like post training benchmarks or whatever doesn't necessarily translate into what users like.”

Claim: The speaker says performance on current AI post-training benchmarks does not necessarily correspond to user preferences.

“The whole field relies a lot on generalization and it's very hard to predict when you're going to get generalization or when you're going to get some kind of out of distribution generalization.”

Claim: The speaker says AI research depends heavily on generalization, whose occurrence—including out-of-distribution generalization—is difficult to predict.

“So we know that if you train on the task you care about, you're going to do better.”

Claim: The speaker says models perform better on a task when trained directly on that task.

“But like the most important advances are often the types of generalization that we have no right to expect. So, for example, from.”

Claim: The speaker says major AI advances often depend on forms of generalization that could not reasonably have been anticipated a priori.

“But like the most important advances are often the types of generalization that we have no right to expect. So, for example, from.”

Claim: The most important advances often involve forms of generalization that researchers have no reason to expect in advance.

“Just pre-training on this very naive next token prediction objective to like various tasks of interest where some very that require understanding of the input in some deep way or learning some skill from pre-training that's like very rare and like not very like heavily represented and then like”

Claim: The speaker says naive next-token pretraining can generalize to tasks requiring deep input understanding or skills that were rare in the training data.

“Also, generalization from these verifiable tasks to less verifiable ones. This is also a type of generalization that there's no reason a priori to expect it.”

Claim: The speaker says transfer from verifiable tasks to less verifiable tasks is an unexpected form of generalization that cannot be assumed in advance.

“Also, generalization from these verifiable tasks to less verifiable ones. This is also a type of generalization that there's no reason a priori to expect it.”

Claim: Generalization from verifiable tasks to less verifiable tasks is not something there is a priori reason to expect.

“um some sort of singularity very rapidly without even scaling up the inputs to ai progress that are not just ai labor is that before every single experiment you run that's you know like a seven figure experiment you spend an equivalent amount of compute on ai labor and so you just have automated versions of you guys spending a century uh thinking about like what is the optimal experiment to run um do you like small scale ablations developing literally like a century's worth of theory so going back even before like deep learning before you decide what experiment to run doing extremely optimal like setting up of the experiment then you do a century of thinking after the experiment is over where you're like analyzing what happened and what the next experiment to run is”

Claim: If AI labor can use compute comparable to the compute for a seven-figure experiment, automated researchers could spend the equivalent of a century planning the experiment, running small-scale ablations, and developing theory before the experiment.

“Well, I think if you think hard enough, you probably could have expected some of these things beforehand.”

Claim: Some of these advances probably could have been anticipated through sufficiently careful thought.

“Like there is probably some very clever way to do a small scale. scale experiment that'll let you build the theory that then will generalize to the large-scale experiment so i would expect that like we're nowhere near the ceiling of how well you can do research and like”

Claim: A clever small-scale experiment could probably enable researchers to build a theory that generalizes to a large-scale experiment.

“scale experiment that'll let you build the theory that then will generalize to the large-scale experiment so i would expect that like we're nowhere near the ceiling of how well you can do research and like”

Claim: Research quality is nowhere near its potential ceiling.

“I would imagine a future where AI is doing a lot of analysis and theory building, like spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far.”

Claim: The speaker expects that AI will eventually perform substantial research analysis and theory-building, using compute comparable to the compute spent on experiments.

“think there's like really concrete examples of this when the objective is well specified.”

Claim: When an objective is well specified, there are concrete examples of reasoning improving outcomes.

“and like all thinking can do is like update your posterior based on like you know the bits that you've gotten since you formed your prior like you can't gain any new bits from like just thinking but when the objective is well specified and there is like this data sitting around like i imagine there will be this big speed up in the current paradigm we're in and like good example this is like”

Claim: Reasoning can update beliefs using information already received, but cannot generate new information beyond the evidence available when the prior was formed.

“you know if you've got an ai to think about like the caplan scaling laws like an ai at this point would have noticed that like you know oh like they've just taken these intermediate checkpoints and didn't account for like the annealing and so like this is wrong and like that would have called that like years earlier we would have made like you know progress like like would have cut off a year or two of progress just from that like observation from an ai um and like again once the objective is well specified which is like lower pre-training loss or whatever”

Claim: An AI examining the Kaplan scaling laws might have identified years earlier that intermediate checkpoints were used without accounting for annealing, potentially saving one or two years of progress.

“like there's many many good examples where if you just thought about it a bit more you would have been able to like cut down a significant on things that you've done so like mu p and like how learning rate scales with like model size and like realizing that model width is important in that as well um like i feel like you can you can really back out a lot of these things and cut off like a lot of low-hanging fruits i would imagine like a 10 times speed up if our thing is just like maximize the objective we're currently on but i don't see that how that generalizes at all to you know come up with the right objective in the first place like just thinking doesn't necessarily buy you the right objective in the first place”

Claim: The speaker does not see how reasoning that improves optimization of the current objective generalizes to discovering the right objective in the first place.

“like there's many many good examples where if you just thought about it a bit more you would have been able to like cut down a significant on things that you've done so like mu p and like how learning rate scales with like model size and like realizing that model width is important in that as well um like i feel like you can you can really back out a lot of these things and cut off like a lot of low-hanging fruits i would imagine like a 10 times speed up if our thing is just like maximize the objective we're currently on but i don't see that how that generalizes at all to”

Claim: More reasoning could yield an estimated tenfold speedup when the task is to maximize the current objective.

“like there's many many good examples where if you just thought about it a bit more you would have been able to like cut down a significant on things that you've done so like mu p and like how learning rate scales with like model size and like realizing that model width is important in that as well um like i feel like you can you can really back out a lot of these things and cut off like a lot of low-hanging fruits i would imagine like a 10 times speed up if our thing is just like maximize the objective we're currently on but i don't see that how that generalizes at all to”

Claim: More careful reasoning could have uncovered issues such as learning-rate scaling with model size and the role of model width, avoiding substantial low-hanging-fruit work.

“i mean yeah i think this is really the key question to like any kind of like very rapid rsis like from current ais is like how well can ais generalize like learning their own objectives because have any kind of like self-propelling automated loop you need the ai to like propose objectives optimize them figure that out propose a new objective and like have this like not go off the rails at like any point for like a long long time”

Claim: A self-propelling automated AI loop requires the AI to propose and optimize objectives, choose subsequent objectives, and remain on track over a long period.

“um coming back to more of x paradox there might be like a case of more of x paradox where like we think this kind of like autonomy and sort of like being like self-encapsulated so we can you know think of what we should do ourselves and then go do it and like have this loop is like super easy because we always do this and like obviously evolution needs to create creatures that can like survive on them by themselves like long period of time and like this just might be something that for some reason it's like really hard for the ai in the same way that like locomotion stuff is really hard but it's like math is super easy despite being super”

Claim: AI autonomy and self-directed action might be unexpectedly difficult for AI, even though humans routinely do them.

“suggest that that's you would yeah exactly i mean this is another possibility which like but i agree like there's no obvious evidence for this like in fact the fact that you know rl agents not being like super persistent and it's quite easy to do this it's kind of evidence against this but like this would be you know potentially like one of the reasons why like we just don't get this like immediate takeoff is like if this is hard”

Claim: If AI autonomy is difficult, that difficulty could be one reason an immediate takeoff does not occur.

“suggest that that's you would yeah exactly i mean this is another possibility which like but i agree like there's no obvious evidence for this like in fact the fact that you know rl agents not being like super persistent and it's quite easy to do this it's kind of evidence against this but like this would be you know potentially like one of the reasons why like we just don't get this like immediate takeoff is like if this is hard”

Claim: There is no obvious evidence that AI autonomy is unusually difficult, and the limited persistence of reinforcement-learning agents provides some evidence against that possibility.

“probably just like iteratively asking the right questions like if you can get the ai to like do any experiment but like you need to decide what experiments to do and like right now i think ai's are not very good at this compared compared to coding the experiment at all like whenever we talk about research they propose like a bunch of like miscellaneous things which are like very very tiny steps”

Claim: Current AI systems asked to conduct research tend to suggest miscellaneous ideas that make only very small advances.

“probably just like iteratively asking the right questions like if you can get the ai to like do any experiment but like you need to decide what experiments to do and like right now i think ai's are not very good at this compared compared to coding the experiment at all like whenever we talk about research they propose like a bunch of like miscellaneous things which are like very very tiny steps”

Claim: The speaker thinks current AI systems are much worse at choosing research experiments than at implementing experiments, and that their research suggestions tend to be miscellaneous, incremental steps.

“probably just like iteratively asking the right questions like if you can get the ai to like do any experiment but like you need to decide what experiments to do and like right now i think ai's are not very good at this compared compared to coding the experiment at all like whenever we talk about research they propose like a bunch of like miscellaneous things which are like very very tiny steps”

Claim: Current AI systems are much less capable at choosing research experiments than at coding experiments.

“a very wide swath of data like and then even once radford discovered that right like it took a while before people decided to scale up because we had to come with the idea of scaling laws and the fact that like you can very reliably predict these things”

Claim: The speaker says that after Radford discovered next-token prediction over a wide range of data, it took time for people to scale up because scaling laws and reliable prediction of outcomes had not yet been established.

“job for humans uh or the the role for humans that will last the longest is like defining the objective uh and like deciding what we actually want um so like in that vein”

Claim: The speaker expects defining objectives and deciding what people want to be the human role that lasts longest as AI capabilities advance.

“something like deciding what how is it like the AI the assistants should behave or what it means to be helpful or what's like the objective when we're doing our all from human feedback is one such thing and then like then later like defining like constitutions and model specs is another one and I think even if the AIs can do all the technical work we'll have to still do a lot of that and decide what we actually want.”

Claim: The speaker expects humans to continue defining desired AI behavior and objectives—including helpfulness, human-feedback objectives, constitutions, and model specifications—even if AI systems can perform all technical work.

“Yeah, alignment is sort of the answer, but it's also alignment itself can be kind of decomposed into into like specification of the objective or figuring out what the right objective should be. And then. like actually like achieving or optimizing the objective you've defined and I think the first one is not going to go any way anytime soon”

Claim: The speaker divides alignment into determining the right objective and optimizing it, and expects the need for humans to determine the objective to persist for the foreseeable future.

“and like if I think about like a post-training team and why you need a lot of people to be on the team it's it's it's just because there are a lot of different like areas where you have to figure out like how the model should behave”

Claim: The speaker says post-training teams require many people because they must determine appropriate model behavior across many areas.

“And like, there's no way of like, there's no way of, it would be very hard to automate the whole thing just because someone has to think about how, how should the model behave in this area?”

Claim: The speaker thinks fully automating the process of determining model behavior would be very difficult because people must decide how the model should behave in each area.

“The thing that I think is most impressive about Antithesis is we started using it in a team that was building high assurance software and being really careful and nonetheless it was able to shake out bugs. That were otherwise going to be really hard to find and that's important both because it helps make those systems more reliable But also because it helps the teams that build it to just move faster I”

Claim: The speaker says Antithesis uncovered difficult-to-find bugs in software developed by a careful team building high-assurance systems.

“That were otherwise going to be really hard to find and that's important both because it helps make those systems more reliable But also because it helps the teams that build it to just move faster I”

Claim: The speaker says Antithesis's discovery of hard-to-find bugs improves system reliability and helps development teams work faster.

“think in general as we've been using agents more and more the key problem that you run into is the verification bottleneck just the the time it takes from people to look at code and figure out is that actually something you want to accept in your production”

Claim: The speaker says increased use of agents makes verification a key bottleneck because people need time to review generated code and decide whether it should enter production.

“software and tools that make testing better are just incredibly helpful there they just ease the verification bottleneck and make it possible for you to get more stuff done and move faster because you can have more confidence that the code generated by the agent is actually not introducing new problems”

Claim: The speaker says better testing tools reduce the verification bottleneck and help teams work faster by increasing confidence that agent-generated code has not introduced new problems.

“Yeah, I think distillation is the main thing that fights against the centralizing force, because basically anything that can be learned through RL can be distilled very easily.”

Claim: The speaker thinks distillation is the main counterforce to centralization among model providers, since capabilities learned through reinforcement learning can be distilled very easily.

“um like it's a small number of bits uh it's something that you can learn from a small amount of data so if you can collect if you can get uh trajectories from the model that uh show a behavior you can easily distill it so i think um i think distillation is one of the uh things that fight centralization uh there's also um”

Claim: The speaker says a model behavior can be distilled from a small amount of data when trajectories demonstrating that behavior are available.

“I mean, there is a possibility that there will be company specific models that it'll be possible to learn from deployment and have a company continually improving its own model.”

Claim: The speaker says it is possible that companies will use company-specific models that learn from deployment and are continually improved by the company.

“And such a system could be provided by the current oligopoly of model providers or some other currently smaller company.”

Claim: The speaker says company-specific, deployment-learning models could be supplied either by the current oligopoly of model providers or by a currently smaller company.

“But I think that'll change the game a bit.”

Claim: The speaker expects company-specific models that learn from deployment to change the competitive landscape somewhat.

“For just distilling with supervised learning, the prompt distribution is extremely important.”

Claim: The prompt distribution is extremely important when distilling a model with supervised learning.

“For just distilling with supervised learning, the prompt distribution is extremely important.”

Claim: The speaker says the prompt distribution is extremely important when distilling a model with supervised learning.

“So it's very non-trivial to distill a model, even if you have full access to it and have the chain of thought and everything.”

Claim: Distilling a model is very non-trivial even when the distiller has full access to the model and its chain of thought.

“So it's very non-trivial to distill a model, even if you have full access to it and have the chain of thought and everything.”

Claim: The speaker says model distillation remains non-trivial even when the distiller has full model access and access to its chain of thought.

“distill all of the all the useful capabilities from it because you need to prompt the model with something and you need to prompt it with like realistic prompts you need to have a really wide distribution of realistic prompts so yeah one thing that's been coming out recently is”

Claim: Capturing all of a model’s useful capabilities through distillation requires prompting it with a wide distribution of realistic prompts.

“distill all of the all the useful capabilities from it because you need to prompt the model with something and you need to prompt it with like realistic prompts you need to have a really wide distribution of realistic prompts so yeah one thing that's been coming out recently is”

Claim: The speaker says distilling all useful model capabilities requires prompting the model with realistic prompts from a very broad distribution.

“Some of the Chinese companies are probably using these router services, which are designed to allow people in China to use the U.S. frontier models, which would otherwise be blocked in China.”

Claim: The speaker suspects some Chinese companies use router services that enable people in China to access U.S. frontier models otherwise blocked in China.

“Some of the Chinese companies are probably using these router services, which are designed to allow people in China to use the U.S. frontier models, which would otherwise be blocked in China.”

Claim: The speaker thinks some Chinese companies are probably using router services to provide people in China access to U.S. frontier models that would otherwise be blocked in China.

“But there are all these router or proxy services that allow people in China to use these models mostly for coding.”

Claim: The speaker says router and proxy services let people in China use U.S. frontier models, mostly for coding.

“But there are all these router or proxy services that allow people in China to use these models mostly for coding.”

Claim: Router and proxy services allow people in China to use the models, mostly for coding.

“And these router services are collecting and selling some of the data.”

Claim: The router services collect and sell some of the data generated through their use.

“And these router services are collecting and selling some of the data.”

Claim: The speaker says router services that provide access to models collect and sell some user data.

“A very useful data set for distillation because it gives you the perfect prompt distribution.”

Claim: The speaker thinks data collected through router services is very useful for distillation because it provides an ideal prompt distribution.

“A very useful data set for distillation because it gives you the perfect prompt distribution.”

Claim: The data collected through router services is a very useful distillation dataset because it provides an ideal prompt distribution.

“models and so it's like you can automate like an awful lot of this like prompt distribution gathering and like environment creation it's just like humans need to provide like increasingly few amounts of bits it's like the models get better”

Claim: As models improve, AI can automate much of prompt-distribution gathering and environment creation, reducing the amount of information humans need to provide.

“Not necessarily, I mean like yeah that's obviously very helpful but like theoretically you can just think about like what users want or like a lot of tasks”

Claim: A service with real users is helpful but not necessarily required to generate useful prompts or tasks, since one can theoretically reason about what users want.

“Yeah, I mean like ultimately like if you have this like fully automated loop that is basically our side right like the AI deciding the data it's deciding the training that that is the loop but yeah I mean like it depends how much human information you need like at some point if you're just like I want traces that look like this you prompt that to the model the model will be able to like come up with like a pretty good approximation.”

Claim: When prompted to produce traces with specified characteristics, a model can generate a fairly good approximation.

“Yeah, I mean like ultimately like if you have this like fully automated loop that is basically our side right like the AI deciding the data it's deciding the training that that is the loop but yeah I mean like it depends how much human information you need like at some point if you're just like I want traces that look like this you prompt that to the model the model will be able to like come up with like a pretty good approximation.”

Claim: A loop in which AI autonomously chooses the data and training constitutes recursive self-improvement.

“this is actually i think easier for the distillers than the frontier labs right because the distillers just like i want a good politician they go to the like frontier model the frontier model already knows how to be a good politician so it just like generates those traces whereas like if you actually want to build the first model that does this you have to like actually somehow like get data on like what politicians do every day and like build that so it's actually much easier to like say like i want something like this and then like And I'd like the AI to produce like a billion variations than to like actually create the thing like this to begin with.”

Claim: It is easier for distillers to use a frontier model to generate many examples of a target behavior than it is to build the capability and its data from scratch.

“the distillation from like methos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so that you can distill like kind of these behaviors and things in i think the prediction from this is that the frontier labs don't necessarily have much of advantage if at all in oral environments now because yes like user distribution matters for like general like behavior and and so on but like the best measure of a capability is the very very”

Claim: The speaker predicts that frontier labs may have little or no advantage in real-world environments, even though user distributions matter for general behavior.

“the distillation from like methos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so that you can distill like kind of these behaviors and things in i think the prediction from this is that the frontier labs don't necessarily have much of advantage if at all in oral environments now because yes like user distribution matters for like general like behavior and and so on but like the best measure of a capability is the very very very hard oral environments you've made at the frontier and so if you have access to those oral environments as anthropic and you have access to logic disflation and you've still made a worse model then maybe”

Claim: The speaker argues that a capability is best measured in very difficult real-world environments created at the frontier.

“the distillation from like methos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so that you can distill like kind of these behaviors and things in i think the prediction from this is that the frontier labs don't necessarily have much of advantage if at all in oral environments now because yes like user distribution matters for like general like behavior and and so on but like the best measure of a capability is the very very”

Claim: The prompt distribution matters greatly for distillation, and observing user activity helps capture the behaviors to be distilled.

“real world deployment matters more than the environment.”

Claim: If a lab has access to difficult frontier environments and distillation yet produces a worse model, real-world deployment may matter more than the environment.

“maybe like maybe we're just in this weird like uncanny valley where you know like actually trying to copy that frontier model too much like the student teacher gap whatever it”

Claim: The speaker hypothesizes that copying a frontier model too closely may fail because the student-teacher gap is too large, creating an uncanny valley.

“is just like too large and like i think people people made this point with opus is it's like the difference between opus forming six and opus five is that opus five really feels like it's got this like ai as a judge checking every possible thing it's done it's like that's why it uses so many tokens it like tries to think about all these things but it doesn't necessarily have the big model smell of fable to know when to like stop doing that or like when's a good path to go down order”

Claim: The speaker says Opus 5 may lack the larger model’s intuition for when to stop checking or which course of action to pursue.

“is just like too large and like i think people people made this point with opus is it's like the difference between opus forming six and opus five is that opus five really feels like it's got this like ai as a judge checking every possible thing it's done it's like that's why it uses so many tokens it like tries to think about all these things but it doesn't necessarily have the big model smell of fable to know when to like stop doing that or like when's a good path to go down order”

Claim: The speaker says Opus 5 appears to use an AI judge to check its actions extensively, resulting in high token use.

“So I would say there are a couple of different axes for the environments you can create. And like one of them is difficulty and the other is realism.”

Claim: Created environments can be characterized along at least two axes: difficulty and realism.

“It's sort of easy to create or it's comparatively easy to create a lot of difficult environments like.”

Claim: Creating a large number of difficult environments is comparatively easy.

“that are just like involve like doing a much more complicated task or doing something that requires a lot more cleverness and you could you could say this is like the benchmarking distribution because a lot of the most prominent benchmarks just involve doing some very hard puzzle like task that's easy to verify and then there's sort of like the realism axis where you want the model to be good in the realistic coding agent setting where there's like multiple back and forth. back and forth with the human and there's like multiple objectives and like I'd say”

Claim: Prominent AI benchmarks often emphasize difficult, easily verifiable puzzle-like tasks, whereas realistic coding-agent settings involve multiple interactions with a human and multiple objectives.

“Like the people like the labs who are crafting the model behavior for the first time need to push in both directions and to get good model behavior,”

Claim: AI labs developing model behavior need to optimize for both benchmark performance and realism.

“you need to really push on the realism axis and have like rubrics or some kind of human feedback that's informing the reward function you use there.”

Claim: Developing good model behavior requires realistic training settings and rubrics or human feedback that inform the reward function.

“But I think when if you try to do distillation naively, you end up just sort of matching the teacher on the bench.”

Claim: Naive distillation tends to make a student match its teacher on the benchmark distribution.

“benchmaxing distribution and um but if you don't have enough of the environments that really exercise the capabilities in these um like trickier realistic settings then you're not going to you're not going to get those into your student model and i think maybe one thing that's happening is the big models uh generalize better from the uh like the tricky narrow tasks to these sort of uh more realistic tasks um so”

Claim: If distillation lacks enough environments that exercise capabilities in difficult realistic settings, the student model will not acquire those capabilities.

“benchmaxing distribution and um but if you don't have enough of the environments that really exercise the capabilities in these um like trickier realistic settings then you're not going to you're not going to get those into your student model and i think maybe one thing that's happening is the big models uh generalize better from the uh like the tricky narrow tasks to these sort of uh more realistic tasks um so”

Claim: Larger AI models may generalize better than smaller models from difficult narrow tasks to more realistic tasks.

“So if you have a really good, like, realistic prompt distribution for distillation, you can match the big model really well.”

Claim: A strong realistic prompt distribution for distillation can enable a smaller model to closely match a larger model.

“But if you only have this, like, this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks.”

Claim: Distillation using only easily verifiable tasks can match a larger model across benchmarks.

“But you do worse on this broader distribution.”

Claim: A model distilled only on easily verifiable tasks performs worse on a broader, more realistic task distribution.

“So you might, that might even explain something about the smaller anthropic models like Sonnet 5 though it's hard to predict exactly what what they're doing to post train those models it could also be that they're always changing their post training stack and they just made they just got a few things wrong in some of these models so they like I don't know they they turned up like so something too high and created some quirks that people really don't like”

Claim: The benchmark-versus-realism tradeoff might explain some behavior of smaller Anthropic models such as Sonnet 5, although the speaker is uncertain about their post-training methods.

“something about the smaller anthropic models like Sonnet 5 though it's hard to predict exactly what what they're doing to post train those models it could also be that they're always changing their post training stack and they just made they just got a few things wrong in some of these models so they like I don't know they they turned up like so something too high and created some quirks that people really don't like”

Claim: Anthropic may frequently change its post-training stack, and mistakes in some models may create quirks that users dislike.

“Like, so it's like really easy to screw up post-training in some way that doesn't show up in benchmarks.”

Claim: Post-training can be mishandled in ways that are not detected by benchmarks.

“I think we'll probably do some combination of learning from human feedback to absorb like the researchers taste and just like creating a lot of practice environments which involve like doing multi-step research projects.”

Claim: The first AI systems capable of automating AI research and development will probably combine human feedback that captures researchers' preferences with many practice environments for multi-step research projects.

“People will in practice do some combination of those two things and just each iteration like patch whatever seems to be most broken in the last iteration. So like researchers will be using the AIs a lot and will notice that they have some consistent weaknesses and then those things will either be patched by like collecting human feedback or like creating environment.”

Claim: AI developers will likely combine human feedback and practice environments, addressing the most apparent weakness in each successive iteration.

“People will in practice do some combination of those two things and just each iteration like patch whatever seems to be most broken in the last iteration. So like researchers will be using the AIs a lot and will notice that they have some consistent weaknesses and then those things will either be patched by like collecting human feedback or like creating environment. the environments”

Claim: As researchers use AI systems, they will identify recurring weaknesses and address them through human feedback or purpose-built training environments.

“yeah yeah maybe maybe just what i think about this is like how much of the lineage we roll back and then let self play from there like i think in the limit like you're picturing like you know just giving them like a gpu and maybe neural nets or something and saying like okay figure out how to train a model to like do this particular task like the way it currently works is like we go up to the very like edge of the of the lineage and say okay like here are the bugs like you know anthropic is because found in their training stacks in the last few months will turn those into environments like you need to train and get better on the frontier and so you obviously lock in all the previous history of the lineage but you could imagine a world in which you roll back to like you know before grpo or something and then you have environments which like trying to get it to discover like the best form to like rl models”

Claim: The current approach to AI improvement retains the existing model lineage and converts recently discovered training-stack bugs into environments for further frontier training.

“on and then maybe you roll further and further back but i think we will be still so compute bottlenecked that like people will just keep like staying at the frontier”

Claim: AI development will likely remain constrained by compute, leading developers to keep improving from frontier models rather than rolling far back in the model lineage.

“yeah and like diffing essentially the bugs and whatever improvements they found since the last model version turning those into training environments”

Claim: Developers can convert bugs and improvements discovered since the previous model version into training environments.

“It's just again, this is basically continual learning within the AI lab of distilling the last three months of AI research progress through environments and like RLHF type stuff back into the model itself.”

Claim: The process amounts to continual learning within an AI lab, using environments and RLHF-like methods to distill the previous three months of AI research progress into the model.

“some of us feel like it's asymptotic is like you're always like just trying to get the last three months of progress and that progress is being contributed to by us of course but it also still has humans in the loop and it feels like you know you're just constantly inching closer and closer to what the human researchers are like finding capable of doing”

Claim: Iteratively distilling roughly the latest three months of research progress, with humans still in the loop, can feel asymptotic as AI systems gradually approach human researchers' capabilities.

“I mean the one thing I will say though is like obviously if you're just distilling on like trajectories you can never go above it but environments can go quite a far way above what a human can do like it's very easy to design an environment that like no human can.”

Claim: Training environments can set challenges that exceed human capabilities.

“I mean the one thing I will say though is like obviously if you're just distilling on like trajectories you can never go above it but environments can go quite a far way above what a human can do like it's very easy to design an environment that like no human can.”

Claim: Distilling only existing trajectories cannot produce performance beyond the capabilities represented in those trajectories.

“can solve for the AI can always be still trying to solve it and so that would be the path to like go ahead of just like what the human AI research”

Claim: Training environments with tasks beyond human ability could allow AI systems to surpass the capabilities of human AI researchers.

“can solve for the AI can always be still trying to solve it and so that would be the path to like go ahead of just like what the human AI research”

Claim: A training environment can be designed to pose a task that humans cannot solve while an AI can continue attempting it.

“I mean, I feel like in AI research, especially it's very easy to define like goals, which like, you know, you could say like the loss needs to be like 1.3 or something and like no human can get that now.”

Claim: AI research can use highly measurable goals, such as achieving a loss of 1.3 that no human can currently achieve.

“But like that's a very extremely measurable, verifiable task. And if the AI gets that, then great.”

Claim: Achieving a target such as a loss of 1.3 is an extremely measurable and verifiable AI task, and an AI reaching that target counts as success.

“I would say a lot of research is not exactly like that, though, where it's like hill climbing on a well-defined goal.”

Claim: Much AI research does not consist of hill climbing toward a well-defined goal.

“It's sort of more like here's an intuition we have about some way models should be better. And then we also have some idea for an algorithm that seems to go a little bit in this direction.”

Claim: Some AI research begins with an intuition about how models should improve and an algorithm that appears to advance that intuition.

“And so let's come up with a task that is sort of designed to show signs of life on this approach and like see if we get those signs of life. And then if we do, we can make successively more realistic versions of the task.”

Claim: Researchers may test an approach with a task designed to reveal early signs of promise, then create progressively more realistic versions if those signs appear.

“right it's like a lot more guided by intuition and then the the inner loop is to elicit the or make tests for that intuition rather than like the test itself leading to the insight.”

Claim: In intuition-led AI research, tests are designed to examine the guiding intuition rather than being the primary source of the insight.

“Like you're not directly optimizing for the eventual objective you care about or the practical like production objective.”

Claim: Some AI research does not directly optimize for its eventual objective or practical production objective.

“It's it's sort of you're you're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work and then like then try to get back to realism later after. after the method matures a little bit and”

Claim: Some AI research temporarily relaxes its objective and task realism to find workable methods, then increases realism after the method matures.

“then there's also like more uh there's research that's more oriented towards explaining things and like uh developing a theory or a sort of yeah often we don't have like mathematical theories in machine learning that are that um predictive but we have like a lot of um like more informal theories um for what's going on”

Claim: Some machine-learning research aims to explain phenomena and develop theory; mathematical theories are often not very predictive, while informal theories about what is happening are common.

“Yeah, I mean, like presumably the models will be trained on like some combination of of all of these tasks and like some will be very easily verifiable some will be like LMS judge or like just ask the human like does this look reasonable and then you will say the hope would be that like these would all generalize to like these much sort of hard sort of more vague fuzzy kind of tasks and like it probably will to some extent whether it generalizes enough that like we could the loop can become like self-sealing without humans being in the loop at all it's like unclear”

Claim: Models will presumably be trained on a mixture of tasks, including easily verifiable tasks and tasks evaluated by an LLM or a human.

“of all of these tasks and like some will be very easily verifiable some will be like LMS judge or like just ask the human like does this look reasonable and then you will say the hope would be that like these would all generalize to like these much sort of hard sort of more vague fuzzy kind of tasks and like it probably will to some extent whether it generalizes enough that like we could the loop can become like self-sealing without humans being in the loop at all it's like unclear”

Claim: It is unclear whether task generalization will be sufficient to make the AI research loop self-sealing without any human involvement.

“of all of these tasks and like some will be very easily verifiable some will be like LMS judge or like just ask the human like does this look reasonable and then you will say the hope would be that like these would all generalize to like these much sort of hard sort of more vague fuzzy kind of tasks and like it probably will to some extent whether it generalizes enough that like we could the loop can become like self-sealing without humans being in the loop at all it's like unclear”

Claim: The speaker expects training across varied tasks will probably generalize to harder and less clearly specified tasks to some extent.

“Yeah, I think it's now hard to separate out like how much of the lab's effort is going towards like direct RSI versus like making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run.”

Claim: SPEAKER_04 believes it is now difficult to separate labs’ effort on direct recursive self-improvement from effort on generally intelligent models deployed to earn revenue that funds subsequent large training runs.

“Yeah, I think it's now hard to separate out like how much of the lab's effort is going towards like direct RSI versus like making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run.”

Claim: The speaker finds it difficult to distinguish AI labs' efforts toward direct recursive self-improvement from efforts to build generally intelligent models that generate revenue to fund subsequent large training runs.

“you know i think for the um latter like yes that's probably just the bet they're making like and it's very clear like the pattern of like where these environments are going over the last few years i mean like anthropic's lineage of environments is like a very clear example of this like you know firstly like we just focus on coding and like we're going to get really really good at that and then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms like data available on the internet to create environments like their own internal”

Claim: Anthropic's training-environment strategy progressed from focusing on coding to expanding the range and horizon of tasks.

“you know i think for the um latter like yes that's probably just the bet they're making like and it's very clear like the pattern of like where these environments are going over the last few years i mean like anthropic's lineage of environments is like a very clear example of this like you know firstly like we just focus on coding and like we're going to get really really good at that and then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms like data available on the internet to create environments like their own internal”

Claim: Coding was an accessible starting point for creating training environments because substantial relevant data was available online.

“you know i think for the um latter like yes that's probably just the bet they're making like and it's very clear like the pattern of like where these environments are going over the last few years i mean like anthropic's lineage of environments is like a very clear example of this like you know firstly like we just focus on coding and like we're going to get really really good at that and then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms like data available on the internet to create environments like their own internal”

Claim: The speaker thinks AI labs are probably betting on building generally intelligent models that can be deployed and generate revenue.

“you know i think for the um latter like yes that's probably just the bet they're making like and it's very clear like the pattern of like where these environments are going over the last few years i mean like anthropic's lineage of environments is like a very clear example of this like you know firstly like we just focus on coding and like we're going to get really really good at that and then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms like data available on the internet to create environments like their own internal little stuff that they can turn into environments then we're going to generalize we're going to go up to finance next and like literally like just so much xl data and all that sort of stuff in the in the rl training and then you know it's powerpoints it's like this long tail of like the working economy and like that seemed to work really well and like all the other labs and thing even the open source labs have now realized that that was the correct thing”

Claim: SPEAKER_04 describes Anthropic’s environment-development progression as starting with coding, expanding to finance, and then extending to PowerPoint and other long-tail work-economy tasks.

“little stuff that they can turn into environments then we're going to generalize we're going to go up to finance next and like literally like just so much xl data and all that sort of stuff in the in the rl training and then you know it's powerpoints it's like this long tail of like the working economy and like that seemed to work really well and like all the other labs and thing even the open source labs have now realized that that was the correct thing”

Claim: SPEAKER_04 says other AI labs, including open-source labs, have come to regard the domain-by-domain approach as the right strategy.

“little stuff that they can turn into environments then we're going to generalize we're going to go up to finance next and like literally like just so much xl data and all that sort of stuff in the in the rl training and then you know it's powerpoints it's like this long tail of like the working economy and like that seemed to work really well and like all the other labs and thing even the open source labs have now realized that that was the correct thing”

Claim: The speaker says expanding training environments from coding into broader work domains seemed to work well and that other labs, including open-source labs, came to regard this as the right approach.

“little stuff that they can turn into environments then we're going to generalize we're going to go up to finance next and like literally like just so much xl data and all that sort of stuff in the in the rl training and then you know it's powerpoints it's like this long tail of like the working economy and like that seemed to work really well and like all the other labs and thing even the open source labs have now realized that that was the correct thing”

Claim: The described training-environment progression expanded from coding to finance, using abundant spreadsheet data for reinforcement learning, and then to PowerPoints and a broader range of work-economy tasks.

“One is just that this is, we expect models to get there soon, but they're not there yet.”

Claim: SPEAKER_00 offers as one possible explanation that models are expected to achieve human-like on-the-job learning soon, although they have not yet done so.

“I mean, if the models were good enough at learning in context, then in theory, you wouldn't need to train them on finance. They would just be able to figure out, read all the books on the fly and figure out how to do everything in the appropriate jurisdiction.”

Claim: SPEAKER_03 argues that sufficiently capable in-context learning could let models avoid finance-specific training by learning from books on the fly and adapting their work to the relevant jurisdiction.

“and you could argue that you need to do a lot of this domain-specific training just to make them more efficient. So even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights so the model would be more efficient at runtime.”

Claim: SPEAKER_03 suggests that domain-specific reinforcement learning can encode useful intuitions in model weights and improve runtime efficiency, even if models could otherwise learn the domain on the fly.

“Yeah, I'd say in practice, it does seem like model providers are... going domain by domain and trying to strengthen the models in the highest value domain.”

Claim: SPEAKER_03 observes that model providers appear to improve models one domain at a time, focusing on the highest-value domains.

“And I'd say that that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high value domains and the most common types of skills.”

Claim: SPEAKER_03 attributes part of recent model improvement to providers covering many high-value domains and common skill types.

“I mean, I think another thing is just that like it's not that expensive to do both at the same time, right? Because like the models are massive, they can easily afford in terms of their parameters to like learn everything.”

Claim: SPEAKER_01 argues that large models can afford to learn both domain-specific material and other capabilities because their parameter capacity is substantial.

“we think Yeah like there is likely some transfer and sort of even just even if like finance is not specifically like the information is important for like RSI just the general like meta learning of like how to figure out what's important how to have taste how to like do long horizon work is potentially generalizable and like there's not that much RSI like data in the world as well like it's kind of hard to generate and like that requires a lot of effort so like if you can sort of advertise in this other data get some transfer from it you already have masses of computing masses of parameter space so like why not do that as well as like obviously”

Claim: SPEAKER_01 says there is little recursive self-improvement training data available, and generating it is difficult and effort-intensive.

“we think Yeah like there is likely some transfer and sort of even just even if like finance is not specifically like the information is important for like RSI just the general like meta learning of like how to figure out what's important how to have taste how to like do long horizon work is potentially generalizable and like there's not that much RSI like data in the world as well like it's kind of hard to generate and like that requires a lot of effort so like if you can sort of advertise in this other data get some transfer from it you already have masses of computing masses of parameter space so like why not do that as well as like obviously”

Claim: SPEAKER_01 suggests that training on finance may transfer to recursive self-improvement by teaching general meta-learning skills, including identifying what matters, developing taste, and handling long-horizon work.

“I mean, there's one question about whether this current paradigm of doing like sim to real will be the dominant one forever. So basically, you look at what the real world tasks are like, and then you try to create a bunch of environments that can be simulated, like in the data center and you can do RL on them.”

Claim: SPEAKER_03 defines the current sim-to-real approach as modeling real-world tasks in simulated environments and training on those environments with reinforcement learning in data centers.

“I think obviously this has been very successful, but it has a lot of weaknesses because a lot of things are just kind of hard to simulate,”

Claim: SPEAKER_03 says sim-to-real training has been very successful, but is limited because many real-world situations are difficult to simulate.

“especially if they involve like interacting with a bunch of humans in real time.”

Claim: SPEAKER_03 says simulating tasks that require real-time interaction with many people is particularly difficult.

“I think some tool has to be the dominant framework where sample efficiency is kind of low. it right now you need like you know thousands and thousands of interactions with the humans and no human is going to sit there and like deal with this basically be in the loop of rl training and”

Claim: SPEAKER_01 argues that a dominant training framework is needed when sample efficiency is low, noting that current methods require thousands of human interactions.

“it right now you need like you know thousands and thousands of interactions with the humans and no human is going to sit there and like deal with this basically be in the loop of rl training and”

Claim: SPEAKER_01 says people will not manually participate in the thousands of interactions currently needed for reinforcement-learning training.

“so like we kind of have to simulate that now to like get the samples you need but like obviously if sample efficiency improves a lot you'd expect learning from deployment to like become like a much bigger part of it this”

Claim: SPEAKER_01 predicts that substantially improved sample efficiency would make learning from deployment a much larger component of model training.

“like at a very basic level, this is already happening, just in the next generation of models so like right now you can obviously take your deployment data and put this in the pre-trained or the mid-trained of like future models especially if you do like some kind of filtering or some kind of like judgment or annotation or like recent you know synthesis of that i”

Claim: At a basic level, model developers already use deployment data—sometimes filtered, judged, annotated, or synthesized—in pretraining or mid-training for future model generations.

“think it explains like quite a bit i mean especially like i mean this is you know i don't know whether the labs do this because you know theoretically they claim not to train on people's data but like the chinese 100 do and like they definitely get this advantage”

Claim: The speaker believes deployment-data use explains quite a bit of generation-over-generation model improvement, is unsure whether labs do this despite their theoretical claims not to train on people's data, and asserts that Chinese companies definitely do it and gain an advantage.

“advantage both like obviously deploying this is basically what distillation is like they take out the models they get some of their like deployment data they get some fraction of that by like pinging the model and then they train their next generation of models on it and they can suddenly do it on their own models as well like there's no reason not to whatsoever”

Claim: The speaker describes distillation as collecting some deployment data by querying a model and training a next-generation model on it, and says the same approach can also be used with one's own models.

“I completely agree I think if you zoom out far enough this is like definitely happening like you're picturing this like and we're all picturing this is like what continual learning like the holy grail is is like this very very organic like live loop of like an individual model like or like getting an experience and like live updating on the spot and learning from that and like a lot of things break when you like zoom into that level of granularity but like yeah the big labs are doing this like the closed the closed models are doing this there's also early signs of life of like people using open source models doing this at a much faster cadence so”

Claim: The speaker says large labs and closed-model developers already perform a slower-cadence form of continual learning, rather than necessarily updating an individual model live after every experience.

“or like getting an experience and like live updating on the spot and learning from that and like a lot of things break when you like zoom into that level of granularity but like yeah the big labs are doing this like the closed the closed models are doing this there's also early signs of life of like people using open source models doing this at a much faster cadence so”

Claim: The speaker says there are early signs that users of open-source models are applying deployment feedback at a much faster cadence.

“like a good example is probably like composer like you have some sort of model and you are able to or like you know Harvey's doing the same thing with like legal legal agents like it is getting very specific environments from the data that you have for that particular task and things that like you know users are complaining about and like all the feedback that you're somehow extracting from like your specific deployments and a lot of these companies have the advantage of the big labs and that they can use this”

Claim: The speaker cites Composer and Harvey's legal agents as examples of systems that use task-specific deployment data, user complaints, and extracted feedback.

“and then they will create environments they will like you know do a big post train of kimi k3 um they will go deploy it they might do some online learning as well like composer did online basically like reinforce for a long time um so yeah like there's still a human in the loop there's still a human saying okay these are the signals we care about here's how we're going to create environments from the data that we have and there's like still a longer cadence than maybe the one that you're thinking of but like it really is happening and like eventually that loop will become like faster and faster”

Claim: The speaker says deployment-feedback training loops are already occurring at a relatively long cadence and expects them to become progressively faster over time.

“and then they will create environments they will like you know do a big post train of kimi k3 um they will go deploy it they might do some online learning as well like composer did online basically like reinforce for a long time um so yeah like there's still a human in the loop there's still a human saying okay these are the signals we care about here's how we're going to create environments from the data that we have and there's like still a longer cadence than maybe the one that you're thinking of but like it really is happening and like eventually that loop will become like faster and faster”

Claim: The speaker says companies can use deployment data to create training environments, post-train and deploy models, and sometimes perform online learning, while humans still select the relevant signals and design the environments.

“they it was it's hard because when you do online reinforcement learning you don't have groups right you just have one user saying one thing and then you get one roller roll out and so like you have a big variance reduction problem”

Claim: The speaker says online reinforcement learning faces a substantial variance-reduction problem when feedback consists of a single user's signal and a single rollout rather than groups of examples.

“Yeah, I think your biggest problem is actually just not knowing what the reward function should be from natural data.”

Claim: A major challenge is determining the appropriate reward function from natural data.

“And if you use some kind of superficial signal, like did they accept the code, the edit, that might get reward hacked in some way.”

Claim: A superficial reward signal, such as whether a user accepts a code edit, might be reward-hacked.

“The longer and longer horizon tasks get, the harder they are to simulate within a data center,”

Claim: Longer-horizon tasks become harder to simulate within a data center.

“even in coding, we're getting to the point where there's like not some year-long coding task that doesn't eventually require you to like talk to a client or interact with the company or interact with the user. And if you think about the gamut of things we would want AI to be capable at,”

Claim: The speaker thinks that even year-long coding tasks may eventually require interaction with a client, company, or user.

“eventually super intelligence should be able to like run a business or like start a new business and make it profitable or like have a profitable day trading in the markets or win a court case and these are all things which are very hard to simulate in a data center like an inherent part of the learning there is interacting with the real world and so maybe maybe they may not learn how to get better at these things from like the transfer between sim to real but alternatively maybe you do need weight updates from these kinds of interactions”

Claim: The speaker is uncertain whether simulation-to-reality transfer will teach models to improve at complex real-world tasks, and suggests that weight updates from real-world interactions may instead be necessary.

“eventually super intelligence should be able to like run a business or like start a new business and make it profitable or like have a profitable day trading in the markets or win a court case and these are all things which are very hard to simulate in a data center like an inherent part of the learning there is interacting with the real world and so maybe maybe they may not learn how to get better at these things from like the transfer between sim to real but alternatively maybe you do need weight updates from these kinds of interactions”

Claim: The speaker expects eventual superintelligence to be capable of running or starting a profitable business, day-trading profitably, and winning a court case.

“eventually super intelligence should be able to like run a business or like start a new business and make it profitable or like have a profitable day trading in the markets or win a court case and these are all things which are very hard to simulate in a data center like an inherent part of the learning there is interacting with the real world and so maybe maybe they may not learn how to get better at these things from like the transfer between sim to real but alternatively maybe you do need weight updates from these kinds of interactions”

Claim: Running a business, profitable day trading, and winning a court case are difficult to simulate in a data center because learning these tasks inherently involves real-world interaction.

“in order to get better at them and then if that is the case if transfer isn't strong enough and you do need weight updates then the fact that the models are quite sample inefficient is like maybe a deeper problem and the reason i'm curious about this”

Claim: If simulation-to-reality transfer is insufficient and models need weight updates from real-world interactions, their low sample efficiency may pose a deeper problem.

“I feel like by default, I don't see how you don't get some kind of crazy recursive stuff from prevent within the next 10 years.”

Claim: The speaker expects some kind of dramatic recursive process to occur within the next 10 years, absent a reason it would not.

“But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans,”

Claim: A possible obstacle to recursive AI progress is that model weight updates appear much less sample-efficient than human learning.

“Like plausibly a million fold behind humans in terms of how much data humans sees from birth to adulthood versus how much a model sees from, you know, like cold start to like finishing training.”

Claim: The speaker estimates that humans may encounter roughly one million times as much data from birth to adulthood as a model encounters from cold start through training.

“running and so yeah this is all to say first of all is there is there going to be good transfer between simulations and extremely long horizon really complicated real shit that we want the ai to do in the real world and if not does that really mean that like the lack of sample efficiency in these models comes to bite us”

Claim: The speaker is uncertain whether simulation-to-reality transfer will work well for extremely long-horizon, complex real-world tasks, and whether models' low sample efficiency will become consequential if it does not.

“I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is like cumulative or like you kind of have to you have this like non-stationary distribution you have to keep learning and like re-litigating a bunch of stuff so like maybe an example of a cumulative task might be RSI like it's theoretically possible to maybe like have less than a million token like you know Python file which like from scratch trains a model that is capable of recursive self-improvement.”

Claim: The speaker says it is theoretically possible for a Python file under one million tokens to train a model capable of recursive self-improvement from scratch.

“I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is like cumulative or like you kind of have to you have this like non-stationary distribution you have to keep learning and like re-litigating a bunch of stuff so like maybe an example of a cumulative task might be RSI like it's theoretically possible to maybe like have less than a million token like you know Python file which like from scratch trains a model that is capable of recursive self-improvement.”

Claim: The speaker proposes that models may excel at cumulative tasks but continue to struggle with tasks involving non-stationary distributions that require ongoing learning and repeated reconsideration.

“improvement and like every discovery that you make is kind of a line in the sand that you hold like if it's true that you know for rsi we don't need to discover a new attention variant or whatever like once you've discovered attention and then once you've discovered you know mixture of experts once you discover glpo you just add that to the training stack and like that's that's there and like a good example of this is like you know 5.6 soul training 5.6 terror or whichever one open ai told us to train like it didn't have to go back and discover attention like”

Claim: The speaker suggests that recursive self-improvement could be cumulative if discoveries such as attention and mixture-of-experts can be retained in the training stack without needing new architectural breakthroughs.

“it basically probably would have called a bunch of like scripts which is like pre-training.sh and post-training.sh and just did that so like that's an example of like a cumulative task i think the real world and the reason like people are thinking so much about like continual learning is it's it's not really a cumulative task like imagine like in a law firm you have an agent acting as a legal associate like that's a very non-station distribution you have to be able to fit in your context like all the relationships between all the important all the important people that come in here which are also changing all the”

Claim: A legal-associate agent in a law firm must handle a non-stationary environment, including changing relationships among important people.

“time you have like all these like implicit like ways about how things are done where to find information etc and like that's not as clean of an example of a cumulative task as like RSI is so I think that there will be this breakdown between tasks but you know like if the labs realize that and they do believe that RSI is cumulative in the sense that like we don't need to go back and discover some brand new like architect or whatever then maybe more and more effort and compute gets focused on that versus the”

Claim: A legal-associate agent must adapt to implicit, changing workplace practices and information sources.

“time you have like all these like implicit like ways about how things are done where to find information etc and like that's not as clean of an example of a cumulative task as like RSI is so I think that there will be this breakdown between tasks but you know like if the labs realize that and they do believe that RSI is cumulative in the sense that like we don't need to go back and discover some brand new like architect or whatever then maybe more and more effort and compute gets focused on that versus the”

Claim: The speaker expects different task types to diverge in difficulty; if AI labs view recursive self-improvement as cumulative and not dependent on new architectures, they may allocate increasing effort and compute to it.

“Yeah, I would say there's like models are today's models are weaker than humans in a lot of different ways, and some of them might have to do with sample efficiency in a certain regime where I mean, in some regimes models are very sample efficient, like learning in context,”

Claim: AI models can be highly sample-efficient in some regimes, including in-context learning.

“Yeah, I would say there's like models are today's models are weaker than humans in a lot of different ways, and some of them might have to do with sample efficiency in a certain regime where I mean, in some regimes models are very sample efficient,”

Claim: Current AI models are weaker than humans across many dimensions.

“but then there might be some like medium length regime where they're less sample efficient because humans can. do some kind of weight update more efficiently than models.”

Claim: In some medium-length learning regimes, humans may update their weights more sample-efficiently than models.

“So I think like being less sample efficient in certain regimes might be one of the sources of weakness, but then I think there are other sources of weaknesses that are completely different than that. For example, having lower diversity of thought than humans or being bad at certain kinds of long horizon judgments.”

Claim: Models may be weaker than humans because they have less diversity of thought and perform poorly on certain long-horizon judgments.

“So I think like being less sample efficient in certain regimes might be one of the sources of weakness, but then I think there are other sources of weaknesses that are completely different than that. For example, having lower diversity of thought than humans or being bad at certain kinds of long horizon judgments.”

Claim: Low sample efficiency in some regimes may contribute to model weaknesses, alongside other distinct causes.

“I mean, I think a lot of what people call taste is something about behavior that works in the long run and that people have realized works in the long run, not everything, but like some aspect of taste,”

Claim: The speaker thinks much, but not all, of what people call taste consists of behavior recognized to work over the long term.

“like especially for something like software engineering, like I think a lot of taste is like what are the systems that are going to be maintainable.”

Claim: In software engineering, a substantial part of taste is judging which systems will remain maintainable.

“like how meta learnable is taste from like shorter horizon episodes and like i feel like there's no obvious reason it's super long because like humans somehow develop taste with not having many long episodes like we don't live to be like 10 000 we have like you know we we develop pretty quickly right and so like you know if you think about like even like in a phd the difference between like a first year phd student and like a final like postdoc or something that's like five years maybe and they've only done like maybe like 10 15 30 research projects in total but somehow they develop develop taste quite quickly from like a relatively short succession of like small things and so like theoretically it's you know possible to develop it like that the ai obviously will have vastly more experience in which to develop taste like metal on it and there's like how well does that generalize to like really long horizon things is i think the question which i think is really unsolved at this point like we don't know”

Claim: The speaker thinks taste may be learnable from relatively short-horizon episodes, noting that humans develop taste through a relatively short succession of small experiences and that there is no obvious reason it must require very long episodes.

“know we we develop pretty quickly right and so like you know if you think about like even like in a phd the difference between like a first year phd student and like a final like postdoc or something that's like five years maybe and they've only done like maybe like 10 15 30 research projects in total but somehow they develop”

Claim: The speaker says that a researcher can develop substantially between the first year of a PhD and the postdoc stage over roughly five years and perhaps 10, 15, or 30 research projects.

“develop taste quite quickly from like a relatively short succession of like small things and so like theoretically it's you know possible to develop it like that the ai obviously will have vastly more experience in which to develop taste like metal on it and there's like how well does that generalize to like really long horizon things is i think the question which i think is really unsolved at this point like we don't know”

Claim: The speaker says it remains unresolved whether taste learned from shorter-horizon experiences generalizes to very long-horizon tasks.

“develop taste quite quickly from like a relatively short succession of like small things and so like theoretically it's you know possible to develop it like that the ai obviously will have vastly more experience in which to develop taste like metal on it and there's like how well does that generalize to like really long horizon things is i think the question which i think is really unsolved at this point like we don't know”

Claim: The speaker thinks AI could theoretically develop taste from a relatively short succession of small experiences and would have vastly more experience available for developing taste.

“I mean, a big part of that is actually about incentives rather than being a technical question.”

Claim: The speaker says that whether AI systems learn from all their deployment experience is substantially an incentives question, not merely a technical question.

“So like companies aren't going to want to have the model provider learn from all of their deployment because that might just reduce the advantage of their business.”

Claim: The speaker says companies may resist model providers learning from all of their deployment data because that could reduce the companies' business advantage.

“the economics of this will pressure not necessarily weight updates to one big like common shared model but like kind of like modules that get subbed in so a very obvious example this is a law but it might be something else like you know there's been a lot of work to try and fit like an arbitrary context length into a fixed size like this is all the linear attention stuff and all that sort of stuff and like cartridges which are essentially kv caches trained to be very very compressed kv caches to fit in a lot of information”

Claim: The speaker describes cartridges as highly compressed KV caches designed to fit a large amount of information into a limited space.

“the economics of this will pressure not necessarily weight updates to one big like common shared model but like kind of like modules that get subbed in so a very obvious example this is a law but it might be something else like you know there's been a lot of work to try and fit like an arbitrary context length into a fixed size like this is all the linear attention stuff and all that sort of stuff and like cartridges which are essentially kv caches trained to be very very compressed kv caches to fit in a lot of information”

Claim: The speaker thinks economic pressures may favor adding specialized modules, such as law-specific modules, over updating one large shared model with deployment data.

“that's another example of like you know something that like companies may be willing to sign up for if that gets subbed into the model and it's not like actually changing the base underlying model itself so like there's many different versions of like learning from from your data in real time and like the latter ones are not really helping the big labs because they are just these modules but I think the like the like the economic pressure will force like the labs to go down that path first before they can embark on this like you know we're”

Claim: The speaker says companies may accept real-time learning modules based on their data when those modules can be added without changing the underlying base model.

“that's another example of like you know something that like companies may be willing to sign up for if that gets subbed into the model and it's not like actually changing the base underlying model itself so like there's many different versions of like learning from from your data in real time and like the latter ones are not really helping the big labs because they are just these modules but I think the like the like the economic pressure will force like the labs to go down that path first before they can embark on this like you know we're”

Claim: The speaker expects economic pressure to push AI labs toward modular learning approaches before directly training one large model on all deployment data.

“so it may be a more indirect form of learning that the big labs are getting and it's that's obviously still really valuable to them but I can't imagine a world in which we start off with like you know we're going to just like directly train this one big model on like all the exact data that we're training”

Claim: The speaker considers it unlikely that AI development will begin by directly training one large model on all collected deployment data.

“i think it will definitely like go through stages because because i mean this is assuming there's like one discontinuous event where suddenly we fix like like weight updates continuously and like in practice i think it's much more likely to be like the cartridges and stuff allow you specialized in deployment then you generate traces you put that in your model like three months later you come up with a model which is better at this stuff you specialize it again you like consolidate it again and then eventually we'll just like make this loop faster and faster so instead of like every three months we release a model now it's like every week and then every like day and then every hour at which point we basically have always solved it”

Claim: The speaker thinks AI learning from deployment experience will more likely develop through stages than through one discontinuous event that suddenly enables continuous weight updates.

“i think it will definitely like go through stages because because i mean this is assuming there's like one discontinuous event where suddenly we fix like”

Claim: The speaker expects continual model improvement to proceed in stages rather than through a single sudden discontinuous event.

“like weight updates continuously and like in practice i think it's much more likely to be like the cartridges and stuff allow you specialized in deployment then you generate traces you put that in your model like three months later you come up with a model which is better at this stuff you specialize it again you like consolidate it again and then eventually we'll just like make this loop faster and faster so instead of like every three months we release a model now it's like every week and then every like day and then every hour at which point we basically have always solved it”

Claim: The speaker expects iterative model improvement to move from roughly quarterly updates to weekly, daily, and hourly updates, using deployment traces to train later models, until the process effectively solves the task continuously.

“like weight updates continuously and like in practice i think it's much more likely to be like the cartridges and stuff allow you specialized in deployment then you generate traces you put that in your model like three months later you come up with a model which is better at this stuff you specialize it again you like consolidate it again and then eventually we'll just like make this loop faster and faster so instead of like every three months we release a model now it's like every week and then every like day and then every hour at which point we basically have always solved it”

Claim: The speaker expects AI deployment learning to progress from specialized cartridges generating traces that are incorporated into a later model, followed by further specialization and consolidation.

“like weight updates continuously and like in practice i think it's much more likely to be like the cartridges and stuff allow you specialized in deployment then you generate traces you put that in your model like three months later you come up with a model which is better at this stuff you specialize it again you like consolidate it again and then eventually we'll just like make this loop faster and faster so instead of like every three months we release a model now it's like every week and then every like day and then every hour at which point we basically have always solved it”

Claim: The speaker predicts that the cycle of AI model improvement and release could accelerate from every three months to weekly, daily, and eventually hourly.

“paradigm we often be able to do this i think we've we've done a bit of research on this when people have done a lot of research like at a large scale like when you wash out enough noise and you have large enough batches like this outer loop process of like putting data into mid training creating our own environments like it does work in like some sort of continual learning regime but the problem is like when you zoom in close enough at like a micro level it's like i've got one model and i'm trying to update it again for like a law firm or something um and i'm trying to do that very continuously like with a relatively”

Claim: The speaker says that large-scale mid-training with sufficiently large batches can support continual learning through an outer loop that adds data and creates environments.

“so there's more around the data like all the methods kind of break down a bit so like if I SFT the model on just like you know successful traces off policy on policy like eventually in the very iterative regime like when you're doing like you know hundreds of these micro updates you see both catastrophic forgetting you see forgetting of like previous information I've learned on top of the base model that was much earlier on and I see degradation of general like use.”

Claim: The speaker says that repeated narrow updates, including SFT on successful traces, can eventually cause catastrophic forgetting and degrade a model's previously learned information and general capabilities.

“just general capabilities um you know just on policy distillation seems to like push this horizon out a little bit but it still eventually succumbs to the same thing um and rl is not very good at like it is good at like getting capabilities in but it's not as good as getting like knowledge in and like just this very explicit knowledge of like okay like this person does this at this law firm and like this is a very specific process we find and you have to put in a lot of compute to create the right environments to get that”

Claim: The speaker says that acquiring specific knowledge through reinforcement learning requires substantial compute to create suitable environments.

“just general capabilities um you know just on policy distillation seems to like push this horizon out a little bit but it still eventually succumbs to the same thing um and rl is not very good at like it is good at like getting capabilities in but it's not as good as getting like knowledge in and like just this very explicit knowledge of like okay like this person does this at this law firm and like this is a very specific process we find and you have to put in a lot of compute to create the right environments to get that”

Claim: The speaker says reinforcement learning is better at adding capabilities than at adding explicit, specific knowledge such as a particular employee's law-firm workflow.

“just general capabilities um you know just on policy distillation seems to like push this horizon out a little bit but it still eventually succumbs to the same thing um and rl is not very good at like it is good at like getting capabilities in but it's not as good as getting like knowledge in and like just this very explicit knowledge of like okay like this person does this at this law firm and like this is a very specific process we find and you have to put in a lot of compute to create the right environments to get that”

Claim: The speaker says on-policy distillation can delay, but does not prevent, the eventual problems caused by repeated model updates.

“A little bit of both. I think like SFT and even like just like on policy distillation can be like way too destructive.”

Claim: The speaker says supervised fine-tuning and on-policy distillation can damage a model excessively.

“Like the reason RL is so nice is because like, yeah, it changes a very, very small amount about the model and there's like a lot of evidence for why this is the case.”

Claim: The speaker says reinforcement learning is attractive because it makes only very small changes to a model.

“space and so like it kind of just like tweaks it in this very very very small like loss value to like get it into the right point but that also then limits what you can do with RL like how much you can actually change the model without”

Claim: The speaker says reinforcement learning's limited effect on model parameters also constrains how much it can change a model.

“like it's easy to distill it into like a different base model like this is where i think it's mostly technique it's not like like it's definitely not like just like there isn't capacity like if you had some modeling with all this data and you take like literally the same size model and pre-train it from scratch with like all of this stuff in mid training it will be better and i think that's a lot of what's happening today and so it's very much like there's you know a bottleneck that stops us from just keeping training the same model forever versus just like getting all the data from the old model and like training a new model from scratch and this is exactly as jolly saying like some combination like plasticity and like catastrophic forgetting and that like that you know if you just naively”

Claim: The speaker argues that continual-learning limitations are mainly methodological rather than due to insufficient model capacity, since a same-size model pretrained from scratch with the accumulated mid-training data would perform better.

“like it's definitely not like just like there isn't capacity like if you had some modeling with all this data and you take like literally the same size model and pre-train it from scratch with like all of this stuff in mid training it will be better and i think that's a lot of what's happening today and so it's very much like there's you know a bottleneck that stops us from just keeping training the same model forever versus just like getting all the data from the old model and like training a new model from scratch and this is exactly as jolly saying like some combination like plasticity and like catastrophic forgetting and that like that you know if you just naively train on like non-stationary data because you're adding new data as you go basically this is messing with the data distribution so like the old stuff is just forgotten and”

Claim: The speaker says that training on a changing, non-stationary data distribution can cause a model to forget older information.

“really and have good methods to like stop that from happening yes”

Claim: The speaker says effective methods for preventing forgetting during continual learning are lacking.

“which of course is like very expensive like training model from scratch is expensive i”

Claim: The speaker says training a model from scratch is expensive.

“mean not necessarily i mean like maybe eventually if you have continuous learning you never train any model you just like just have a model and it keeps learning and like expanding right”

Claim: The speaker suggests that continuous learning might eventually let one model keep learning and expanding, eliminating the need to train separate new models.

“the question yeah i think we have pushback like how much from scratch we need to do like it is definitely possible now to take like the pre-trained base and like do very good mid training on top of that like kind of continuously plus some rl from like different checkpoints that are later on in the training and like that's looking more like a genuine but certainly not the case of like you know take the most recent model apply a couple of very small updates and like iteratively like you like never lose it but”

Claim: The speaker says continuous mid-training and checkpoint-based RL do not mean that repeatedly making small updates to the latest model will prevent information loss.

“the question yeah i think we have pushback like how much from scratch we need to do like it is definitely possible now to take like the pre-trained base and like do very good mid training on top of that like kind of continuously plus some rl from like different checkpoints that are later on in the training and like that's looking more like a genuine but certainly not the case of like you know take the most recent model apply a couple of very small updates and like iteratively like”

Claim: The speaker says current methods can perform strong, ongoing mid-training on a pretrained base and add reinforcement learning from later training checkpoints.

“it's still at a large enough scale i think you're washing out a lot of like the noise and like you're not just focused on one distribution which as brown said is like you know that is now a very if you're just like focusing on one task right like that's now very specific”

Claim: The speaker says large-scale training can wash out much of the noise, while concentrating on a single task makes the training distribution highly specific.

“And I mean, in the eventual regime, you'd be doing, I don't know, there's billions of deployed instances. you're learning from all of them at once. And so hopefully there's some. washing out of noise and stuff from that,”

Claim: In a hypothetical eventual regime with billions of deployed instances, learning from all instances at once might help wash out noise.

“yeah i mean i think like definitely as i was saying like you can do continual mid-training for like a long time and you can like roll back to a checkpoint give a new mid-training data but at the same time like you can't do this like indefinitely like if you just keep continuing mid-training the same base forever it'll just like get it does it sort of asymptotes at some point like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you just keep mid-training the same base forever”

Claim: A model can undergo continual mid-training for a long time, with training rolled back to a checkpoint to introduce new mid-training data.

“yeah i mean i think like definitely as i was saying like you can do continual mid-training for like a long time and you can like roll back to a checkpoint give a new mid-training data but at the same time like you can't do this like indefinitely like if you just keep continuing mid-training the same base forever it'll just like get it does it sort of asymptotes at some point like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you just keep mid-training the same base forever”

Claim: People train new base models because continual mid-training of one base model eventually stops enabling further learning.

“yeah i mean i think like definitely as i was saying like you can do continual mid-training for like a long time and you can like roll back to a checkpoint give a new mid-training data but at the same time like you can't do this like indefinitely like if you just keep continuing mid-training the same base forever it'll just like get it does it sort of asymptotes at some point like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you just keep mid-training the same base forever”

Claim: Continually mid-training the same base model indefinitely eventually reaches an asymptote, limiting its ability to learn new information.

“i think the existence of this is obvious it's just like whether we can create the right environment to get”

Claim: The speaker considers it obvious that some data distribution could train current architectures into superintelligence, while the open question is whether the necessary environment can be created.

“in the trivial case we could just train it to output the Python file which like trains the actual superintelligence like just have that memorized in the weights.”

Claim: In a trivial hypothetical case, a model could be trained to output a Python file that trains a superintelligence by memorizing the file in its weights.

“like yes there's probably like a ladder of rl environments that is possible to construct such that you would get a ai researcher which is at least as good as a human researcher but the effort to climb”

Claim: The speaker thinks it is probable that a sequence of reinforcement-learning environments could produce an AI researcher at least as capable as a human researcher.

“each successive wrong grows like kind of exponentially um and that's going to be the two things that you have to trade off against as to whether like like how fast we're going to hit like that final wrong where it's where it's better um i i think that's fairly clear and i think there's like you know we're still relatively early in like our environment creation like there's a lot of asymmetries that we exploit in order to create good environments so one of those asymmetries which we've talked about before is like it's”

Claim: The speed at which AI reaches a level better than humans depends on the tradeoff between the effort of advancing through the environment ladder and the relevant bottlenecks.

“each successive wrong grows like kind of exponentially um and that's going to be the two things that you have to trade off against as to whether like like how fast we're going to hit like that final wrong where it's where it's better um i i think that's fairly clear and i think there's like you know we're still relatively early in like our environment creation like there's a lot of asymmetries that we exploit in order to create good environments so one of those asymmetries which we've talked about before is like it's”

Claim: The effort required to advance through each successive rung of the proposed reinforcement-learning environment ladder grows roughly exponentially.

“each successive wrong grows like kind of exponentially um and that's going to be the two things that you have to trade off against as to whether like like how fast we're going to hit like that final wrong where it's where it's better um i i think that's fairly clear and i think there's like you know we're still relatively early in like our environment creation like there's a lot of asymmetries that we exploit in order to create good environments so one of those asymmetries which we've talked about before is like it's”

Claim: The speaker says AI environment creation remains relatively early and that researchers exploit asymmetries to build useful environments.

“there's environments where it's easier to go backwards and forwards and like what i mean by that is like it's very easy to define this like complex data generating process and this is like this kind of latent variable you keep hidden from the model um you can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generation process was um there's asymmetries in terms of like you know you can inject information from the real world so like anthropic finds a bug you”

Claim: Some training environments exploit an asymmetry in which a complex data-generating process is easy to construct or run backward from, while a model must spend substantial work inferring the hidden process from its outputs.

“there's environments where it's easier to go backwards and forwards and like what i mean by that is like it's very easy to define this like complex data generating process and this is like this kind of latent variable you keep hidden from the model um you can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generation process was um there's asymmetries in terms of like you know you can inject information from the real world so like anthropic finds a bug you bug um through like you know tens of thousands of human and llms combined and like turn that into a very very neat environment which in a single lm could theoretically find within like you know a few million tokens so like there's all these asymmetries which we're cherry picking and like we're counting on like kind of this task horizon generalization um but i think yeah again it's just going to hit diminishing returns at some point like at some point it's diminishing returns and how hard it is to create those environments in the first place like”

Claim: The speaker gives an example in which Anthropic found a bug using tens of thousands of humans and language models, then converted it into an environment that a single language model could theoretically discover within a few million tokens.

“there's environments where it's easier to go backwards and forwards and like what i mean by that is like it's very easy to define this like complex data generating process and this is like this kind of latent variable you keep hidden from the model um you can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generation process was um there's asymmetries in terms of like you know you can inject information from the real world so like anthropic finds a bug you”

Claim: A training environment can be generated from a complex process hidden as a latent variable, requiring the model to spend irreducible tokens and work to infer the process.

“there's environments where it's easier to go backwards and forwards and like what i mean by that is like it's very easy to define this like complex data generating process and this is like this kind of latent variable you keep hidden from the model um you can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generation process was um there's asymmetries in terms of like you know you can inject information from the real world so like anthropic finds a bug you”

Claim: Researchers can use information from the real world when constructing training environments.

“bug um through like you know tens of thousands of human and llms combined and like turn that into a very very neat environment which in a single lm could theoretically find within like you know a few million tokens so like there's all these asymmetries which we're cherry picking and like we're counting on like kind of this task horizon generalization um but i think yeah again it's just going to hit diminishing returns at some point like at some point it's diminishing returns and how hard it is to create those environments in the first place like”

Claim: The speaker expects returns from creating training environments to diminish eventually.

“bug um through like you know tens of thousands of human and llms combined and like turn that into a very very neat environment which in a single lm could theoretically find within like you know a few million tokens so like there's all these asymmetries which we're cherry picking and like we're counting on like kind of this task horizon generalization um but i think yeah again it's just going to hit diminishing returns at some point like at some point it's diminishing returns and how hard it is to create those environments in the first place like”

Claim: Researchers are selecting favorable asymmetries in training environments and relying on models to generalize across task horizons.

“coming up with them because you can't necessarily just like have these really like these processes where it's easier to go backwards than forwards like you actually have to sit down and construct like something that looks like with humans like you know a long enough time horizon like it's going to be a really complex task to create and then it's also going to be like the compute and time bottlenecks for the agent to actually do those tasks so like I think you're just going to start seeing this like curve to flatten out”

Claim: Building environments that resemble sufficiently long human time horizons is complex, while agents also face compute and time bottlenecks in carrying out the tasks.

“coming up with them because you can't necessarily just like have these really like these processes where it's easier to go backwards than forwards like you actually have to sit down and construct like something that looks like with humans like you know a long enough time horizon like it's going to be a really complex task to create and then it's also going to be like the compute and time bottlenecks for the agent to actually do those tasks so like I think you're just going to start seeing this like curve to flatten out”

Claim: The speaker expects progress from training environments to begin flattening out.

“model which is only trained on data up to 1930 on this like modern coding agent data and and it did better than Claude 3 Opus on on SweepBench so so this model that has like no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and like behave better as a coding agent than this much larger pre-trained model”

Claim: The speaker reports that a model trained on data only through 1930, after fine-tuning on modern coding-agent data, outperformed Claude 3 Opus on SweepBench.

“It's pretty crazy and it kind of shows you that like once you have an example of like the right expert behavior, it's actually surprisingly easy to like copy that into a like relatively weak model.”

Claim: The speaker believes that providing an example of expert behavior can make it surprisingly easy to transfer that behavior into a relatively weak model.

“kind of is there was a paper recently where they trained it up to like fifth grade maths and like also like primary school like English and stuff so it was like a decent language model and they tried to RL it to do like you know late high school and college maths and the gap was just too large like they couldn't get it to climb at all like but if you did like successive rungs of like you know your seventh maths and then your eighth maths and so on like you could obviously climb to year 12 so like again it's just like what is the distance between the rungs on those ladders and how hard is it”

Claim: In a cited paper, a model trained through fifth-grade mathematics and primary-school English could not be reinforced-trained directly to perform late-high-school and college mathematics because the capability gap was too large.

“and like also like primary school like English and stuff so it was like a decent language model and they tried to RL it to do like you know late high school and college maths and the gap was just too large like they couldn't get it to climb at all like but if you did like successive rungs of like you know your seventh maths and then your eighth maths and so on like you could obviously climb to year 12 so like again it's just like what is the distance between the rungs on those ladders and how hard is it what is it to create yeah”

Claim: The difficulty of capability progression depends on the distance between successive rungs and the effort required to create those rungs.

“and like also like primary school like English and stuff so it was like a decent language model and they tried to RL it to do like you know late high school and college maths and the gap was just too large like they couldn't get it to climb at all like but if you did like successive rungs of like you know your seventh maths and then your eighth maths and so on like you could obviously climb to year 12 so like again it's just like what is the distance between the rungs on those ladders and how hard is it”

Claim: The speaker says a model unable to jump directly from elementary to advanced mathematics could reach year-12 mathematics by progressing through successive intermediate levels.

“yeah this just comes back to like the RL signal problem like RL is not very good at like exploring right now and so like if you if the model can't like get in like you know 128 rollout it's very unlikely to get signal to like progress and this is why like in RL we need like curricular whereas like in pre-training we don't because like it's that's not a problem for pre-training at all”

Claim: The speaker says current reinforcement learning explores poorly, making a model very unlikely to receive a progress-enabling signal if it cannot find a useful result within 128 rollouts.

“yeah this just comes back to like the RL signal problem like RL is not very good at like exploring right now and so like if you if the model can't like get in like you know 128 rollout it's very unlikely to get signal to like progress and this is why like in RL we need like curricular whereas like in pre-training we don't because like it's that's not a problem for pre-training at all”

Claim: The speaker says reinforcement learning needs curricula, unlike pre-training, because the exploration problem is not an issue for pre-training.

“and and again pre-training data is different to post-training data and i imagine as we continue on like yeah humans will be involved less and less but that's”

Claim: The speaker expects human involvement in AI training to decrease over time as training continues.

“that doesn't change the fact that you're bottlenecked on like how much signal you can extract from the real world so like there's a lot of signal in the world and that's true like you know there's people doing like spreadsheet tasks there's people doing like legal tasks and all this sort of stuff but you know the capability frontier of where the models are at now like how many bits in the world are actually like really relevant to like”

Claim: The speaker says that reducing human involvement does not eliminate the bottleneck imposed by how much useful learning signal can be extracted from the real world.

“improving the model's capabilities like you know how many new maths problems are being solved like couldn't just beyond the reach or grasp of the current models like how many new coding problems are being created or solved that are beyond the reach of the current models and like I think that's why the diminishing returns kicks in because like even the world as a whole is not giving you the bits that are useful for tipping you into the next like basin of capability.”

Claim: The speaker thinks diminishing returns occur because the real world does not provide enough useful information to move current models into a new capability basin.

“like the signal doesn't you know in pre-training the signal is like already in common core right like for the tasks that you care about in pre-training the problem is there's not just like getting signal at all it's like filtering out all the noise that exists”

Claim: The speaker says that for tasks of interest in pre-training, useful signal is already present in Common Core, and the challenge is filtering noise rather than obtaining signal.

“and that's quite an automatable process but like as the models get better as we enter mid-training and post-training the signal just like doesn't exist anywhere in the original data we have like no amount of filtering will like get this you know there's no like hidden proof of like the millennium prize problem sitting in common core we can just like”

Claim: The speaker says that as models improve into mid-training and post-training, the required learning signal is absent from the original data and cannot be recovered by filtering alone.

“and that's quite an automatable process but like as the models get better as we enter mid-training and post-training the signal just like doesn't exist anywhere in the original data we have like no amount of filtering will like get this you know there's no like hidden proof of like the millennium prize problem sitting in common core we can just like”

Claim: The speaker says that filtering data to extract useful pre-training signal is fairly automatable.

“just like filter until we see it right and so like at that point you have to get bits some other way either from humans like directly like asking them to like write out their reasoning or like by like creating environments where like humans decide like what environment should be created what the objectives of these environments are or like you know some kind of like training on like the human data that exists in deployment like you have to get the bits from somewhere”

Claim: The speaker says that when filtering cannot supply the needed training signal, it must be obtained elsewhere, for example from human-written reasoning, human-designed environments and objectives, or human data from deployment.

“i think my my prior is that like again the low-hanging fruit is like somewhat exhausted with like like we got the internet as this big block and like there's it's not like the internet is necessarily like growing at the same like rate and all the useful stuff on the internet is growing at the same rate so like we've probably got like a bunch of like 0.1 percent loss drops to go but like not definitely not as many as have currently occurred”

Claim: The speaker's prior is that much of the easily obtained improvement from internet data has already been exhausted.

“i think my my prior is that like again the low-hanging fruit is like somewhat exhausted with like like we got the internet as this big block and like there's it's not like the internet is necessarily like growing at the same like rate and all the useful stuff on the internet is growing at the same rate so like we've probably got like a bunch of like 0.1 percent loss drops to go but like not definitely not as many as have currently occurred”

Claim: The speaker says the internet as a whole and the useful content available on it are not necessarily growing at the same rate.

“i think my my prior is that like again the low-hanging fruit is like somewhat exhausted with like like we got the internet as this big block and like there's it's not like the internet is necessarily like growing at the same like rate and all the useful stuff on the internet is growing at the same rate so like we've probably got like a bunch of like 0.1 percent loss drops to go but like not definitely not as many as have currently occurred”

Claim: The speaker thinks further 0.1% loss reductions are probably still possible, but expects fewer such improvements than have already occurred.

“I don't know, I feel like data is actually more important with scale.”

Claim: The speaker's view is that data becomes more important relative to other factors as model scale increases.

“I feel like architectures are kind of like a one-time like you know.”

Claim: The speaker characterizes architectural improvements as largely one-time gains.

“architect i say like combining like saying just like an x percent efficiency gain is kind of misleading because like what an architect does is like letting you reach like a qualitatively new regime which you couldn't reach with the old architecture and then within that regime obviously the data is like the primary thing determining it but like you know if we say it didn't have like even like gqa we're doing like full attention all day we wouldn't be able to do like a million like ridiculous expenses do a million context and then like because of that we couldn't we can never use the data which is like actually at a million context”

Claim: Architecture changes can enable qualitatively new operating regimes that previous architectures could not reach, so describing their benefit only as a percentage efficiency gain can be misleading.

“architect i say like combining like saying just like an x percent efficiency gain is kind of misleading because like what an architect does is like letting you reach like a qualitatively new regime which you couldn't reach with the old architecture and then within that regime obviously the data is like the primary thing determining it but like you know if we say it didn't have like even like gqa we're doing like full attention all day we wouldn't be able to do like a million like ridiculous expenses do a million context and then like because of that we couldn't we can never use the data which is like actually at a million context”

Claim: Without GQA, full attention would make million-token contexts prohibitively expensive and prevent models from using data at that context length.

“so we couldn't get these capabilities even though like if you just do a naive like how much does this do at like 2k context where the architecture isn't unlocking anything then like the data you know that will look much more important than in some sense it is right it's unclear to me that these things are like really just like multiplicative gains in this way”

Claim: Comparing models at a 2,000-token context where the architecture enables no new capability can make data appear more important than it is.

“so we couldn't get these capabilities even though like if you just do a naive like how much does this do at like 2k context where the architecture isn't unlocking anything then like the data you know that will look much more important than in some sense it is right it's unclear to me that these things are like really just like multiplicative gains in this way”

Claim: The speaker is uncertain whether architecture and data improvements combine as simple multiplicative gains.

“so i mean on scale dependence i think like a lot of the like mid-training and post-training data we have now is like actually but gets better with scale because like a lot of it like the very long context horizon environment stuff really requires like big models to be able to like make use of them and like this is not you know if you try and train like your 100 million parameter model on like three bench traces it's not going to get anywhere like it's not going to show you the same kind of improvement that you would get if you train like an actual”

Claim: Training a 100-million-parameter model on three benchmark traces would not yield the same improvement as training a substantially larger model.

“so i mean on scale dependence i think like a lot of the like mid-training and post-training data we have now is like actually but gets better with scale because like a lot of it like the very long context horizon environment stuff really requires like big models to be able to like make use of them and like this is not you know if you try and train like your 100 million parameter model on like three bench traces it's not going to get anywhere like it's not going to show you the same kind of improvement that you would get if you train like an actual”

Claim: Much current mid-training and post-training data becomes more useful with model scale because long-context and long-horizon environment tasks require large models to exploit it.

“yeah i think for the next few years we're going to be like like because we're so focused on doing longer and longer horizon rollouts for rl where like inference efficiency matters a lot um”

Claim: Over the next few years, longer-horizon reinforcement-learning rollouts will make inference efficiency especially important.

“it feels like the the malls aren't necessarily saturated on the ability that where the bottleneck is still the environments and so we might see like a little bit of plateau like i i have a feeling that you know like mythos and and the gpt models are much smaller than like you know the 10 trillion parameter range that”

Claim: Model capability is not necessarily saturated; the current bottleneck is still the availability or difficulty of environments.

“it feels like the the malls aren't necessarily saturated on the ability that where the bottleneck is still the environments and so we might see like a little bit of plateau like i i have a feeling that you know like mythos and and the gpt models are much smaller than like you know the 10 trillion parameter range that that people are talking about um even just naively comparing the open source models you can probably back out that conclusion um”

Claim: The speaker suspects that Mythos and GPT models are far smaller than the 10-trillion-parameter scale sometimes discussed.

“so yeah probably for the next few years i wouldn't imagine a huge growth in the number of parameters but again like there's so many different different things to trade off here like you decide the size of your model based on like how much pre-training data you have and then like the difficulty of the RL environments that you've got to train on and you ideally want to like get to the optimal point you”

Claim: The speaker probably does not expect frontier model parameter counts to grow dramatically over the next few years.

“so yeah probably for the next few years i wouldn't imagine a huge growth in the number of parameters but again like there's so many different different things to trade off here like you decide the size of your model based on like how much pre-training data you have and then like the difficulty of the RL environments that you've got to train on and you ideally want to like get to the optimal point you”

Claim: Choosing model size depends on how much pre-training data is available and how difficult the reinforcement-learning training environments are.

“point where you know you can get like a decent pass at one or something on like the the hardest environments you have and like it wouldn't make sense to like make a bigger bigger model pass there because then you're just paying like much more inference That. what than you need to um so there's there's a lot of impetus like depends on how quickly you know like macore and then in-house these these guys can scale up the complexity the oral environments they're training on”

Claim: A model should ideally be large enough to achieve a decent pass rate on the hardest training environments, but making it larger than necessary would waste inference compute.

“point where you know you can get like a decent pass at one or something on like the the hardest environments you have and like it wouldn't make sense to like make a bigger bigger model pass there because then you're just paying like much more inference That. what than you need to um so there's there's a lot of impetus like depends on how quickly you know like macore and then in-house these these guys can scale up the complexity the oral environments they're training on”

Claim: How quickly model sizes grow depends partly on how quickly organizations can scale the complexity of their training environments.

“I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger.”

Claim: Models will continue to grow in size as available compute scales up and GPUs become larger.

“bigger but i would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways so one thing is that i think like data efficiency is going to be a bigger driver than compute efficiency of like the exact architectures people use now that we're getting to the regime where we're sort of running low on like high quality pre-training data so that might affect how sparse you want to make the model”

Claim: The extent of model growth depends in non-obvious ways on scaling laws, and data efficiency may matter more than compute efficiency as high-quality pre-training data becomes scarce.

“And then I also think we don't understand sparsity that well.”

Claim: Current understanding of sparsity in models is limited.

“And it's like parameters are a different resource than active parameters,”

Claim: Total parameters and active parameters represent distinct resources in a model.

“but it's and like sparsity has definitely increased a bit, but it's not clear that it's going to keep increasing without bound.”

Claim: Model sparsity has increased somewhat, but it is uncertain whether it will keep increasing without limit.

“There might be some kind of sweet spot.”

Claim: There may be an optimal level of model sparsity rather than sparsity increasing indefinitely.

“There's an argument that sparsity should make data efficiency worse because you might have to learn the same thing on. and multiple experts. So that's debatable.”

Claim: One debated argument is that sparsity may reduce data efficiency because multiple experts may need to learn the same information.

“So I think we don't, I don't think we have a good enough theory of scaling laws that we really understand why sparsity is helping and to what, how much it'll help and if that'll like plateau at some point at a certain level of sparsity.”

Claim: Current scaling-law theory is insufficient to explain why sparsity helps, the magnitude of its benefit, or whether that benefit plateaus at a particular sparsity level.

“So you have all your choices you can make on the architecture and each of these gives you a different scaling law.”

Claim: Each architectural choice can yield a different scaling law.

“And then like traditionally you would look at some kind of envelope based on compute. So you would look at performance versus compute.”

Claim: Traditional compute-based scaling analysis examines an envelope of model performance as a function of compute.

“mean this could well yeah this depends on like the ratio of like training and inference compute really it's like if your super bowl neck on data not on compute you should go bigger if a super bowl neck on compute you should always go smaller and then like yeah”

Claim: The speaker says model size should increase when data is the bottleneck, but decrease when compute is the bottleneck.

“you can also use computer-generated synthetic data, so it's like one of these very hard things to predict.”

Claim: Computer-generated synthetic data is another possible source of training data.

“you can also use computer-generated synthetic data, so it's like one of these very hard things to predict.”

Claim: The speaker considers the implications of synthetic data and data-versus-compute constraints very difficult to predict.

“I think part of the reason it took people so long to figure out the scaling laws in the first place was that if you don't get all these things right, then you don't get such a clean relationship.”

Claim: The speaker says scaling laws took a long time to discover partly because failing to account for all relevant factors obscures the clean relationship.

“And like the beautiful straight lines on graphs like hide a lot of complexity on how you have to make sure like to scale every hyperparameter the right way or like parameterize your optimizer in a way that scales and where you don't have to change your hyperparameters as you change the model.”

Claim: The speaker says apparently simple linear scaling-law graphs conceal the complexity of scaling hyperparameters appropriately and designing optimizer parameterizations that remain valid across model sizes.

“and bugs have their own clean scaling laws as well right like you know like with Kaplan for getting the cosine annealing thing or like even just like not considering embedding parameters I think and so that messed up the estimated smaller models because embedding parameters are a decent size of the model”

Claim: The speaker says that issues including cosine annealing and omitting embedding parameters distorted estimates for smaller models, because embedding parameters constitute a substantial part of a model.

“and bugs have their own clean scaling laws as well right like you know like with Kaplan for getting the cosine annealing thing or like even just like not considering embedding parameters I think and so that messed up the estimated smaller models because embedding parameters are a decent size of the model”

Claim: The speaker says bugs also exhibit clean scaling laws.

“So I feel like a year ago, a lot of people were making this argument that RL will not be super successful at scaling for models.”

Claim: The speaker says that about a year earlier, many people argued reinforcement learning would not scale successfully for models.

“So I feel like a year ago, a lot of people were making this argument that RL will not be super successful at scaling for models.”

Claim: About a year before the conversation, many people argued that reinforcement learning would not scale successfully for models.

“And I wrote some blog posts earlier this year where I was like, it's even worse than that because when the pass rate is low and the models are very unlikely to get... Got to get the answer right. It learns almost nothing at all from an RL episode.”

Claim: When a model has a low pass rate and is unlikely to answer correctly, it learns almost nothing from a reinforcement-learning episode.

“And I wrote some blog posts earlier this year where I was like, it's even worse than that because when the pass rate is low and the models are very unlikely to get... Got to get the answer right. It learns almost nothing at all from an RL episode.”

Claim: The speaker previously argued that low-pass-rate models learn almost nothing from an individual reinforcement-learning episode when they are very unlikely to answer correctly.

“But I look at the models today and they seem pretty smart and it seems to be the result of scaling up RL.”

Claim: The speaker believes current models seem quite capable and that their capability appears to result from scaling up reinforcement learning.

“But I look at the models today and they seem pretty smart and it seems to be the result of scaling up RL.”

Claim: The speaker thinks current models seem quite capable and that scaling up reinforcement learning appears to have contributed to their capability.

“I mean so I think the success of RL comes down to a bunch of different things so first I think what is slightly underestimated is actually the mid training so an awful lot of like what we see as successes of RL actually comes from like very very good mid training data which is basically where we're like essentially doing pre-training but on like synthetic reasoning data and like the kind of environments that like get the model warm started for RL and so this actually takes the model like almost like 80% of the way to like the final RL checkpoint often.”

Claim: Much of the apparent success of reinforcement learning comes from high-quality mid-training data, such as synthetic reasoning data and environments that prepare models for reinforcement learning.

“I mean so I think the success of RL comes down to a bunch of different things so first I think what is slightly underestimated is actually the mid training so an awful lot of like what we see as successes of RL actually comes from like very very good mid training data which is basically where we're like essentially doing pre-training but on like synthetic reasoning data and like the kind of environments that like get the model warm started for RL and so this actually takes the model like almost like 80% of the way to like the final RL checkpoint often.”

Claim: Mid-training often brings a model nearly 80% of the way to its final reinforcement-learning checkpoint.

“often and then what rl does on top of that is it does like a lot of you know essentially tweaking to the policy and so this is one of the reasons why it doesn't need like as many bits as you would naively think it doesn't have to learn all of these behaviors from scratch it needs just like a few bits from these episodes which you do get and then the other thing that i really point out in my blog is that these bits are actually extremely high signal compared to like regular like pre-training which is why you need rl at all versus just like sfting on like successful reasoning traces”

Claim: Reinforcement-learning feedback provides higher-signal information than ordinary pre-training, making RL useful beyond simply supervised fine-tuning on successful reasoning traces.

“often and then what rl does on top of that is it does like a lot of you know essentially tweaking to the policy and so this is one of the reasons why it doesn't need like as many bits as you would naively think it doesn't have to learn all of these behaviors from scratch it needs just like a few bits from these episodes which you do get and then the other thing that i really point out in my blog is that these bits are actually extremely high signal compared to like regular like pre-training which is why you need rl at all versus just like sfting on like successful reasoning traces”

Claim: After mid-training, reinforcement learning mainly makes policy adjustments instead of learning behaviors from scratch, which means it needs fewer bits than might be naively expected.

“well there's two things so yes one it's exactly the bits about how to get the answer right but like this is not exactly how you think of it because in sot you have a trace right you have like a bunch of math reasoning and then the answer at the end the bit is still there like you're still sot on the answer token so that bit is still there what's important is that the object ignores all the other bits so in sft you like have like you know to try and match like the exact reasoning tokens that the model produces so you're essentially getting like too many bits about like the exact way this other model you're training on reasons for rl you only get the one bit and that means that like this that signal is not drowned out in the noise of like all the other bits the model has and so that's what really like it's really super dramatic like increase into the signal to noise ratio during training which is why like rl is like”

Claim: Supervised fine-tuning includes the correctness information in the answer token while also training the model to match the exact reasoning tokens in a trace.

“ignores all the other bits so in sft you like have like you know to try and match like the exact reasoning tokens that the model produces so you're essentially getting like too many bits about like the exact way this other model you're training on reasons for rl you only get the one bit and that means that like this that signal is not drowned out in the noise of like all the other bits the model has and so that's what really like it's really super dramatic like increase into the signal to noise ratio during training which is why like rl is like It's like so dramatically efficient in terms of steps.”

Claim: Because reinforcement learning supplies a correctness signal without requiring the model to match all reasoning tokens, its training signal has a much higher signal-to-noise ratio.

“It's like so dramatically efficient in terms of steps.”

Claim: Reinforcement learning can be dramatically efficient in the number of training steps required.

“If you have a large enough amount of compute to sample a large enough group size such that your probability of getting a bunch of correct answers is past some not insignificant probability, then it will be upweighted. And to Vernon's point,”

Claim: If compute allows sampling a large enough group that multiple correct answers occur with non-negligible probability, reinforcement learning will upweight those correct answers.

“basically mid-training and more pre-training, the pass at one, the starting point for RL scales in the log number of pre-training tokens.”

Claim: The pass-at-one starting point for reinforcement learning scales logarithmically with the number of pre-training tokens.

“So like one thing I want to point out here is that like it doesn't necessarily imply that RRL has a small like effect, Even if you have a few bits and like you only change the parameters a small amount like the actual impact on like function space, the model ends like the input to output map. for mapping can still be like super dramatic you know even if it's like even like one bit can change like your function space a lot and it can like rule out like half the hypothesis space which is huge so like i don't think it's necessarily the case it's like small amounts of bits small amounts of rl once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily”

Claim: Even a small amount of reinforcement-learning feedback and a small parameter change can dramatically alter a model's input-output function.

“can still be like super dramatic you know even if it's like even like one bit can change like your function space a lot and it can like rule out like half the hypothesis space which is huge so like i don't think it's necessarily the case it's like small amounts of bits small amounts of rl once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily”

Claim: A single bit of information can substantially change a model's function space by ruling out half of the hypothesis space.

“can still be like super dramatic you know even if it's like even like one bit can change like your function space a lot and it can like rule out like half the hypothesis space which is huge so like i don't think it's necessarily the case it's like small amounts of bits small amounts of rl once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily”

Claim: Starting from a strong model, a small amount of reinforcement learning does not necessarily produce only a small behavioral effect.

“I think it comes down to two things. I think the first thing is that everyone was hoping that RL would like generalize this reasoning across like all these different domains. and I don't think we necessarily got this like horizontal generalization like just training on math doesn't necessarily make you the greatest coder like you do have to do RL on on code environments I think what we did get though is like horizon generalization like the models just learned how to use more tokens for for longer and still make progress on some sort of task and so like you can train on environments where they get longer and longer and longer and then put them into a completely new environment and yes like”

Claim: Reinforcement learning has not necessarily yielded broad cross-domain reasoning transfer: math training alone does not necessarily make a model a strong coder, so code environments require code-specific RL.

“and I don't think we necessarily got this like horizontal generalization like just training on math doesn't necessarily make you the greatest coder like you do have to do RL on on code environments I think what we did get though is like horizon generalization like the models just learned how to use more tokens for for longer and still make progress on some sort of task and so like you can train on environments where they get longer and longer and longer and then put them into a completely new environment and yes like it's like they may not have generalized the reasoning patterns which allow them to do well in that environment but they've at least generalized the ability to like continue on that task longer which is correlated with like success i”

Claim: Training in environments with progressively longer tasks can transfer to a new environment as the ability to persist longer, even when the model's environment-specific reasoning patterns do not transfer.

“and I don't think we necessarily got this like horizontal generalization like just training on math doesn't necessarily make you the greatest coder like you do have to do RL on on code environments I think what we did get though is like horizon generalization like the models just learned how to use more tokens for for longer and still make progress on some sort of task and so like you can train on environments where they get longer and longer and longer and then put them into a completely new environment and yes like”

Claim: Reinforcement learning has enabled horizon generalization, in which models use more tokens over longer periods while continuing to make progress on tasks.

“it's like they may not have generalized the reasoning patterns which allow them to do well in that environment but they've at least generalized the ability to like continue on that task longer which is correlated with like success i”

Claim: A model's ability to persist on a task for longer is correlated with task success.

“there's a paper called edge bench which showed that the rate at which models can work for longer is like doubling every three months and so that's a clear evidence of generalization and i think like the final way to think about is like um in pre-training there's this idea of like”

Claim: The speaker says the EdgeBench paper found that models’ ability to work for longer durations roughly doubles every three months, and regards this as clear evidence of generalization.

“quanta so you have this very smooth like pre-training Moscow and when you actually look at what's happening in the model like the model is learning all these like very discrete like tasks and there's like always like emergent points where there's like kind of a phase transition like it didn't have induction heads now it has induction heads and there's like tens of thousands millions probably like hundreds of millions of these things and you average them all together and you get this very like smooth loss curve I think like to us to an extent like a similar thing is happening happening for RL”

Claim: The speaker says pre-training produces a smooth-looking loss curve while models learn many discrete capabilities, some of which emerge through phase transitions, such as the appearance of induction heads.

“bro like there is this very slow outer loop as baron mentioned of you know we will train a model and then aurella and then like the next kind of model iteration of training we will dump a bunch of these synthetic reasoning traces into the mid training data like we're kind of hitting all these quantum for all these different tasks and like on an individual task level it may look like a phase transition and like you're suddenly going from like a 0.5 percent pass rate to a 90 percent pass rate on like a particular like finance task or excel task or whatever but your average average all these things together and plus the horizon generalization you kind of would go wow we've got like qualitatively better models”

Claim: The speaker thinks reinforcement learning may improve models through discrete, task-level capability jumps that appear smoother when averaged across tasks and combined with horizon generalization.

“bro like there is this very slow outer loop as baron mentioned of you know we will train a model and then aurella and then like the next kind of model iteration of training we will dump a bunch of these synthetic reasoning traces into the mid training data like we're kind of hitting all these quantum for all these different tasks and like on an individual task level it may look like a phase transition and like you're suddenly going from like a 0.5 percent pass rate to a 90 percent pass rate on like a particular like finance task or excel task or whatever but your average”

Claim: The speaker says that an individual task can show a capability jump in which a model’s pass rate rises from 0.5% to 90%.

“i mean i think a lot of this as well is just like i think rl does generalize a bit like suddenly you get like some transfer between like math and code or like puzzles and math and this kind of stuff also just like the amount the sheer amount of environments i think that people are targeting is just like vastly greater so like you know before when you try to do you know some task which like you do in your daily life like two years ago like the labs wouldn't really care about this they wouldn't like train train them on for it and now like it's just so much broader they have a lot of environments targeting this specific thing”

Claim: The speaker says AI labs now target a much broader range of environments and everyday tasks than they did two years earlier.

“i mean i think a lot of this as well is just like i think rl does generalize a bit like suddenly you get like some transfer between like math and code or like puzzles and math and this kind of stuff also just like the amount the sheer amount of environments i think that people are targeting is just like vastly greater so like you know before when you try to do you know some task which like you do in your daily life like two years ago like the labs wouldn't really care about this they wouldn't like train”

Claim: The speaker thinks reinforcement learning produces some transfer across domains, including between math and code and between puzzles and math.

“i mean so a couple of things here like first off i think that the alpha go is using mcts which obviously does like more exploration and like stuff than regular policy gradients but i kind of also think that like rl doesn't necessarily like reduce the creativity and like i mean even if we i think you know this is obviously qualitative but if we look at like the you know the open air hiking phase incident like these models were coming up with like multiple zero days at a time to like break out of the sandbox and like this is clearly like some level of like move 37 creativity”

Claim: The speaker says AlphaGo uses Monte Carlo tree search (MCTS), which explores more than regular policy gradients.

“like stuff than regular policy gradients but i kind of also think that like rl doesn't necessarily like reduce the creativity and like i mean even if we i think you know this is obviously qualitative but if we look at like the you know the open air hiking phase incident like these models were coming up with like multiple zero days at a time to like break out of the sandbox and like this is clearly like some level of like move 37 creativity creativity i think already which we just get from just like the general generalization properties of the lm's like i don't think it's definitely not the case that like rl is like totally destroying”

Claim: The speaker thinks language models’ generalization properties can produce creative behavior.

“like stuff than regular policy gradients but i kind of also think that like rl doesn't necessarily like reduce the creativity and like i mean even if we i think you know this is obviously qualitative but if we look at like the you know the open air hiking phase incident like these models were coming up with like multiple zero days at a time to like break out of the sandbox and like this is clearly like some level of like move 37 creativity creativity i think already which we just get from just like the general generalization properties of the lm's like i don't think it's definitely not the case that like rl is like totally destroying”

Claim: The speaker thinks reinforcement learning does not necessarily reduce creativity, citing language models that produced multiple zero-day exploits at once to escape a sandbox as an example of Move 37-like creativity.

“Yeah, I mean, one thing that people call creativity is just solving hard search problems.”

Claim: The speaker defines one form of creativity as solving hard search problems.

“So and so that's like like Move 37 is obviously an example of that or like writing some kind of poem that satisfies a ton of different constraints.”

Claim: The speaker gives AlphaGo’s Move 37 and writing a poem that satisfies many constraints as examples of creativity involving hard search.

“So that's something AI is obviously going to be extremely good at if trained for it.”

Claim: The speaker predicts that AI will be extremely good at solving hard search problems if trained for those tasks.

“uh there's another way in which the models uh like uh the diversity of their outputs uh is a lot lower after rl and they sort of develop these ticks and like um even though the models seem like they're good at writing when you do some kind of like distributional analysis you find that like they're reusing uh certain themes uh like all the time and they're using the same character names all the time”

Claim: The speaker says reinforcement learning reduces the diversity of model outputs, with models repeatedly using certain themes and character names and developing recurring stylistic ticks.

“um so there's actually um it's not like you're getting the same kind of diversity that you get when you like from human authors you're sort of getting one really good uh like style so i think that um like that kind of diversity has definitely been like cut down by rl a lot and in fact uh now oh yeah since we were talking about distillation earlier that's sort of something yeah one thing that's happening is that so many people are distilled distilling mostly from Claude that like like all the open weight models right the same way as Claude and use the same like have the same ticks so this seems kind of concerning to me that we're having this like this monoculture emerge”

Claim: The speaker says widespread distillation from Claude makes open-weight models resemble Claude and share its stylistic ticks, creating a concerning model monoculture.

“um so there's actually um it's not like you're getting the same kind of diversity that you get when you like from human authors you're sort of getting one really good uh like style so i think that um like that kind of diversity has definitely been like cut down by rl a lot and in fact uh now oh yeah since we were talking about distillation earlier that's sort of something yeah one thing that's happening is that so many people are distilled”

Claim: The speaker says reinforcement learning has substantially reduced stylistic diversity in model writing compared with human authors, leaving models with a narrower, highly developed style.

“again i don't think this is like fundamental to rl as like a method though and same with distillation like even with distillation like you're just training on the data it's like just because your data is not like super broad”

Claim: The speaker does not consider the discussed limitations fundamental to reinforcement learning or distillation as training methods.

“again i don't think this is like fundamental to rl as like a method though and same with distillation like even with distillation like you're just training on the data it's like just because your data is not like super broad”

Claim: The speaker argues that reduced diversity is not inherent to reinforcement learning or distillation, and that distillation’s results depend on how broad its training data is.

“but that doesn't mean like the training method itself is somehow wrong it's like a problem with the data and i think a lot of for instance like the rl like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments because like for instance like the right thing i think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to what hack the judge and that's why like it collapses but like this is really a problem with the judge it's not a problem with like rl in general”

Claim: The speaker considers verifier exploitation and resulting collapse a problem with the judge rather than with reinforcement learning in general.

“but that doesn't mean like the training method itself is somehow wrong it's like a problem with the data and i think a lot of for instance like the rl like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments because like for instance like the right thing i think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to what hack the judge and that's why like it collapses but like this is really a problem with the judge it's not a problem with like rl in general”

Claim: The speaker attributes reinforcement-learning entropy collapse largely to models exploiting simple verifiers in insufficiently diverse environments, such as writing models learning to hack their grading judge.

“but that doesn't mean like the training method itself is somehow wrong it's like a problem with the data and i think a lot of for instance like the rl like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments because like for instance like the right thing i think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to what hack the judge and that's why like it collapses but like this is really a problem with the judge it's not a problem with like rl in general”

Claim: The speaker attributes some reinforcement-learning entropy collapse to models exploiting simple verifiers in environments with insufficient diversity; for writing, models may learn a judge’s stylistic preferences, which the speaker sees as a judge problem rather than a flaw in reinforcement learning itself.

“yeah i mean i would say maybe like for the like full generality maybe like three years but i think”

Claim: The speaker estimates that models with the full generality of a remote worker may be available in about three years.

“to Charlie's point we will end up with like a lot of people like making their organizations easier for the AIs to use and so you get like 80 90 percent of the way there before that like”

Claim: The speaker expects organizations to adapt their systems for AI use, enabling AI to reach 80–90% of the described remote-worker capability before achieving full generality.

“I think there's going to be like a long tail of like miscellaneous stuff which like some human can do which like will take the models like quite a while to do”

Claim: The speaker expects models to take quite a while to handle the long tail of miscellaneous tasks that humans can perform.

“I mean I think this this really comes down to a question of like how quickly can we solve We solve this kind of like online learning and like whether we can like get like 80 90% of the way there with like compaction and like writing files to yourself and stuff and like that's my big uncertainty.”

Claim: The speaker says progress toward general remote-worker capability depends on solving online learning, and is uncertain whether compaction and self-written files can get AI 80–90% of the way there.

“that maybe, yeah, we'll have some version of this in a year or so that's like, okay, and it will be able to do, maybe we'll have that form factor and it'll be able to do some things really well, some things not so well, and things will be improving from there.”

Claim: The speaker predicts that a version of the described AI remote-worker capability will exist in about a year, perform some tasks very well and others less well, and improve thereafter.

“I think I would just refuse to give you a scaler on this.”

Claim: The speaker declines to estimate whether the proposed 10x total productivity uplift will occur.

“Like we might already be past that in some like types of work.”

Claim: The speaker thinks AI may already have achieved the proposed productivity uplift in some types of work.

“I would say like two years.”

Claim: The speaker estimates that AI researchers may achieve a 10x productivity uplift in about two years.

“I can kind of see that actually because like it really is just like right now it's already like definitely more than 10x of like coding stuff and so it's like if it can do even like one or two loops of like experimental feedback that would actually be massive already.”

Claim: AI already provides more than a 10x uplift on coding tasks; adding even one or two loops of experimental feedback would be a major advance.

“I can kind of see that actually because like it really is just like right now it's already like definitely more than 10x of like coding stuff and so it's like if it can do even like one or two loops of like experimental feedback that would actually be massive already.”

Claim: The speaker says AI already provides more than a 10x productivity improvement for coding, and that adding one or two loops of experimental feedback would be a major gain.

“if you just plug it into like a very naive model of like ai progress and how much is coming from ai researchers and they're like there's like a 10x increase in their productivity yeah you have like radically accelerated pace of ai progress starting two years from now of”

Claim: Under a naive model in which AI researchers become 10 times as productive, the pace of AI progress would accelerate radically starting two years later.

“Yeah, I mean I think like this will mean that AI progress doesn't get bottlenecked on like AI research's ability to run like small experiments it gets bottlenecked on other things so”

Claim: The speaker expects AI progress to stop being bottlenecked by AI researchers' ability to run small experiments and instead to be bottlenecked by other factors.

“Like my capacity to absorb information and make the like Bayesian optimal decision on the next experiment”

Claim: The speaker says their own capacity to absorb information and choose the Bayesian-optimal next experiment is the crux.

“yeah i mean i'm assuming that like you can delegate some of this to the ai so like the ai is becoming decent at like deciding you know it run this experiment it's got this result it runs like the next experiment”

Claim: The speaker assumes AI can take on some experimental work and is becoming reasonably capable of selecting a next experiment based on the previous experiment's result.

“and then if it can run like two or three experiments in a row without like crashing then like that is actually big update in like uplift”

Claim: The speaker says that AI successfully running two or three consecutive experiments without crashing would substantially increase their estimate of AI's uplift.

“In AI, which dominates top human experts across every single field of work that can be done over computer. So not only AI research, but all cognitive work. and not just like short horizon work but like literally if it takes like three years or something the AI will still do better than humans yeah”

Claim: The speaker predicts that AI will outperform top human experts in all computer-based fields, including tasks that take about three years to complete.

“I would say like three or four years.”

Claim: The speaker estimates a three-to-four-year timeline for AI to reach the level of capability being discussed.

“mean, I would say like AI is obviously being more, getting more attention. So it's like one of the harder things. But it's like a lot of energy is being put into it.”

Claim: The speaker says AI is receiving increasing attention and substantial effort is being devoted to it.

“And it's also like not one of the hardest things for AI because it's like involves a lot of code and math, which models are really good at. Maybe for things that involve like 3D and like spatial stuff and physical stuff,”

Claim: The speaker argues that AI research is not one of the hardest areas for AI to automate because it involves substantial coding and mathematics, areas where models are strong.

“which models are really good at. Maybe for things that involve like 3D and like spatial stuff and physical stuff, I think that that will take a little longer.”

Claim: The speaker expects AI to take longer to become capable in tasks involving 3D, spatial, or physical work.

“So especially if it's not like, yeah, if it's like mechanical engineering or something and it's not. Not getting like the most attention right now that might take a little longer”

Claim: The speaker suggests that AI may take longer to master fields such as mechanical engineering when those fields are not receiving much attention.

“but it also doesn't include fields where there is relatively little data because of the nature of the field and it has to like learn that data on the fly for example it has to become superhuman at like being an engineer at tsmc or something”

Claim: The speaker notes that AI may need to learn information on the fly to become superhuman in roles such as engineering at TSMC, where the field has relatively little available data.

“So you would have to assume that like the onboarding, yeah, you can give the AI the same onboarding material and then there's some like something has to be solved about like sort of longer horizon learning.”

Claim: The speaker says AI could receive the same onboarding materials as a human, but longer-horizon learning would still need to be addressed.

“i think so”

Claim: The speaker thinks that automating AI research is an AI-complete problem.

“um yeah i think there's so many things in the world which like even if you have some sort of memory system external to the model and even if like context length grows a little bit like there are just fundamentally things like even if you could research the information or write notes yourself like you'd need more than the memory context field to do”

Claim: The speaker argues that external memory and a modestly longer context window would still be insufficient for some tasks, which require more than the model's memory context.

“yeah i mean i kind of agree in like the the five-year range at least for like the stuff that like labs are focusing on but i think like there's going to be a long tail of stuff which like yeah i could theoretically go out and learn about but like no one has bothered to do it and like the computer isn't being allocated to that so that might take longer for like literally every single human expert”

Claim: The speaker broadly agrees that within about five years AI could surpass human experts in areas labs focus on, while less-studied areas may take longer because they have not been researched or allocated computing resources.

“I mean, I think that's not necessarily necessary, actually, because like the AI will have vastly greater experience than like any human.”

Claim: The speaker argues that AI need not learn a new domain as quickly as a human if it has vastly more experience than any individual human.