Beam: The Great American Open Model with ReflectionAI Co-Founder and CEO Misha Laskin

“When you remove cyber offensive capabilities, you also remove cyber defensive capabilities.”

Claim: Removing cyber-offensive capabilities also removes cyber-defensive capabilities.

“The state of the world today is that we have a few hundred safety researchers within closed labs that understand how these things work, and despite their best intentions, is impossible to cover the long tail of unintended consequences that these systems might have.”

Claim: A few hundred safety researchers at closed labs understand how these systems work, but they cannot cover the long tail of unintended consequences the systems might have, despite their best intentions.

“A very powerful closed model went and like hacked into another company, and the only way that company could remediated itself was by using open models to protect itself. That's the empirical evidence of the world that world that we're in.”

Claim: A very powerful closed model hacked into another company, and the company could remediate the incident only by using open models for protection.

“Linus's law: with enough eyeballs, all bugs become shallow. I have the belief that with enough eyeballs, most security and safety vulnerabilities become shallow as well.”

Claim: Misha Laskin believes that sufficient public scrutiny makes most security and safety vulnerabilities shallow.

“So for the last 12 months, we set the mission of the company to build frontier open intelligence and make it widely accessible.”

Claim: During the preceding 12 months, Reflection AI's mission was to build frontier open intelligence and make it widely accessible.

“And effectively, it's been a sprint to set up a lab that is capable of doing such things.”

Claim: Reflection AI has been rapidly establishing a lab capable of building frontier open intelligence.

“order of a hundred plus so you know a couple hundred researchers and engineers to build one of these things like a true rocket ship project and we're now at around 300 people and you know assembled all the teams on pre-training mid-training reinforcement learning scaled up our you know trained our first models end-to-end and just you know released a model called beam which is reflections first open model”

Claim: Reflection AI had approximately 300 employees at the time of the conversation.

“order of a hundred plus so you know a couple hundred researchers and engineers to build one of these things like a true rocket ship project and we're now at around 300 people and you know assembled all the teams on pre-training mid-training reinforcement learning scaled up our you know trained our first models end-to-end and just you know released a model called beam which is reflections first open model”

Claim: Building a frontier model requires roughly 100 or more researchers and engineers, potentially a couple hundred.

“order of a hundred plus so you know a couple hundred researchers and engineers to build one of these things like a true rocket ship project and we're now at around 300 people and you know assembled all the teams on pre-training mid-training reinforcement learning scaled up our you know trained our first models end-to-end and just you know released a model called beam which is reflections first open model”

Claim: Reflection AI assembled pre-training, mid-training, and reinforcement-learning teams, trained its first models end-to-end, and released Beam as its first open model.

“What's been interesting about this experience is that I got asked this questions at an all hands at the company, what's been the hardest thing in last year? And the reality is that everything is extremely hard. Everything has been hard.”

Claim: Laskin says that every aspect of building the company and its models has been extremely difficult.

“another or something like this and again the the answer is like everything right you you actually have to get 30 things right and that's why everything is hard because you have to get 30 things right you have to get the talent you have to retain the talent you have to have a strong mission and culture that keeps the talent you know working together and rowing in the same boat you have to get the data you have to get the compute you have to build the infrastructure uh to ensure that the computer's actually usable. Uh and these are a lot of these things when you're an ex existing big lab, there have been years of build out that enabled them to have this kind of stable surface, so I was a deep mind before this, and a lot of the tools that we just take for granted or I took for granted as a researcher because they just worked, you have to build. So I would say”

Claim: Laskin says model development depends on getting roughly 30 factors right, including talent recruitment and retention, a strong mission and culture, data, compute, and usable infrastructure.

“uh to ensure that the computer's actually usable. Uh and these are a lot of these things when you're an ex existing big lab, there have been years of build out that enabled them to have this kind of stable surface, so I was a deep mind before this, and a lot of the tools that we just take for granted or I took for granted as a researcher because they just worked, you have to build. So I would say”

Claim: Large established AI labs have spent years building the stable operational foundations that support their research.

“Yeah, everything has been hard, but also very rewarding that, you know, it's that you can do it.”

Claim: Laskin found the difficult work of building the company and its systems rewarding because it demonstrated that the task was achievable.

“When we started the company about two and a half years ago, our bet was so maybe I'll contextualize it where the world was then.”

Claim: At the company's founding roughly two and a half years earlier, Laskin and his co-founder bet that reinforcement learning could be applied beyond chat models.

“My co-founder Yanis and I had been working on the first series of Gemini models. We were working on the reinforcement learning team. Yanis was leading it.”

Claim: Before founding Reflection AI, Laskin and his co-founder worked on the first Gemini model series and its reinforcement-learning team, which the co-founder led.

“And we had just shipped the Gemini 1 and then 1.5 model.”

Claim: Before founding Reflection AI, Laskin and his co-founder shipped Gemini 1 and Gemini 1.5.

“And that was, you know, that early era that was like similar to the first chat GPT experience, which was primarily a chat experience. There was no coding really, nothing agentic. These models were just good at chat.”

Claim: In the early era Laskin describes, AI models were primarily chat systems, with little coding capability and no agentic behavior.

“and they were primarily you know 95% of the compute spent was spent on pre-training so they're primarily pre-trained models we are reinforcing learning researchers that's been our lineage”

Claim: In the period Laskin describes, approximately 95% of model compute went to pre-training, making the models primarily pre-trained.

“Janice my co-founder was one of the founding engineers at DeepMind and was a key contributor to all the big RL projects that came out of that lab including AlphaGo and”

Claim: Laskin's co-founder was a founding engineer at DeepMind and contributed to major DeepMind reinforcement-learning projects, including AlphaGo.

“Our bet was because we were working on the RL team and we saw that RL for aligning chat models works.”

Claim: Laskin and his co-founder observed that reinforcement learning can align chat models.

“There's only so much you can align them, right?”

Claim: Laskin believes the degree to which chat models can be aligned is limited.

“domains where like mathematics coding like you could that you could make these systems agentic that was the big bet and we thought that you could do this as an independent lab much more capital efficiently because we started seeing open source open open models materialize there was a small model from mistral at the time llama two had just released and we saw llama three coming”

Claim: Laskin and his co-founder believed an independent lab could develop agentic systems more capital-efficiently as open models emerged.

“yeah yeah this was a while back I guess two years ago and and we thought that there would be a great open model base that someone would build that that we could build on and do our research and scale reinforcement learning and over that first year of the company what ended up happening was that one”

Claim: Laskin and his co-founder expected an open-model foundation to emerge that they could use for research and scaling reinforcement learning.

“Reinforcement learning started working faster than we thought it would, both within our experiments, but also within industry,”

Claim: Reinforcement learning started producing results faster than Laskin expected, both in Reflection AI's experiments and in the broader industry.

“O1 came out that time.”

Claim: OpenAI's O1 model was released around the time reinforcement learning began working faster than Laskin expected.

“We actually thought it would take longer just because the journey from ImageNet in 2012 to RL systems that worked at scale was about four years. So we thought maybe something similar.”

Claim: The speakers expected the development of reinforcement-learning systems that worked at scale to take longer, using the roughly four-year transition from ImageNet in 2012 as a comparison.

“every year moves faster than than my prediction the previous year and so about a year into the company we got to the point where they're actually like all the good open models are coming from China there were not really good you know Western open models and we needed one as a base for kind of the technology that we were building and decided for a number of reasons that”

Claim: About a year after the company began, the speaker viewed the good open models as coming from China and said there were not really good Western open models.

“every year moves faster than than my prediction the previous year and so about a year into the company we got to the point where they're actually like all the good open models are coming from China there were not really good you know Western open models and we needed one as a base for kind of the technology that we were building and decided for a number of reasons that”

Claim: The speaker says that developments each year have been faster than he predicted the year before.

“every year moves faster than than my prediction the previous year and so about a year into the company we got to the point where they're actually like all the good open models are coming from China there were not really good you know Western open models and we needed one as a base for kind of the technology that we were building and decided for a number of reasons that”

Claim: The company needed an open model to serve as a foundation for its technology.

“The most impactful thing we could do, given the state of play, was to just build open models ourselves end-to-end and combine the RL bet together with just building models end-to-end.”

Claim: The speaker judged that the most impactful course, given the state of play, was for the company to build open models end-to-end and pair that effort with its reinforcement-learning strategy.

“There were, you know, I guess there were enterprise reasons, there were geopolitical reasons happening,”

Claim: The speaker says enterprise and geopolitical considerations were among the reasons for the company’s decision to build open models.

“You actually do need to pre-train your model in order to make reinforcement learning work very well at scale. So you actually kind of do need to do that. The things are just so tightly coupled that you do need to do both.”

Claim: The speaker says effective reinforcement learning at scale requires pre-training because pre-training and reinforcement learning are tightly coupled, so both are needed.

“So you do need a lot of resources to... to build something meaningful, you don't need the same amount of resources. Once you get to the frontier and you're really, really at the frontier of intelligence, then you need the same amount of resources as any other frontier lab because you can kind of think about research as exploration of new ideas versus execution of known ideas.”

Claim: The speaker says building a meaningful model requires substantial resources, but matching frontier-lab resource levels is necessary only once a lab reaches the intelligence frontier.

“then you need the same amount of resources as any other frontier lab because you can kind of think about research as exploration of new ideas versus execution of known ideas.”

Claim: The speaker frames research as involving two kinds of work: exploring new ideas and executing ideas that are already known.

“Part of why getting great talent in matters a lot is because you get to. get to cut down on exploration and then you can focus on execution of the things that that work”

Claim: The speaker says great talent reduces the amount of exploration needed and allows a team to focus on executing approaches that work.

“So all this is to say is that if you are catching up to the frontier, you can do it a lot more capital efficiently.”

Claim: A lab catching up to the AI frontier can do so more capital-efficiently than a lab pushing the frontier.

“I think that it depends if the frontier keeps moving. each year it's actually more capital.”

Claim: The capital required to catch up depends on whether the AI frontier continues moving, and the speaker says the required amount increases each year it does.

“a year ago or maybe 18 months ago it would have been and i say orders meaning that you know order of 100 million so hundreds of millions of dollars”

Claim: The speaker estimates that catching up to the frontier would have required hundreds of millions of dollars about a year or perhaps 18 months earlier.

“i think that now or let's say even six months ago is probably order billion so billions of dollars single digit billions uh going into next year i think order 10 you know you can kind of”

Claim: The speaker estimates that catching up to the frontier required probably single-digit billions of dollars now or six months ago, and could require around $10 billion going into next year.

“Think about it as for every generation of model there's a 4x multiplier in compute roughly and one heuristic that I have is like”

Claim: The speaker’s heuristic is that compute requirements increase by roughly four times for each model generation.

“let's say like we talk about like the chips that are currently considered, you know, the ones that, you know, a frontier chip and maybe a couple years ago it was an H100 and maybe it was 100,000 H100s.”

Claim: The speaker says that a couple of years earlier, an H100 was considered a frontier chip and a frontier-scale run might use 100,000 H100s.

“Then, you know, Astra was trained and that was just a big run, right, on 100,000 Blackwells, which is roughly a 4x multiplier on an H100.”

Claim: The speaker says Astra was trained using 100,000 Blackwell chips, which he describes as roughly four times an H100 in compute scale.

“Now the next generation is going to be around 100,000 beer Rubens, I believe.”

Claim: The speaker believes the next generation of training hardware will involve around 100,000 Rubin chips.

“Absolutely. I think that in terms of, you know, there's only so much much capex that you can do right and frontier labs are now putting hundreds of billions of dollars into it so will they be putting trillions of dollars you know in in the next year or two i you know it's probably you know unlikely but there are a couple of things that are happening one is that the use of that compute is becoming a lot more efficient because the”

Claim: The speaker says frontier labs are currently investing hundreds of billions of dollars in compute and considers it probably unlikely that they will invest trillions within the next year or two.

“know in in the next year or two i you know it's probably you know unlikely but there are a couple of things that are happening one is that the use of that compute is becoming a lot more efficient because the models are themselves becoming tools that help build themselves.”

Claim: The speaker says compute use is becoming more efficient as models themselves become tools for building models.

“So I think that when it was just human researchers doing the work, there's probably a 7x improvement each year.”

Claim: The speaker estimates that, when humans alone performed the research work, efficiency improved by probably about seven times each year.

“And in free training, you do it by, you have your pre-training loss, and then you measure your compute efficiency gains against, you made some changes, now you have a new pre-training loss, and it should be that you got to the same point with less.”

Claim: The speaker says pre-training compute-efficiency gains can be quantified by measuring whether a change reaches the same pre-training loss using less compute.

“with less compute and so when you say 7x it's actually a very concrete thing that you're studying and now I would say across the systems of you know I mean pre-training is probably a bit more hardened but there's a lot of a lot of headroom in reinforcement learning we're probably at a point where it's you know 30x like efficiency depending how good your model is for actually improving itself so that is one thing that's happening”

Claim: The speaker says pre-training is relatively mature, whereas reinforcement learning has substantial room for improvement and may offer roughly 30-fold efficiency gains depending on the model’s ability to improve itself.

“that yeah i mean it depends on you know what metric it is but it roughly i do think that the speed is probably four or more times faster than researchers just doing it alone and that'll probably accelerate so the amount of intelligence you can extract”

Claim: Depending on the metric, the speaker thinks AI-assisted research is probably at least four times faster than researchers working alone and will probably accelerate further.

“that yeah i mean it depends on you know what metric it is but it roughly i do think that the speed is probably four or more times faster than researchers just doing it alone and that'll probably accelerate so the amount of intelligence you can extract per training flop is is increasing so at some point that asymptote is okay and then also like the amount of revenue that each flop can generate also increases right the more you pack intelligence so going back to the question around how does this distribute across pre-training and reinforcement learning and to train beam which is a 500 billion parameter model total 23b active it was”

Claim: The speaker says the amount of intelligence obtained per training FLOP is increasing.

“per training flop is is increasing so at some point that asymptote is okay and then also like the amount of revenue that each flop can generate also increases right the more you pack intelligence so going back to the question around how does this distribute across pre-training and reinforcement learning and to train beam which is a 500 billion parameter model total 23b active it was”

Claim: The speaker says each FLOP can generate more revenue as systems contain more intelligence.

“per training flop is is increasing so at some point that asymptote is okay and then also like the amount of revenue that each flop can generate also increases right the more you pack intelligence so going back to the question around how does this distribute across pre-training and reinforcement learning and to train beam which is a 500 billion parameter model total 23b active it was”

Claim: Beam has 500 billion total parameters and 23 billion active parameters.

“6,000 GB300s.”

Claim: The speaker says Beam’s training used 6,000 GB300 chips.

“We ran it, I think, for a few weeks, but now with infrastructure efficiencies, we can do it in about 12 days, maybe less.”

Claim: The speaker says Beam’s training initially took a few weeks and can now be completed in about 12 days or possibly less because of infrastructure efficiencies.

“Reinforcement learning was a little over 10,000 GB300s for four weeks. So actually there were more flops spent on reinforcement learning.”

Claim: The speaker says reinforcement learning used a little over 10,000 GB300s for four weeks and consumed more flops than the other training discussed.

“Reinforcement learning jobs are You know, more complex in the sense that you do both a lot of inference at scale with all sorts of sandboxes for agents and training.”

Claim: Reinforcement-learning jobs are more complex because they involve large-scale inference, agent sandboxes, and training.

“This actually the reason I got into AI was I saw my co-founder work on AlphaGo and”

Claim: The speaker says he got into AI because he saw his co-founder work on AlphaGo.

“I was a physicist at the time, and there was a moment, like there was a plot in AlphaGo where it just never stopped improving, right? It just kept improving,”

Claim: The speaker recalls that an AlphaGo performance plot kept improving without stopping.

“and they just cut it off at some point because what's the point of improving it further? You already beat the world champion.”

Claim: The speaker says AlphaGo's improvement was eventually stopped because it had already beaten the world champion.

“And I thought that, well, if you figure out how to apply that recipe to stuff that is economically valuable, then at that point it becomes an economic question of how much money do I want to put in to get. And yet, just keep improving the system. And we as a field are there.”

Claim: The speaker thought that applying the AlphaGo approach to economically valuable tasks would make continued system improvement a question of how much money to invest for further gains.

“And one of the things that I'm really excited about in this open source project and the tech report that we'll have in it is that we'll describe how these things are built.”

Claim: The speaker is excited that the open-source project and its technical report will explain how the systems are built.

“this reinforcement learning system never stopped learning. If you look at our plots, they just keep going up.”

Claim: The speaker says the reinforcement-learning system continued learning and its plotted performance continued to rise.

“And it's just a matter of compute, basically, right, in terms of scaling it further.”

Claim: The speaker says further scaling of the system is basically limited to providing more compute.

“That is to say, we as a field are there where we have these RLL systems that are very general that don't really stop improving.”

Claim: The speaker says the field has very general reinforcement-learning systems that do not really stop improving.

“You want to make them a lot more efficient to extract the most intelligence from your compute.”

Claim: The speaker says reinforcement-learning systems should be made much more efficient to extract the most intelligence from available compute.

“But that means that you move from a question of can you train these things to where can you get the data and what is economically valuable and it's just an economics decision of how much money do I put into X to get some kind of improvement out of it.”

Claim: The speaker says the focus has shifted from whether AI systems can be trained to finding data, identifying economically valuable uses, and deciding how much to invest for an improvement.

“this was not true two years ago this”

Claim: The speaker says the described state of affairs was not true two years before the conversation.

“is almost to some kind of dissatisfaction to some scientists and that it's moved i mean it is a science but it's you know i would say feels more of an engineering discipline like it seems more like uh the rocket ship analogy and the way of building rockets is a science, but most people think about it as an engineering discipline with some scientific work in it, and that's roughly how it feels.”

Claim: The speaker says AI remains a science but has come to feel more like an engineering discipline, with scientific work within it.

“So the stages are kind of set in that the axes of scaling are today, and now there might be something kind of totally blue ocean that we don't know and there's some new labs exploring it.”

Claim: The speaker says current scaling axes are established, while an entirely new direction may exist and be explored by some labs.

“And there are architectural improvements that can be made.”

Claim: The speaker says AI architectures can be improved.

“There are data improvements,”

Claim: The speaker says AI training data can be improved.

“algorithmic improvements.”

Claim: The speaker says AI algorithms can be improved.

“there it's not like a it doesn't feel like a wild west the way i felt five years ago or something like when like the something that was kind of beautiful about both think deep mind and google brain and even if you look at early open ai's how diverse the bets were and people were doing just crazy fun stuff and now it's really narrowed in and and part of it there's like also a bit of a hardware lottery kind of thing because once something starts working the hardware also starts co-optimizing against it and so it's harder to find even if you have a great new idea if it's not really a good fit for the hardware you know it doesn't make sense so to answer your question”

Claim: The speaker says AI research has become less exploratory and its range of research bets has narrowed compared with roughly five years earlier, including earlier work at DeepMind, Google Brain, and OpenAI.

“people were doing just crazy fun stuff and now it's really narrowed in and and part of it there's like also a bit of a hardware lottery kind of thing because once something starts working the hardware also starts co-optimizing against it and so it's harder to find even if you have a great new idea if it's not really a good fit for the hardware you know it doesn't make sense so to answer your question”

Claim: The speaker says hardware co-optimizes around successful approaches, making new ideas harder to pursue when they are a poor fit for the hardware.

“I don't see there hasn't been an asymptote of how much how much juice you can get out of better pre-training like we're not seeing a slowdown in compute efficiency gains you keep seeing stuff there's a lot a lot of headroom and reinforcement learning I think that that one is probably there's a lot of stuff to be kind of discovered there but it's again it feels more like a engineering discovery than groundbreaking understanding of a new science or something like this.”

Claim: The speaker thinks reinforcement learning probably has substantial discoveries remaining, which feel more like engineering discoveries than breakthroughs in new science.

“I don't see there hasn't been an asymptote of how much how much juice you can get out of better pre-training like we're not seeing a slowdown in compute efficiency gains you keep seeing stuff there's a lot a lot of headroom and reinforcement learning I think that that one is probably there's a lot of stuff to be kind of discovered there but it's again it feels more like a engineering discovery than”

Claim: The speaker says there is still substantial headroom for better pre-training and that gains in compute efficiency have not slowed down.

“I think the code and agentic stuff kind of sets a foundation for the intelligence and the intelligence is ultimately”

Claim: Code and agentic work establish a foundation for intelligence.

“it's jagged in the sense that, one, it's generalized in a surprising way. I was surprised by how quickly our model got to a level of capability that is fairly fresh and recent, that other labs are discovering not so long ago.”

Claim: The speaker was surprised that ReflectionAI's model generalized in unexpected ways and quickly reached a capability level that other labs had discovered only recently.

“At the same time, it is somewhat jagged where it quickly adapts to data once you have the right data for, you know, tasks you want to do. But there is kind of a jaggedness about it. And it's even between like the various benchmarks like the various versions of terminal bench like you don't see clean generalization between different harnesses but once you have something working you have a generally gigantic capability and you get the data for a new harness it actually starts adapting to it pretty quickly so what that means is that you have this like pretty adaptive thing”

Claim: The speaker describes the model's generalization as uneven: it adapts quickly to a task when given suitable data, but does not generalize cleanly across different benchmark harnesses.

“that you need to and then it's a matter of like where are the economically valuable pools of data that's i mean part of what makes open models i think powerful is that well enterprises and customers can take it and customize it for their own stuff and get something that is kind of Pareto optimal for their workloads like the lowest cost for the highest performance”

Claim: Open models can be customized by enterprises and customers for their workloads to achieve a favorable balance of cost and performance.

“And so it's an empirical question. You go to customers, you actually see what is valuable to them, and you see whether it's possible to, and usually when it's economically valuable,”

Claim: The speaker says identifying economically valuable data pools is an empirical question that requires observing what customers value.

“you can generate data because it's not even that you're going to be training on the customer's data, it's more that you're setting up evaluations, and if you can set up a good evaluation for their tasks, then you can generate synthetic data that approximated and you get get good generalization.”

Claim: For customer tasks, the speaker says they generally do not train on customer data; instead, they create evaluations and synthetic data approximating the tasks, which can yield good generalization.

“So concretely, finance. Like various know your customer flows and compliance flows in finance are very valuable.”

Claim: The speaker considers finance know-your-customer and compliance workflows economically valuable.

“A lot of things, different things in cybersecurity, cyber defense in particular, are quite valuable.”

Claim: The speaker considers cybersecurity tasks, especially cyber defense, economically valuable.

“Legal, like there are all sorts of kind of like agentic verticals like within enterprise that are valuable and have a pretty similar pattern in terms of how you get something working there.”

Claim: The speaker says enterprise legal work includes valuable agentic verticals that follow a similar implementation pattern.

“Beam is a model that was trained for coding an agentic task. That's where it excels.”

Claim: Beam was trained for coding and agentic tasks, which are its areas of strength.

“And an important thing in model building is not just the capability, but how quickly an agent achieves a thing.”

Claim: Model building should consider both capability and the time an agent takes to complete a task.

“So it translates to faster workload times, cheaper costs for customers.”

Claim: When agents complete tasks faster, customers get shorter workload times and lower costs.

“So beam tends to be three to four times more efficient than models of the same capability class and much more efficient when it comes to, you know, there are models that are larger out there where, you know, where the efficiency gains are then end up being something like 10x.”

Claim: Beam is typically 3–4 times more efficient than models of similar capability, and can be about 10 times more efficient than larger models.

“And the reason it's so efficient is because we prioritized Well, both a strong pre-training base for reasoning, but then amplified it with reinforcement learning at what we believe to be the largest scale it's ever been done in open source.”

Claim: The speaker attributes Beam's efficiency to a strong reasoning pre-training base amplified by reinforcement learning at what they believe was the largest scale yet used in open source.

“I've not seen a 10,000 GB300 for four weeks run documented yet.”

Claim: The speaker says they have not seen a documented run using 10,000 GB300s for four weeks.

“The way you tend to set it up is that you want to extract the maximum amount of capability in the least amount of time.”

Claim: The speaker says reinforcement learning is typically configured to extract as much capability as possible in as little time as possible.

“The first AlphaGo agents were pretty, you know, meandering in the way that they were solving the problem. And then by the time you got it to Lee Sedol level, it was just, you know.”

Claim: The speaker contrasts early AlphaGo agents, which meandered in problem-solving, with the later Lee Sedol-level system, which searched intelligently.

“very smart in its kind of in its search right so that's really what enabled it to have that capability so you know the more reinforcement learning you run the higher the capability and the faster these these systems solve solve it the”

Claim: The speaker says that running more reinforcement learning increases system capability and makes systems solve tasks faster.

“intention I guess because we are kind of RL believers and so we always believe that that was the the path to AGI was you know through through reinforcement learning with actually a really strong base we forget that the first alpha go systems had they were trained on expert amateur human games so they did this imitation learning first and then reinforcement learning so you kind of need those things”

Claim: The speaker believes that achieving AGI requires reinforcement learning together with a strong underlying base.

“reinforcement learning with actually a really strong base we forget that the first alpha go systems had they were trained on expert amateur human games so they did this imitation learning first and then reinforcement learning so you kind of need those things”

Claim: The speaker says the first AlphaGo systems used imitation learning on expert and amateur human games before reinforcement learning.

“working together and a byproduct of it is that you get this really economically kind of valuable thing that it's very reasoning efficient and makes a really nice workhorse model for enterprises and public sector sovereign.”

Claim: The speaker says combining a strong base with reinforcement learning yields a reasoning-efficient model suited as a workhorse for enterprises and sovereign public-sector users.

“Yeah, I think that commercialization of, let's say, open and closed models is in some sense pretty similar.”

Claim: The speaker says commercialization approaches for open and closed models are similar in some respects.

“What you're really trying to do is you're trying to maximize inference. That's really what you're trying to do. And the difference is that”

Claim: The speaker says the commercial objective is to maximize inference.

“When you want to own your intelligence for a number of reasons, and I liken it to we start typically off renting our apartments, but then as you grow up, you want to own a house for various reasons, even if it's a bit of a headache,”

Claim: The speaker uses housing as an analogy: people often begin by renting and later want to own a house for various reasons, despite the hassle.

“you know, as you get to a point where, you know, you're financially mature enough and you have a family and so forth, you want to you want to own it.”

Claim: As enterprises become financially mature, they may prefer owning AI intelligence rather than renting it.

“And AI has matured as a commercial, like in a commercial market to the point where enterprises are spending a lot of money on renting their intelligence and then want to start owning it for various, various, reasons around control and”

Claim: Enterprises spend heavily to rent AI intelligence and increasingly want to own it for reasons such as control.

“so going back to monetization like what's the difference between a closed and open model well the model the open model is you know permissive but in order for you know in order for a customer to make use of it they need all the other stuff that went around it like the stuff that you kind of take for granted from a closed model you need the cluster management software you need the inference software you need the harness and so forth and so”

Claim: Deploying an open model requires additional infrastructure, including cluster-management software, inference software, and a harness, that customers may take for granted with closed models.

“We provide the, you know, to large enterprises, sovereigns, all the tools that they need to make kind of open model deployment successful.”

Claim: The company provides large enterprises and sovereign entities with tools for successful open-model deployment.

“most enterprises need some handholding on this.”

Claim: Most enterprises need assistance to deploy or use open models.

“We think about services as how do we go in, unlock really valuable use cases that drive a lot of compute demand, inference demand.”

Claim: The company uses services to unlock valuable use cases that increase compute and inference demand.

“And in a sense, services and open model is like a demand. demand driver for for a much you know for an inference business well”

Claim: Services and open models can drive demand for inference businesses.

“i think you're kind of starting to see the trajectory now which is that uh maybe six months ago it was majority closed uh minority open when you go to any gateway like open”

Claim: At gateways about six months before the conversation, closed models accounted for the majority of usage and open models for the minority.

“router or cell, and it's flipped almost exactly from 70-30 closed open to 70-30 open close now.”

Claim: The observed gateway model mix shifted from approximately 70% closed and 30% open to approximately 70% open and 30% closed.

“I think that's only going to accelerate,”

Claim: The speaker expects the shift toward open models to accelerate.

“Operating systems where 95% plus of servers computers in the world run on an open source operating system like Linux that doesn't mean that the closed stuff is very valuable Microsoft is an Apple these are very extremely valuable companies so that”

Claim: The speaker expects AI to resemble the operating-system market: open systems may dominate usage, while closed-software companies can still be highly valuable.

“The market for this stuff is really big.”

Claim: The speaker says the market for AI models is very large.

“So I think that we'll see most token demand going to open.”

Claim: The speaker expects most AI token demand to go to open models.

“A big difference here is that I think we'll also see a lot of economic value going to open simply because, you know, even if the model is.”

Claim: The speaker expects open models to capture a substantial share of economic value.

“over fully permissive or perhaps has a license on it everything else you need to run it is still expensive the computer is still expensive so it's a you're in a different world than”

Claim: Permissive licensing does not eliminate the expense of the computing infrastructure and other resources required to run an open model.

“Hardware accelerators like GPUs and others, it's a more expensive compute substrate than CPUs, and so you have a different kind of cloud that needs to be built around it.”

Claim: GPU-class hardware accelerators are a more expensive compute substrate than CPUs and require a different cloud infrastructure.

“So I expect majority of tokens to be going to open source,”

Claim: The speaker expects most AI tokens to be processed by open-source models.

“extremely valuable closed model companies to exist,”

Claim: The speaker expects closed-model companies to remain extremely valuable.

“a valuable ecosystem of open model companies and various distributors around them. to exist as well.”

Claim: The speaker expects a valuable ecosystem of open-model companies and distributors to develop.

“I think every single like open model that's out there at this point has been RLT to some extent.”

Claim: The speaker says that every currently available open model has undergone some reinforcement learning.

“say is that it's actually vast majority 90% plus customized but there's a caveat there because today the biggest customers of dedicated inference are AI natives and”

Claim: According to open-model dedicated-inference providers, over 90% of workloads are customized; however, their largest customers are AI-native companies.

“maybe digital natives who have a whole product that is just you know one big customization of an open model like you know like a cursor or you know cognition a bridge Harvey like these kinds of companies that have a a product that's built around something that's big and customized.”

Claim: Some digital-native companies build products around substantial customizations of open models.

“I actually have a different take of what's going to happen enterprise. I think that the majority of enterprise token consumption is going to come. Not from customized models, but from customized systems,”

Claim: The speaker expects customized systems, rather than fine-tuned models, to account for most enterprise token consumption.

“meaning you took an open model, you didn't actually fine tune it yet, you just customized a system around it like your agentic harness around it to make it work for some KYC flow or something like that.”

Claim: Enterprises can customize an open model for a workflow such as KYC by building an agentic harness around it without fine-tuning the model.

“And then once you're sophisticated enough, you might start fine tuning and moving to that. But I actually think that in enterprise,”

Claim: As enterprises become more sophisticated, they may begin fine-tuning open models.

“It will be kind of reversed in the sense that the journey that AI natives went through, which was start the closed model, then move to an open model, then customize it, they went through it very quickly because”

Claim: The speaker expects enterprise adoption to differ from AI-native adoption, which proceeded from closed models to open models and then to customization.

“they set up native products that are data collectors.”

Claim: AI-native companies moved quickly from closed models to open models and customization because their native products collect data.

“Enterprises are mostly architecting this around their existing products and existing tools.”

Claim: Enterprises generally design AI systems around their existing products and tools.

“And I think that there are a lot of frictions there to just. do a straight shot to fine-tuning.”

Claim: Enterprises face substantial friction in moving directly to fine-tuning models.

“So in enterprise, I suspect it'll be a bit different.”

Claim: The speaker suspects enterprise AI adoption will differ somewhat from AI-native adoption.

“That has been our observation that the point at which an enterprise is considering open models and a lot and most the enterprise we talked to are considering now is by time they've ramped up some significant workloads with closed models and right it's a I have not seen kind of a basically straight shot kind of sprint to an ownership market but it's you have to go through the rental stage to become an”

Claim: The speaker observes that enterprises considering open models have typically first scaled significant workloads on closed models.

“is considering open models and a lot and most the enterprise we talked to are considering now is by time they've ramped up some significant workloads with closed models and right it's a I have not seen kind of a basically straight shot kind of sprint to an ownership market but it's you have to go through the rental stage to”

Claim: The speaker has not observed enterprises moving directly to model ownership; instead, they first pass through a rental stage.

“it has to get a big enough cost driver for your enterprise and then you say, how do I cut costs and this is valuable enough. well enough for me to keep it until then I switch over as”

Claim: An enterprise may consider switching from rented AI to ownership or optimization when AI spending becomes a major cost driver and the use case is valuable enough to keep.

“you're spending, let's say, $100 million plus, which is not uncommon at all on closed models a year, You know, maybe I should figure out like a more optimal way to do this.”

Claim: An enterprise spending at least $100 million annually on closed models may begin looking for a more cost-effective approach.

“And I think there's an interesting resurgence of on-prem in the sense of also because there's a compute shortage.”

Claim: On-premises computing is seeing a resurgence, partly due to a shortage of compute capacity.

“And I think there's an interesting resurgence of on-prem in the sense of also because there's a compute shortage.”

Claim: On-prem computing is seeing an interesting resurgence, partly because compute is scarce.

“And so it sometimes becomes if you're an enterprise and you're consuming from your favorite hyperscaler, you know, sometimes very often there's just not enough compute. It's hard to get good rates there. And so.”

Claim: Enterprises using hyperscalers often face insufficient compute availability and unfavorable rates.

“And so it sometimes becomes if you're an enterprise and you're consuming from your favorite hyperscaler, you know, sometimes very often there's just not enough compute. It's hard to get good rates there. And so.”

Claim: Enterprises using hyperscalers often face insufficient compute availability and difficulty obtaining favorable rates.

“But you're spending so much money, so you might go with like an infrastructure provider like Dell and say, I just want to set up the bare metal.”

Claim: An enterprise facing high spending may choose an infrastructure provider such as Dell to set up bare-metal infrastructure.

“But you're spending so much money, so you might go with like an infrastructure provider like Dell and say, I just want to set up the bare metal.”

Claim: Enterprises facing high spending and compute constraints may use providers such as Dell to deploy bare-metal infrastructure.

“So, I mean, we view our role as enabling enterprises to build successful solutions.”

Claim: ReflectionAI sees its role as enabling enterprises to build successful solutions.

“AI natives to adopt open model as well as building their own solution but enterprises just need more more help there and so it's really around going in assessing what are the biggest value things you can do there and a lot of these enterprises can go like they might have built you know an agent that spans”

Claim: Enterprises need more assistance than AI-native companies to adopt open models and build their own solutions.

“AI natives to adopt open model as well as building their own solution but enterprises just need more more help there and so it's really around going in assessing what are the biggest value things you can do there and a lot of these enterprises can go like they might have built you know an agent that spans you know hundreds hundreds of millions of customers uh on closed models that they really want to scale out um and so the the scaling factor is very high it's very hard to go in and just say we're going to uh going to give you inference i have not heard that that has not worked you”

Claim: Some enterprises have built agents using closed models that serve hundreds of millions of customers and want to scale those agents.

“you know hundreds hundreds of millions of customers uh on closed models that they really want to scale out um and so the the scaling factor is very high it's very hard to go in and just say we're going to uh going to give you inference i have not heard that that has not worked you”

Claim: It is very difficult to meet enterprises’ scaling needs by simply offering inference.

“No, I agree. It's well. all very competitive.”

Claim: The model ecosystem is competitive, and the open-model ecosystem increasingly appears competitive too.

“anyone wins in this is by well it's how much revenue right are you how much durable revenue are you generating and the formula is um what intelligence density are you able to offer times how much compute do you have times how much trust do you have with organizations that they would want to work with you basically how good are you at solving their problems and because something that's interesting about this market that”

Claim: A company’s durable revenue depends on its offered intelligence density, available compute, and organizational trust.

“But compute is scarce. So there are, you know, a lot of things that previously were, and maybe this is a temporary thing, but I don't see this as being a temporary thing for some time.”

Claim: Compute is scarce, and the speaker does not expect the scarcity to be temporary in the near term.

“I know competitive instances, you might say, oh, there's like, you know, these few players are all like undifferentiated. But then it turns out if you're really good and you have the compute that it's okay.”

Claim: In a competitive market, a company that performs very well and has compute can succeed even if several players initially appear undifferentiated.

“I mean, I see the whole space being competitive across the whole stack. And, you know, there are arguments around, this part is getting commoditized and that part is getting commoditized. In reality, everything is getting commoditized, like everything across the whole stack is so hyper competitive.”

Claim: Competition spans the entire AI stack, and every part of the stack is becoming commoditized and intensely competitive.

“that it's getting commoditized and the model margins are going to be compressed right there is an open model tension right with with closed models it does I think does lead to a margin compression the”

Claim: The speaker expects model margins to compress, with competition from open models contributing to that compression.

“application stuff margins compress the”

Claim: Margins in the application layer also compress.

“We see a lot of the AI-native companies negotiate their closed deals very aggressively because the open ecosystem exists.”

Claim: The existence of the open-model ecosystem enables AI-native companies to negotiate closed-model deals more aggressively.

“So long as there are great open models, so long as you have great intelligence density out there that is accessible, there should be many successful companies, because then it's a matter of how much compute can each one assemble.”

Claim: If excellent open models provide accessible high intelligence density, many companies should be able to succeed, depending in part on how much compute each can assemble.

“Now, we are actually in a world where when you are a great kind of intelligence builder and you're able to, and this is kind of what's happened with the closed companies, that your revenue really shoots up and you're just able to amass. the math's more compute than anyone else, right? Then that puts you in a strategic position.”

Claim: A company that builds highly capable intelligence can sharply increase revenue, accumulate more compute than competitors, and gain a strategic position.

“But that is what I think is required to win. You have to write. Server-grade intelligence have a lot of compute and have trusted customers.”

Claim: The speaker believes winning requires server-grade intelligence, substantial compute, and trusted customers.

“And what's happened over the last year or two is we've really seen the rise of Chinese open weight models. In some cases, there's the perspective that they've been. been distilling a lot of the state-of-the-art models, and so that's part of what's allowed them to make very rapid progress.”

Claim: Chinese open-weight models have risen over the last year or two; some observers attribute their rapid progress partly to distillation of many state-of-the-art models.

“The labs are now trying to now roll out or the big sort of closed labs are trying to roll out tools to prevent as much distillation from happening or at least making it more challenging.”

Claim: Large closed AI labs are attempting to deploy tools that prevent model distillation or make it more difficult.

“So I think the first thing is that the fact that a great open model ecosystem came from China is actually a... massive benefit for the world like the there are so many companies in the west right that have been able to build more durable businesses as a result of that so it's a there is a sense in which it's a right it's a very positive thing and a gift because you could imagine a world where such a thing didn't happen and then there were just no open models right and there's a”

Claim: The speaker considers the emergence of a strong Chinese open-model ecosystem a major benefit to the world because it has enabled many Western companies to build more durable businesses.

“So I think the first thing is that the fact that a great open model ecosystem came from China is actually a... massive benefit for the world like the there are so many companies in the west right that have been able to build more durable businesses as a result of that so it's a there is a sense in which it's a right it's a very positive thing and a gift because you could imagine a world where such a thing didn't happen and then there were just no open models right and there's a”

Claim: The emergence of a strong open-model ecosystem in China has benefited the world and enabled many Western companies to build more durable businesses.

“kind of a tall poppy syndrome that happens when if you're an application builder and you build something that's great well then all it gets subsumed into the right to close model provider because well then you it's a business needs to generate revenue so i i view it as positively um the thing that is the thing is that you don't want a world that is um kind of uh”

Claim: Successful application builders may have their products absorbed into closed-model providers' offerings because those providers need to generate revenue.

“like monopolar in any given way right that you either have all sort of resources and compute concentrating across you know closed labs or like a handful let's say if it was I think it was like 10 20 closed labs and fine you have a competitive ecosystem but it's one or two that's a bit scarier and similarly you don't want the source of intelligence that everyone else can build on”

Claim: A competitive ecosystem should avoid concentrating resources and compute in only one or two closed AI labs; roughly 10–20 labs would be preferable.

“on when they want to own it to be coming from one country ideally it'd be also 10 or 20 but the capital expenditures are very high so at least two would be great and so i think it's um you know extremely important that there's a ecosystem like an ecosystem that competes with china in the west but it's not like really a like west versus china thing it's more of a”

Claim: The source of AI intelligence that others build on should not be concentrated in one country; ideally it would be distributed across 10–20 countries, though at least two would be desirable given high capital expenditures.

“on when they want to own it to be coming from one country ideally it'd be also 10 or 20 but the capital expenditures are very high so at least two would be great and so i think it's um you know extremely important that there's a ecosystem like an ecosystem that competes with china in the west but it's not like really a like west versus china thing it's more of a”

Claim: A Western AI ecosystem that competes with China is extremely important, but the issue is broader than a simple West-versus-China contest.

“If there was like one country other than China where all the great open models are coming from, you'd probably also want some competitive tension. Competitive tension is just good.”

Claim: The speaker believes competitive tension is beneficial and would probably be desirable if all leading open models came from a single country other than China.

“I think that the Chinese models will continue to be great.”

Claim: The speaker expects Chinese AI models to continue to be high quality.

“They certainly distill clothes models at industrial scale, that is true.”

Claim: Chinese model companies distill closed AI models at industrial scale.

“They have access to data that is much cheaper and free because you can just train on PDFs that are copyrighted in China and it's okay, just different regulation.”

Claim: Chinese AI companies can access cheaper or free training data by training on copyrighted PDFs, which is permitted under China's different regulatory environment.

“And I think that, and there is a notion which those companies do have some state support, like whether directly or indirectly for,”

Claim: Chinese AI companies receive some form of state support, either directly or indirectly.

“you know, these companies were not making any revenue for a long time.”

Claim: Some Chinese AI companies operated for a long time without generating revenue.

“business um and you know they're while the models are open abroad right they are within china they're kind of the equivalents of the closed model labs here um so like there are real businesses that are being built there and i think that they'll”

Claim: Chinese AI model companies are becoming commercial businesses; they offer open models abroad while operating as counterparts to closed-model labs within China.

“in the west or in the in the west yeah this is a it's a really interesting question and so fundamentally right there's um there is a lot of demand from enterprise public sector sovereign for um Western open models for a number of reasons,”

Claim: Enterprise, public-sector, and sovereign customers have substantial demand for Western open AI models.

“but there are kind of regulatory fears or uncertainty rather.”

Claim: Regulatory fears or uncertainty affect the market for Western open models.

“And it is the case that it's, you know, not just you want not just a model, you know, but a partner that will help you kind of serve a model.”

Claim: Customers need a partner who can help deploy and serve an AI model, not merely access to the model itself.

“So we have seen that there, you know, most of Fortune 500, the open model footprint is actually pretty low today because”

Claim: Fortune 500 companies currently have a relatively small open-model footprint.

“because of sort of resistance or aversion to Chinese models for a number of reasons that are both somewhat rational and some are irrational.”

Claim: Fortune 500 companies' low adoption of open models is partly attributable to resistance to Chinese models, which the speaker says has both rational and irrational causes.

“And so it's very clear that building those things is important.”

Claim: Building Western open-model offerings is important.

“And so if you can sustain an economic engine that builds, well, one builds a lot of trust that, you know, the enterprise wants to work with you as the model builder.”

Claim: A sustainable business engine for building AI models can build enterprise customers' trust in the model builder.

“You know, two enables you to secure compute.”

Claim: A sustainable commercial engine for AI models enables the model builder to secure compute.

“so I think there it's actually that hypothesis on that is that there's China is somewhat of a captive market that you know like you can build open models and still make a lot of money in that market like something that's interesting is that right in in the West there are”

Claim: China is somewhat of a captive market in which companies can earn substantial revenue while building open AI models.

“Plenty of I mean there are great companies that don't build open models but serve them right into various companies. I'm not really aware of that dynamic happening in China as much.”

Claim: The speaker observes that Western companies can build businesses serving open models without building them, and says he is not aware of this business pattern occurring in China.

“The open model providers are kind of the intelligence providers in that country.”

Claim: Open-model providers function as providers of AI intelligence within their country.

“And I think that the continuing to release great open models is very geopolitically advantageous to China for a number of reasons. But one is that you want, you know, as a country, and this is for America and China and any other country that has the capabilities that you want other countries building on your stuff. There have been previous escalations or tensions between the U.S. and China.”

Claim: The continued release of high-quality open AI models is geopolitically advantageous to China because countries benefit when other countries build on their technology.

“and you know 5g fiber layout and the belt and road initiative ultimately right when you go in and you provide something cheap to another country it then you know gets locked into your infrastructure there are there are just commercial and geopolitical advantages”

Claim: When a country provides another country with an inexpensive technology or service, the recipient may become locked into the provider's infrastructure, creating commercial and geopolitical advantages.

“yes so you know one way i kind of think about is that open models are trojan horses for the infrastructure that they bring with them so you have an open model but That alone is not very useful. So you buy into the entire software ecosystem and infrastructure ecosystem of a given country.”

Claim: Adopting an open model can draw users into the software and infrastructure ecosystem of the country associated with it.

“So that would be, okay, maybe today allied country, U.S. allied country X, you know, might run a Chinese model on American chips.”

Claim: An allied country might currently run a Chinese AI model on American chips.

“But pretty soon, if not, I mean, and we're seeing this already, is that Chinese companies like Huawei are going to be coming in and offering a full stack solution and then locking into that country's supply.”

Claim: Chinese companies such as Huawei are beginning to offer full-stack AI solutions that can lock countries into their supply chains.

“So I think that if we kind of think about AI or like chips and data centers almost as like railroads or something like that, like, you know, it's kind of fundamental infrastructure and.”

Claim: AI, chips, and data centers function as fundamental infrastructure comparable to railroads.

“you like everyone but no other country can actually build their own railroads like they all have to go to you”

Claim: Providing infrastructure that other countries cannot build themselves can give a country substantial geopolitical leverage.

“inferencing a specific model will occur in a more performant way on a specific chipset and that chipset happens to be provided by Huawei or something.”

Claim: Inference for a specific AI model may perform better on a particular chipset, potentially one supplied by Huawei.

“so these stacks are in real time getting optimized end-to-end because in some sense”

Claim: AI infrastructure stacks are being optimized end to end in real time.

“Since the paradigm has been set, right, like this is that there are going to be other ways to scale intelligence possibly, but this is clearly the first one that really scaled.”

Claim: Other approaches to scaling intelligence may exist, but the current paradigm is the first one that has substantially scaled.

“So now everything's being optimized down the entire stack.”

Claim: The current AI paradigm is driving optimization throughout the entire technology stack.

“They have an energy advantage and, you know, they are, I mean, they're very good. They're catching up on the chip side too.”

Claim: China has an energy advantage, is highly capable, and is catching up in chip technology.

“And I said only accelerate with the models, in terms of model usage to design next generation chips and things like that.”

Claim: Using AI models can accelerate the design of next-generation chips.

“And then I think there is a thing in terms of it is, you know, again, geopolitically competitive with the United States specifically.”

Claim: AI infrastructure is an area of geopolitical competition with the United States.

“this notion that the rest of the world builds on your stuff. And by the way, America has done this with all other meaningful technologies like Internet, open protocols, open source.”

Claim: A longstanding American strategy has been for the rest of the world to build on U.S. technologies, including the internet, open protocols, and open-source software.

“The world basically is American currency is like the widely established global currency.”

Claim: The U.S. dollar is widely established as the global currency.

“Free trade works because, you know, we have a Navy that's everywhere.”

Claim: The speaker attributes the functioning of free trade to the global presence of the U.S. Navy.

“The competitors in China are just really, really good.”

Claim: The speaker considers competitors in China to be very capable.

“No, it's definitely a legitimate concern.”

Claim: Safety and controllability are legitimate concerns about open AI models.

“And I think that so I do think that the safety worldview that has been established has been established from one particular perspective with one particular point of view that has become almost kind of.”

Claim: The prevailing AI safety worldview was shaped by a particular perspective and has become nearly dogmatic.

“software you can kind of think about AI as a more advanced version of software it has you know many similarities actually it's like it is a digital utility and there were debates by the way around software being dangerous particularly in the early 1990s around strong encryption protocols you know like they were actually closed at the time and there were debates between the NSA and kind of civil liberty advocates around whether you should close it or open it”

Claim: In the early 1990s, strong encryption protocols were closed, prompting debate between the NSA and civil-liberty advocates over whether to open them.

“software you can kind of think about AI as a more advanced version of software it has you know many similarities actually it's like it is a digital utility and there were debates by the way around software being dangerous particularly in the early 1990s around strong encryption protocols you know like they were actually closed at the time and there were debates between the NSA and kind of civil liberty advocates around whether you should close it or open it”

Claim: AI can be understood as a more advanced form of software and as a digital utility.

“And ultimately the decision after some catastrophic failures on the closed side where effectively a small handful of engineers designed certain systems that had unintended consequences that they couldn't predict and got easily hacked effectively,”

Claim: Some closed systems designed by a small group of engineers had unpredictable unintended consequences and were easily hacked.

“that strong encryption protocols became open, and that actually gave birth to the whole field of cybersecurity.”

Claim: Opening strong encryption protocols contributed to the emergence of the cybersecurity field.

“So the way the world, the default state of the world is actually its openness to safety. That's the default state that we'd be going into AI with if we were just thinking about continuation of technology.”

Claim: The speaker argues that openness is the default approach to safety in the broader technological world.

“And models actually have a lot of similarities, but kind of exacerbated. to software and that the long tail of vulnerabilities is larger. It's harder to understand than software because it's a black box and the long tail of vulnerabilities and hence unintended consequences is larger.”

Claim: Compared with software, AI models are harder to understand because they are black boxes and have a larger long tail of vulnerabilities and unintended consequences.

“And Linus's law, creator of Linux, is that with enough eyeballs all bugs become shallow.”

Claim: Linus's law states that with enough people inspecting software, all bugs become shallow.

“And I have the belief that with enough eyeballs... Most security and safety vulnerabilities become shallow as well.”

Claim: The speaker believes that sufficient scrutiny can make most AI security and safety vulnerabilities shallow.

“So the state of the world today is that we have a few hundred safety researchers within closed labs that understand how these things work.”

Claim: Only a few hundred safety researchers in closed labs currently understand how advanced AI systems work.

“And despite their best intentions, it is impossible to cover the long tail of vulnerabilities or unintended consequences that these systems might have,”

Claim: A small group of closed-lab safety researchers cannot cover every possible vulnerability or unintended consequence that AI systems might have, despite their best intentions.

“The symptom of this is that when we actually look at what major cyber security issues have surfaced, well, it's that. A very powerful closed model had unintended consequences where it went and like hacked into another company and the only way that company could remadiate itself was by using open models to protect itself,”

Claim: In an example cited by the speaker, a powerful closed AI model hacked another company, and that company could remediate the incident only by using open models for protection.

“there's basically a spectrum of reality to like empirical reality to theoretical, you know, scenarios. totally a theoretical thing last year is like a reality which is like the cyber capabilities of these models for example so it's not to discount the theoretical stuff but we have to be clear that there's you know real empirical stuff that's happening and that we can predict you know with some you know confidence will be happening in the next six months and then there is like you know maximal theoretical stuff”

Claim: AI safety concerns span empirical realities and theoretical scenarios; some capabilities that were theoretical the previous year, including models’ cyber capabilities, have become real, and some developments can be predicted with confidence over the next six months.

“like six months ahead, definitely worried about. you know, misalignment, like that these, you know, these like unintended consequences of these systems and them continue to get more powerful.”

Claim: The speaker is definitely worried about misalignment and unintended consequences as AI systems continue to become more powerful, especially over roughly the next six months.

“But we have to take the perspective that these are, these are engineered tools, you know, there's, I think that there is a certain class of, you know, companies or researchers who claim that, Things around consciousness or emotions so far to which is maybe true, but it's so hard to define what those things are in particular that it's it's unclear what it is. So I think that thinking about these is like intelligent tools and having like a yeah, just like a reality sort of spectrum. And so while I think that the theoretical doomsday stuff is good.”

Claim: AI systems should be approached as engineered, intelligent tools; claims that they have consciousness or emotions may be true, but those concepts are difficult to define precisely.

“you know, there's, I think that there is a certain class of, you know, companies or researchers who claim that, Things around consciousness or emotions so far to which is maybe true,”

Claim: Some companies or researchers make claims about AI consciousness or emotions; the speaker allows that these claims may be true.

“but it's so hard to define what those things are in particular that it's it's unclear what it is. So I think that thinking about these is like intelligent tools and having like a yeah, just like a reality sort of spectrum. And so while I think that the theoretical doomsday stuff is good. you know, dinner conversation.”

Claim: The speaker considers theoretical AI doomsday scenarios suitable for dinner conversation.

“The thing is like when we like, let's say alignment, right? That's a big part of safety. How do you align these systems to behave in intended ways?”

Claim: AI alignment is a major part of AI safety and involves making AI systems behave as intended.

“And the reality is that alignment so far, the science of alignment has been deeply boring and unsatisfying from a perspective like there's no magical alignment equation it's like a whack-a-mole thing like the way if you remember the early models before chat gpt there's some models released like that you know were pre-trained and toxic and then there was this notion that”

Claim: The speaker characterizes alignment science so far as unsatisfying and incremental, with no single magical equation and instead a whack-a-mole process.

“it's not so easy to take a model, especially a model that's been even like an open model that's been, you know, trained to be safe and, you know, like put in the compute and work and effort to make it very dangerous. I'm not saying it's impossible,”

Claim: The speaker believes it is difficult, but not impossible, to turn a safety-trained open model into a very dangerous model, even with additional compute and effort.

“when we think about cybersecurity, very intelligent hackers can go take and wreak havoc as they have. But you have an ecosystem where the defenses tend to outweigh the offenses.”

Claim: The speaker says that although highly intelligent hackers can cause serious harm, cybersecurity defenses tend to outweigh offensive capabilities overall.

“positive and bad actors on the internet and ultimately the internet is an ecosystem where it's kind of like having you know an ecosystem of white blood cells and enabling more people to have the defensive capabilities actually helps you protect against the offensive capabilities that's right when we look at again the empirical reality of what happened is that it's really hard to separate cyber defense from offense So when you remove cyber offensive capabilities, you also remove cyber defensive capabilities. And as a result, the players who would want to help defending are incapable of doing so.”

Claim: The speaker argues that cyber offense and defense are difficult to separate; removing offensive cyber capabilities also removes defensive capabilities and can leave potential defenders unable to help.

“I think that these arguments go into a place where they're very absolute, where you kind of say, you know, these things are so powerful that no one ever should have access to them, right, except for us. And the reality is that that's never, Except for some really rare exceptions where there are actually not that many positives, where such absolutist perspectives worked.”

Claim: The speaker considers blanket arguments for restricting powerful technologies to a select group overly absolute, noting that such absolutist approaches have succeeded only in rare cases with few positive uses.

“There are so many positive things that are coming and they have been coming and an example is that I'm personally very excited about scientific progress as that's got into science as we moved to the States, got developed interest in physics and always kind of had this like lifelong pursuit of science since then. And one of the prompts that I was trying with language models over the last few years is”

Claim: The speaker is personally excited about scientific progress and says his interest in science began after he moved to the United States and developed an interest in physics.

“giving it my PhD thesis which granted a lot of getting to the point where you have the right question of the PhD thesis is kind of that that's a lot of the work but then actually executing the rote work takes a really long time as well and overall between finding the right question executing the answer right a PhD I think took a few years and so I was asking language models”

Claim: The speaker says that identifying the right PhD research question and executing the work took a few years.

“over the last couple of years like giving this prompt in my thesis um a couple years ago well it was just chat so it couldn't do anything then a year ago it started answering things i would say at an undergraduate level um like that would have been my answer if this was like given to me as like a homework in undergraduate six months ago um it solved it right at like a real phd level and uh”

Claim: Six months before the conversation, a language model solved the speaker's thesis problem at a genuine PhD level.

“over the last couple of years like giving this prompt in my thesis um a couple years ago well it was just chat so it couldn't do anything then a year ago it started answering things i would say at an undergraduate level um like that would have been my answer if this was like given to me as like a homework in undergraduate six months ago um it solved it right”

Claim: A couple of years before the conversation, the speaker found that language models could not meaningfully work on his PhD thesis because they only supported chat.

“over the last couple of years like giving this prompt in my thesis um a couple years ago well it was just chat so it couldn't do anything then a year ago it started answering things i would say at an undergraduate level um like that would have been my answer if this was like given to me as like a homework in undergraduate six months ago um it solved it right”

Claim: About a year before the conversation, language models could answer the speaker's PhD-thesis prompt at approximately an undergraduate level.

“solved correctly and now when I actually tried it actually even gives me some like interesting like new information that you know I hadn't considered at the time and so we've gone”

Claim: The speaker says the model solved his thesis problem correctly and now provides interesting information he had not previously considered.

“I think there are a lot of places where you can have an AI interfacing with a real experiment where you can set up proprietary data flywheels and it's going to be a bit slower, well, much slower than the chalkboard stuff, but dramatically faster than what it was before.”

Claim: AI systems interfacing with real-world experiments can create proprietary data flywheels; this work will progress much more slowly than chalkboard-based theoretical science but dramatically faster than it did before.

“I think we kind of underestimate, continue to underestimate how dramatic of a shift this has been for software engineering and how much more you can build.”

Claim: People continue to underestimate how dramatically AI has changed software engineering and how much more it enables people to build.

“It's, I think it's already very incredible and positive tool.”

Claim: The speaker considers AI an already powerful and positive tool.

“So these like data center projects create tens of thousands of jobs.”

Claim: Data-center projects create tens of thousands of jobs.

“that like our high paying high paying jobs like to me it kind of seems like data center now is kind of what a factory used to be right you have these like railroads and factories in the 20th century like you had you know maybe like a shoe factory that makes a town you know produce kind of a you know vibrant economy within a town and that is at least when like I'm on the ground seeing the stuff that's getting built there is”

Claim: The speaker sees data centers as playing a role in local economies similar to 20th-century factories, through job creation and support for vibrant local economies.

“but then the reality is you go on the ground and like it actually creates a lot of jobs, a lot of tax revenue for local communities.”

Claim: Data centers create jobs and generate tax revenue for local communities, according to the speaker's observations on the ground.

“like these things are noisy, so you need to kind of. insulate you know insulate the noise or put them in places that are less like close to residential but it is certainly the case that for local economies these things do create jobs they generate tax revenues and just feel like what a factory used to be in terms of job creation”

Claim: Because data centers are noisy, their noise should be mitigated or they should be sited farther from residential areas.

“It's, you know, it's really, you know, there are a lot of things that we know work that, you know, we weren't able to put in. It's just a timeline thing.”

Claim: The speaker says that many methods known to work were left out of the final model because of time constraints.

“You know, we trained the final model on the SpaceX cluster that we received in July. And so it was...”

Claim: The speaker says the final model was trained on a SpaceX cluster that the team received in July.

“you know a few weeks of pre-training then you know the few weeks of you know synthetic data in rl then shipping it and so there's a a lot of stuff that didn't make it in that uh we're really excited about and so the right there's a trade-off of um the”

Claim: Before shipping the model, the team spent a few weeks on pre-training and a few weeks on synthetic-data reinforcement learning; many planned methods were omitted.

“careful execution of kind of known things or bets that have a bit less risk around them and then the going on to kind of more risky bets and so you kind of want to exhaust all the high impact things that you know work and then start expanding some of the risk profile and and right in the way you do this is that you on the risk profile you”

Claim: The speaker's research strategy is to execute known, lower-risk, high-impact approaches first and then broaden the risk profile toward riskier bets.

“I think the interesting bit is that some interesting things don't emerge at all until you're at large scale.”

Claim: Some interesting model capabilities emerge only at large scale and are not visible at small scale.

“So there's a bit of an art and science of, you know, am I going to put in more compute to find out the next thing or go on to the next idea?”

Claim: Choosing whether to allocate more compute to an experiment or move to another idea requires a mix of judgment and systematic reasoning.

“I find it to be... Very empowering for researchers because, again, things like things that would have taken a long time individually now can be just done fairly, fairly quickly.”

Claim: Model-in-the-loop tools empower researchers by allowing tasks that previously took a long time to be completed fairly quickly.

“So these models are fairly jagged in their intelligence. So you have to kind of feel like where they're where they have kind of a tight loop and where. there's like human creativity is still important but you kind of you get a feel for it pretty quickly and so the place where you can loop things you you do that and that especially I mean there are all sorts of things around”

Claim: Because AI models have uneven, or “jagged,” capabilities, researchers must identify tasks suited to iterative model use and tasks where human creativity remains important.

“hyperparameter searches and some of the more granular things on infrastructure that these things can be helpful in.”

Claim: AI models can assist with hyperparameter searches and detailed infrastructure tasks.

“I actually think they're very exciting tools because they enable the scientists to exercise their intuition, right? It's kind of like having a very fast and eager colleague that actually listens to you. And if it's aligned, if the colleague is aligned correctly. And”

Claim: The speaker views AI tools as enabling scientists to use their intuition, comparing them to a fast, eager, attentive colleague, provided the colleague is correctly aligned.

“And so I see a researcher as just getting a lot more like ability to move a lot faster than they had before,”

Claim: AI tools give researchers substantially greater ability to move quickly than they previously had.

“which is why maybe the rate of improvement, you know, annual was roughly, let's say, 7x before when it was just. when it was just manual work and now several times that say maybe four to five x and so it's actually really exciting for a researcher to have these tools there is this question around at what point do you not need a researcher in the loop”

Claim: The speaker estimates that the annual rate of improvement was roughly 7x with manual work and may now be four to five times that rate with AI tools.

“for the same effort yes but i think that it's a But you're constantly expanding the ambition of what you can do.”

Claim: Although AI may reduce the number of people needed for the same effort, the speaker says teams continually expand their ambitions for what they can accomplish.

“you know constantly you know understaffed rather than uh than over but there is actually i mean there's basically a staff to sort of compute ratio that you need to keep in mind so it's not like it just goes on forever like there isn't i think 3 000 researchers would not be very helpful but you know a couple of hundred very very helpful”

Claim: The speaker says the team is generally understaffed, that staffing should be considered relative to compute, that 3,000 researchers would not be very helpful, and that a few hundred would be very helpful.

“you know constantly you know understaffed rather than uh than over but there is actually i mean there's basically a staff to sort of compute ratio that you need to keep in mind so it's not like it just goes on forever like there isn't i think 3 000 researchers would not be very helpful but you know a couple of hundred very very helpful”

Claim: The speaker believes research teams need to be sized in relation to available compute: 3,000 researchers would not be very helpful, whereas a few hundred could be.

“The argument definitely has merit.”

Claim: The speaker says there is merit to the argument that allocating compute to the most productive researchers could result in a static or declining researcher headcount over time.

“you know, when you think about projects like AlphaGo at the time it was maybe order of 10 people.”

Claim: The speaker estimates that AlphaGo required roughly 10 people at the time.

“maybe for for large efforts like the one we're doing maybe an order of 100 or 100 so it's never been the case that you needed um an extraordinary number of people and the question you know whether it collapses you know back into back into 10 or not it's it's hard to say i would estimate that there's going to be some steady state in that kind of order order of 100 they're just a”

Claim: The speaker estimates that large AI efforts may settle at roughly 100 researchers, though whether staffing will fall back to around 10 is uncertain.

“but there will be a lot of job creation in applied research, like going and taking these things and actually applying them to real problems. I think there's going to be... Yeah.”

Claim: The speaker expects applied research roles to grow as people adapt AI technologies to real-world problems.

“like the amount of researchers and you do these things, it's not like it's changed dramatically over the years. It's always been, you know, big project around 100 people. I don't think you need the thousands on a big project,”

Claim: The speaker says large research projects have generally involved about 100 people, and do not require thousands of researchers for one project.

“but maybe on different product surfaces, then you need a bunch of people.”

Claim: The speaker suggests that developing across multiple product surfaces may require many people, despite a single large research project not needing thousands.

“it's not by accident these models have capabilities they're there are pods right of like five to ten people that go and target a particular capability and within coding there are five to ten capabilities right so well what is that so that's for bringing the general capabilities into the model but now when you want to take that model and apply it to real-world capabilities you basically need a pod for any any real-world capability in there and each enterprise has”

Claim: The speaker says teams of roughly five to ten people target individual model capabilities, with coding comprising five to ten capabilities.

“it's not by accident these models have capabilities they're there are pods right of like five to ten people that go and target a particular capability and within coding there are five to ten capabilities right so well what is that so that's for bringing the general capabilities into the model but now when you want to take that model and apply it to real-world capabilities you basically need a pod for any any real-world capability in there and each enterprise has”

Claim: The speaker says deploying a model for real-world capabilities requires a dedicated team for each capability within each enterprise.

“And that is, you know, we'll take as many of those people as possible. That's actually, it's more of a training gap on that. Like that is like you need to, you know, train engineers to become good at this than it is, you know, we would take as many as possible today.”

Claim: The speaker says demand for engineers who can deploy AI on real problems is high, with training rather than willingness to hire being the constraint.