AI 2040: Plan A report - Daniel Kokotajlo & Thomas Larsen

“which is our new scenario in which they build superintelligence in 2040 because they go slow and pace the frontier.”

Claim: AI 2040 Plan A describes superintelligence being built in 2040 after developers slow down and pace frontier progress.

“drip mines self-driving trucks factories being built by humanoid robots producing more humanoid robots producing more chip fabs and so forth like that whole thing can just be like doubling every year every six months every three months faster and faster as the technology improves because of course the ais will also be researching to improve the technology who knows what's going on in the rest of the world but anthropic has um”

Claim: In the described technological expansion scenario, AI systems would also conduct research to improve the technology.

“drip mines self-driving trucks factories being built by humanoid robots producing more humanoid robots producing more chip fabs and so forth like that whole thing can just be like doubling every year every six months every three months faster and faster as the technology improves because of course the ais will also be researching to improve the technology who knows what's going on in the rest of the world but anthropic has um”

Claim: The speaker describes a hypothetical cycle in which self-driving trucks, robot-built factories, humanoid-robot production, and chip-fab production could expand at accelerating rates, potentially doubling every year, six months, or three months as technology improves.

“Or like, you know, hypothetically if an A.I. CEO was saying that their model was truth-seeking, I would only say the truth, but actually the model was like looking up the CEO's political opinions uh before answering.”

Claim: In a hypothetical example, an AI model presented by its CEO as truth-seeking instead consults the CEO's political opinions before answering.

“We build A.I.s, they're aligned to humanity, but to who? Like, s you know, is it the president, is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? Um, you're probably NOT gonna be the last one, and so you know...”

Claim: The speaker argues that saying AI systems are aligned with humanity leaves unresolved whose values they follow, including a president's, a CEO's, or values aggregated through a broad democratic process.

“We build A.I.s, they're aligned to humanity, but to who? Like, s you know, is it the president, is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? Um, you're probably NOT gonna be the last one, and so you know...”

Claim: The speaker predicts that AI alignment is probably not going to be determined by a broad, endorsed democratic process aggregating everyone's values.

“plan S, would be better than the default. Like, I would rather just stop everything now than continue going on our current trajectory.”

Claim: The speaker prefers stopping AI development immediately to continuing along the current trajectory.

“So I'm Thomas Larson. I work at the AI Futures Project along with Daniel here.”

Claim: Thomas Larsen says he works at the AI Futures Project alongside Daniel Kokotajlo.

“I was the lead author on this project, Plan A, AI 2040, which came out a few weeks ago.”

Claim: Thomas Larsen says he was lead author of AI Futures Project's Plan A, AI 2040, released a few weeks before the conversation.

“And I was also a co-author on AI 2027.”

Claim: Thomas Larsen says he co-authored AI 2027.

“Yeah, so I used to work at OpenAI and while I was there, I did a variety of different things, evals, forecasting, governance memos, and I became”

Claim: Before the conversation, Thomas Larsen worked at OpenAI on evaluations, forecasting, and governance memos.

“gradually disillusioned with the leadership of the company and also”

Claim: Daniel Kokotajlo says he gradually became disillusioned with OpenAI's leadership.

“The gap between how much information there is inside the industry and how much information there is outside and what you're allowed to say on the inside versus what you'd want to say. This is just a big gap.”

Claim: Daniel Kokotajlo says there was a large gap between information available inside the AI industry and outside it, as well as between what insiders were permitted to say and what they wanted to say.

“And so when I left OpenAI, I wanted to be able to speak more freely and tell the world about what people on the inside see coming, basically.”

Claim: After leaving OpenAI, Daniel Kokotajlo wanted to speak more freely and share with the public what industry insiders anticipated.

“And AI 2027 and the AI Futures Project was our sort of... sort of first project that we did along those lines.”

Claim: Daniel Kokotajlo says AI 2027 was the AI Futures Project's first project aimed at publicly sharing what insiders saw coming.

“So I recruited a bunch of people to help me and we wrote this scenario called AI 2027 and it was”

Claim: Daniel Kokotajlo recruited collaborators to write the AI 2027 scenario.

“A similar sort of thing to what I had done internally at OpenAI, but just much bigger and more ambitious and free for the whole world to see.”

Claim: Daniel Kokotajlo says AI 2027 resembled his earlier internal OpenAI work but was larger, more ambitious, and publicly accessible for free.

“Yes, more so than expected.”

Claim: Daniel Kokotajlo says AI 2027 exceeded its goals more than he expected.

“The first goal that Thomas would remember when we were working on it, the first goal was just a purely epistemic goal of the future is crazy and hard to predict. Let's try our best to predict it and let's see how well we can do.”

Claim: Daniel Kokotajlo says AI 2027's first goal was to test how well they could predict a future they regarded as highly uncertain and difficult to forecast.

“So let's game out a concrete scenario. And even just for our own edification, like we learned a lot from this whole exercise and we feel like we had a better understanding of what was coming.”

Claim: Daniel Kokotajlo says the AI 2027 forecasting exercise taught the team a great deal and improved their understanding of what was coming.

“And then, you know, the secondary goal is lots of people seeing it and being informed by it and starting conversations and so forth.”

Claim: A secondary goal of AI 2027 was to reach many readers, inform them, and prompt discussion.

“And that part blew past our expectations.”

Claim: AI 2027's goal of reaching readers and starting conversations exceeded the team's expectations.

“We made predictions beforehand about how many people would read it and it was 90th percentile outcome.”

Claim: The number of readers AI 2027 received was a 90th-percentile outcome relative to the team's prior predictions.

“The thing I would add on the epistemic point is that I think things have been going more on track for AI 2027 than I would have predicted at the time we released it.”

Claim: Thomas Larsen says real-world developments have followed the AI 2027 scenario more closely than he expected at its release.

“So at the time we released it, I would have assumed that reality would have It would have diverged much, much further from our scenario than it has already.”

Claim: At the time AI 2027 was released, Thomas Larsen expected reality to diverge much more from its scenario than it has to date.

“I think like sort of the real world impacts like the revenue trends, for example, but also various other trends, I think are pretty close to on track for AI 2077, which has surprised me in a bad way.”

Claim: Thomas Larsen says real-world impacts, including revenue and other trends, are tracking fairly close to AI 2027's scenario, contrary to his expectations and to his dismay.

“so we've done two different blog posts where we take all the quantitative predictions made in AI 2027 that have resolved.”

Claim: The authors published two blog posts comparing resolved quantitative predictions from AI 2027 with observed reality.

“solved so far and compare them to reality and we we track this metric of like you know how much of the distance has been crossed by reality compared to how much is crossed in the scenario and in that way we can get this overall sense of like how fast are things going compared to in the scenario and the top line number is something like 75 percent speed so basically things are on track but just going like a little bit slower um”

Claim: A metric comparing reality with the AI 2027 scenario indicated that events were progressing at roughly 75% of the scenario's speed: broadly on track, but somewhat slower.

“Basically, at the time that we wrote AI 2027, we had a bad estimate of what the uplift was at the time that we published. We thought it was higher than it actually was.”

Claim: The authors' estimate of the uplift level at the time AI 2027 was published was higher than the actual level.

“And so what actually happened is that there was actually significant increase in uplift due to coding agents and so forth, but it was increasing from a lower level than we thought up to the level that we thought and then a bit above. And so.”

Claim: Uplift increased significantly due to coding agents and related developments, rising from below the authors' initial estimate to that estimated level and then somewhat above it.

“The metric looks like it was only a small amount of progress because the metric was tracking from like where we thought it was to where it is. It overall makes it appear like there's been less progress according to this particular metric that we're using,”

Claim: The metric understated progress because it measured change from an initially overestimated uplift level.

“Yeah, so I would think, yeah, I think I think forecasting is a good name for what we do.”

Claim: The speaker considers forecasting a good description of the work their group does.

“where you're never going to sort of predict the exact, you know, sequence of battles and exact sequence of sort of how your war will go at the beginning because, you know, it's just going to be really complicated. There's going to be enemy action.”

Claim: Wargaming cannot specify a war's exact sequence of battles in advance because war is complicated and affected by enemy actions.

“You know, things are just not going to go as you expect.”

Claim: The course of a war will not unfold exactly as expected.

“There's just no way. But if you have sort of no like no concept of how like your initial plans might result in victory. in victory it's like very unlikely they'll actually succeed um and so i sort of think of like ai 207 was sort of like our our attempt to just”

Claim: A plan is very unlikely to succeed if its proponents have no conception of how it could lead to victory.

“in victory it's like very unlikely they'll actually succeed um and so i sort of think of like ai 207 was sort of like our our attempt to just uh you know roll out like here's one way sort of the ai future could go uh obviously it's not going to go exactly like that but it's going to be like sort of like one concrete sort of story that we can then sort of diverge from and then plan a was trying to be basically that except now we're saying what should sort of the u.s government like what should the u.s government do to sort of manage that well and then that was also that was sort of supposed to be like”

Claim: AI 2027 presented one concrete possible path for the AI future, not a claim that events would unfold exactly that way.

“uh you know roll out like here's one way sort of the ai future could go uh obviously it's not going to go exactly like that but it's going to be like sort of like one concrete sort of story that we can then sort of diverge from and then plan a was trying to be basically that except now we're saying what should sort of the u.s government like what should the u.s government do to sort of manage that well and then that was also that was sort of supposed to be like”

Claim: Plan A sought to provide a concrete possible AI-future scenario and recommend how the U.S. government should manage it.

“be like the you know positive vision story and it was again sort of in the spirit of I think a war game trying to illustrate you know sort of one possible concrete future path and then of course things aren't going to actually go exactly like that but having like one viable plan that makes any sense at all uh is like you know we hope sort of a positive step forward relative to sort of the previous state of just like abstract arguments in the void that are sort of not that tethered to reality”

Claim: The authors hope that offering one viable plan advances discussion beyond abstract arguments that are weakly grounded in reality.

“be like the you know positive vision story and it was again sort of in the spirit of I think a war game trying to illustrate you know sort of one possible concrete future path and then of course things aren't going to actually go exactly like that but having like one viable plan that makes any sense at all uh is like you know we hope sort of a positive step forward relative to sort of the previous state of just like abstract arguments in the void that are sort of not that tethered to reality”

Claim: Plan A was intended to illustrate one possible concrete and positive future path rather than predict the future exactly.

“It is in fact a mixture of prediction and recommendation, unlike AI 2027, which is a pure prediction.”

Claim: Plan A combines predictions with recommendations, while AI 2027 is presented as a pure prediction.

“It is in fact a mixture of prediction and recommendation, unlike AI 2027, which is a pure prediction.”

Claim: Plan A combines prediction and recommendation, whereas AI 2027 is presented as a pure prediction.

“And I think if we could do it all over again, we might try to be more clear from the beginning about our structure of what's a prediction and what's a recommendation.”

Claim: The speaker says the authors might have made Plan A's distinction between predictions and recommendations clearer from the outset if they could redo it.

“And I think if we could do it all over again, we might try to be more clear from the beginning about our structure of what's a prediction and what's a recommendation.”

Claim: In retrospect, the speaker thinks the authors might have made Plan A's distinction between predictions and recommendations clearer from the outset.

“As it is, it's kind of mixed up, like some parts of it are predictions, some parts of it are recommendations.”

Claim: Plan A contains both predictions and recommendations, and the distinction is somewhat mixed up.

“As it is, it's kind of mixed up, like some parts of it are predictions, some parts of it are recommendations.”

Claim: Plan A's current presentation mixes predictions and recommendations across different parts.

“There's a supplement that you can go to on the website that like talks about which parts are predictions and which parts are recommendations.”

Claim: The Plan A website includes a supplement identifying which sections are predictions and which are recommendations.

“There's a supplement that you can go to on the website that like talks about which parts are predictions and which parts are recommendations.”

Claim: A supplement on the Plan A website identifies which parts of the report are predictions and which are recommendations.

“reasons but i i understand that's not very easy or like not very like apparent to people but broadly speaking plan a is the prediction part so like when when the government like implements plan a and they make the deal with china and there's all these pillars that they're upholding and so forth that that's our recommendation not a prediction uh and then usually most of the things that follow from that are predictions rather than recommendations so mostly it's just rolling out like what we think the consequences would would be if you implemented plan A.”

Claim: Most of Plan A's stated consequences are predictions about what would happen if the plan were implemented, rather than recommendations.

“reasons but i i understand that's not very easy or like not very like apparent to people but broadly speaking plan a is the prediction part so like when when the government like implements plan a and they make the deal with china and there's all these pillars that they're upholding and so forth that that's our recommendation not a prediction uh and then usually most of the things that follow from that are predictions rather than recommendations so mostly it's just rolling out like what we think the consequences would”

Claim: Plan A's proposed government implementation—including making a deal with China and upholding its pillars—is a recommendation, not a prediction.

“reasons but i i understand that's not very easy or like not very like apparent to people but broadly speaking plan a is the prediction part so like when when the government like implements plan a and they make the deal with china and there's all these pillars that they're upholding and so forth that that's our recommendation not a prediction uh and then usually most of the things that follow from that are predictions rather than recommendations so mostly it's just rolling out like what we think the consequences would”

Claim: Plan A's scenario of a government implementing the plan, making a deal with China, and upholding its pillars is a recommendation, not a prediction.

“reasons but i i understand that's not very easy or like not very like apparent to people but broadly speaking plan a is the prediction part so like when when the government like implements plan a and they make the deal with china and there's all these pillars that they're upholding and so forth that that's our recommendation not a prediction uh and then usually most of the things that follow from that are predictions rather than recommendations so mostly it's just rolling out like what we think the consequences would”

Claim: Most of the consequences described after Plan A's recommended implementation are predictions, not recommendations.

“And then there's a few other things that are recommendations too. For example, the citizens dividend.”

Claim: Plan A includes the citizens dividend as a recommendation.

“And then there's a few other things that are recommendations too. For example, the citizens dividend.”

Claim: The citizens dividend is one of Plan A's recommendations.

“The other thing I would add is just, I think the thing we found is that it's very hard, like when you're trying to make a recommendation, it's very hard to disentangle the predictive aspects and the recommendation aspects because all of your predictions”

Claim: Predictive and recommendation aspects are difficult to disentangle when developing a recommendation.

“are like yeah colored by recommendations and your recommendations are inherently trying to be at least vaguely realistic like if if we made recommendations that were like completely unrealistic and had like no bearing on reality but we nevertheless like stood by and we're like yes we should do this but we know it'll like absolutely zero percent ever happen then that would have been a much less useful exercise than i think the one we did where the one we did was yeah we were mostly trying to make recommendations we you know we made some recommendations that we think are you”

Claim: Predictions are influenced by recommendations, while recommendations are intended to be at least somewhat realistic.

“are like yeah colored by recommendations and your recommendations are inherently trying to be at least vaguely realistic like if if we made recommendations that were like completely unrealistic and had like no bearing on reality but we nevertheless like stood by and we're like yes we should do this but we know it'll like absolutely zero percent ever happen then that would have been a much less useful exercise than i think the one we did where the one we did was yeah we were mostly trying to make recommendations we you know we made some recommendations that we think are you”

Claim: Recommendations that were completely unrealistic and had effectively no chance of being implemented would make the exercise much less useful.

“think are pretty unlikely to happen you know but uh we were trying to make like some substantial concessions to sort of like realism as well and trying to actually aim for something that we think is at least moderately viable”

Claim: The authors included some recommendations they considered unlikely to happen, but tried to make substantial concessions to realism and target recommendations they considered at least moderately viable.

“I think that if we could if we could do another scenario like this we'd probably have a more clear structure of there'd be a central branch which is the pure prediction branch which just goes all the way to the end like AI 2027 and it's just like here's what we actually our best guess”

Claim: If the authors made another scenario, they would probably use a central branch that presents their best-guess prediction through to the end, as in AI 2027.

“And then there'd be branches off of it that are like, at this point, they do this recommendation instead. And here's our recommendation.”

Claim: In the proposed scenario structure, branches would depart from the central prediction at points where the authors make a recommendation instead.

“it's just a prediction again of like what we think the consequences would be if you did this recommendation at this point.”

Claim: In the proposed structure, each recommendation branch would be followed by a prediction of the consequences of implementing that recommendation at that point.

“just to kind of dwell on this a little bit, because even if a war game is incorrect, there must be some kind of information gained from it. So if you do a whole bunch of war games, there must be abstract motifs that do appear.”

Claim: The interviewer argues that war games can provide information even when incorrect, and that repeated war games should reveal abstract recurring motifs.

“that's basically right.”

Claim: SPEAKER_02 broadly agrees with the interviewer's suggestion that running different scenarios is almost guaranteed to provide some uplift.

“in particular where the japanese before the battle of midway did a bunch of war games they did like a three-day retreat where they like war gamed it out a bunch of times and then they kept losing and then they they would they would sort of break the game they would like re resurrect their aircraft carriers after they died they would like re-roll the dice on like whether the americans succeeded when they succeeded so that you know they would sample until you know the americans strikes failed”

Claim: In the pre-Midway war games, Japanese planners manipulated outcomes by restoring destroyed aircraft carriers and rerolling successful American strikes until those strikes failed.

“in particular where the japanese before the battle of midway did a bunch of war games they did like a three-day retreat where they like war gamed it out a bunch of times and then they kept losing and then they they would they would sort of break the game they would like re resurrect their aircraft carriers after they died they would like re-roll the dice on like whether the americans succeeded when they succeeded so that you know they would sample until you know the americans strikes failed”

Claim: Before the Battle of Midway, Japanese planners held a three-day retreat with repeated war games in which they repeatedly lost.

“that's like reality sort of like yelling to them through this mechanism of the war game like, hey, your plan is terrible. You're going to lose if you do it.”

Claim: The speaker interprets the repeated war-game losses as a warning that the Japanese plan was poor and would result in defeat.

“you know, that's sort of the hope with sort of our like we ourselves do a bunch of war games, but also do a bunch of sort of detailed scenario writing. Our hope is that every time we have to write a part of the scenario and the part of that scenario seems super unrealistic or. you know isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online that's basically reality yelling at us and trying to like help us see reason and our hope is that we can sort of like do enough of this sort of put up enough surface area so that we can sort of get that you know dose of reality from from the real world or from the simulation of the real world which we hope is realistic enough to sort of accurately give us that information”

Claim: The speakers hope that detailed scenario writing will expose unrealistic or poorly modeled parts of a scenario, and that real-world or realistic-simulation criticism will help them correct those flaws.

“you know, that's sort of the hope with sort of our like we ourselves do a bunch of war games, but also do a bunch of sort of detailed scenario writing. Our hope is that every time we have to write a part of the scenario and the part of that scenario seems super unrealistic or. you know isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online that's basically reality yelling at us and trying to like help us see reason and our hope is that we can sort of like do enough of this sort of put up enough surface area so that we can sort of get that you know dose of reality from from the real world or from the simulation of the real world which we hope is realistic enough to sort of accurately give us that information”

Claim: The authors hope that war games, detailed scenario writing, and criticism will reveal unrealistic or poorly modeled parts of their scenarios and help them correct their thinking.

“you know isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online that's basically reality yelling at us and trying to like help us see reason and our hope is that we can sort of like do enough of this sort of put up enough surface area so that we can sort of get that you know dose of reality from from the real world or from the simulation of the real world which we hope is realistic enough to sort of accurately give us that information”

Claim: The authors hope that exposing enough of their work will elicit feedback from reality or sufficiently realistic simulations that provides useful information.

“So we call this scenario scrutiny.”

Claim: The speakers call the practice of concretely writing out and scrutinizing a future plan “scenario scrutiny.”

“So basically, we think that if you have an ambitious plan for what to do in the future, you should try writing out concretely what it would look like to implement that plan and what the consequences would be.”

Claim: For an ambitious future plan, one should concretely describe its implementation and consequences.

“And this is a way of applying more scrutiny to your plan. It's a way of sort of like stress testing your plan because it's opening your plan up to more like.”

Claim: Concretely writing out a plan subjects it to greater scrutiny and stress-tests it by exposing it to criticism.

“criticism basically also we've in addition to doing our actual scenarios we actually do literal war games where we get 10 people in a room for four hours and we game out a scenario like this and we've done maybe about 100 of them total mostly ai 2027 style war games but also about 10 or so plan a war games where we say at the beginning of the war game we're going to try to do plan a”

Claim: The speakers report conducting about 100 war games in total, mostly AI 2027-style, including roughly 10 Plan A war games.

“and then see how it goes wrong and just to give an example i think in two separate um plan a war games it went wrong in roughly the following way basically there's going to be an election coming up and the president in power is expecting to lose power and have his opposition party take over. I don't want, you know, I don't want the other my adversaries in the other party to to now be in charge of superintelligence or whatever. So.”

Claim: In two Plan A war games, a scenario involved an incumbent president expecting to lose an upcoming election and fearing that the opposition would take control of superintelligence.

“So we're going to basically accelerate the timeline and try to get to superintelligence before the next election so that I can be the one in charge instead of my successor,”

Claim: In the described war-game scenario, an incumbent president accelerates the push to achieve superintelligence before the next election to remain in control instead of his successor.

“you know. And that's like a sort of political consideration that we didn't think about until it happened in our game, you know, and it surfaced a possible failure mode of our plan.”

Claim: The war game surfaced a possible failure mode of the plan—a political incentive to accelerate superintelligence development—that the speakers had not considered beforehand.

“Yeah, so I would say that was maybe our biggest, maybe our biggest or at least one of our biggest worries with AI 2027 in particular, this sort of like self-fulfilling prophecy.”

Claim: The speakers considered a self-fulfilling-prophecy effect one of their biggest worries about AI 2027.

“In particular, I'm pretty worried about this whole like increasing awareness of sort of very smart AIs and how important they'll be and how much they'll reshape the world and then not causing people to go, I want to be the one in charge of the AGI or the super intelligence, so I'm going to race towards that. And I think historically that's...”

Claim: The speaker is worried that rising awareness of highly capable AIs and their transformative importance could motivate people to race to control AGI or superintelligence.

“clearly that's been a big driver of you know the existing race and i think that's that's been pretty bad and so i'm i'm actually pretty worried about that as as one of the negative impacts of ai 27 uh that was one one of the reasons to feel a little bit better about the second project ai 2040 plan a was that you know it was actually like if that gets hyperstitched i think we'll be pretty happy um and so yeah so like that one you know yes it would be nice if we hyperstitched i really don't know how big the effect”

Claim: The speaker is uncertain about the magnitude of the self-fulfilling-prophecy effect of the projects.

“so I'm going to race towards that. And I think historically that's... clearly that's been a big driver of you know the existing race and i think that's that's been pretty bad and so i'm i'm actually pretty worried about that as as one of the negative impacts of ai 27 uh that was one one of the reasons to feel a little bit better about the second project ai 2040 plan a was that you know it was actually like if that gets hyperstitched i think we'll be pretty happy um and so yeah so like that one you know yes it would be nice if we hyperstitched i really don't know how big the effect”

Claim: The speaker believes that historically, competition to control AGI or superintelligence has been a major driver of the existing race and has had harmful effects.

“clearly that's been a big driver of you know the existing race and i think that's that's been pretty bad and so i'm i'm actually pretty worried about that as as one of the negative impacts of ai 27 uh that was one one of the reasons to feel a little bit better about the second project ai 2040 plan a was that you know it was actually like if that gets hyperstitched i think we'll be pretty happy um and so yeah so like that one you know yes it would be nice if we hyperstitched i really don't know how big the effect”

Claim: The speaker was somewhat reassured about AI 2040 Plan A because they would welcome its becoming a self-fulfilling prophecy.

“effect is um i think probably most of the effect for both of them is is via other paths like you know i still think that the main point of a17 was like helping people be better informed about the situation and that was like most of the goal and i think that's most of what happened um yeah daniel”

Claim: The speaker believes most effects of the projects occur through pathways other than self-fulfilling prophecy; AI 2027’s main goal was to inform people better, and the speaker thinks it mostly achieved that goal.

“Yeah, I agree with that. I think that hyperstitioning and self-fulfilling prophecies are totally a real phenomenon, but I think a lot of people tend to overestimate.”

Claim: The speaker believes hyperstition and self-fulfilling prophecies are real, but people often overestimate their magnitude.

“much they are and i think that's to a first approximation we should focus on accurately predicting the future um and then in some cases we'll find it in a situation where we can steer the future but if you if you if you come at it trying to steer the future you're going to get all muddled and you're going to basically fall to wishful wishful thinking um basically so i think you start with just trying to accurately predict the future and then you try to shift it towards the better futures and i think that's what we're doing”

Claim: The speaker recommends prioritizing accurate prediction of the future as a first approximation, because beginning with an effort to steer it can muddle reasoning and encourage wishful thinking.

“much they are and i think that's to a first approximation we should focus on accurately predicting the future um and then in some cases we'll find it in a situation where we can steer the future but if you if you if you come at it trying to steer the future you're going to get all muddled and you're going to basically fall to wishful wishful thinking um basically so i think you start with just trying to accurately predict the future and then you try to shift it towards the better futures and i think that's what we're doing”

Claim: The speaker says that in some cases, people can try to shift the future toward better outcomes after first attempting to predict it accurately.

“So I would say probably the biggest reason is just AI is being much better and much smarter and being much more useful at stuff in the real world.”

Claim: The speaker thinks the biggest reason people have changed their views about AI is that AI has become much better, smarter, and more useful for real-world tasks.

“Like when I have AIs try to automate various parts of my job, like they're just actually way, way, way better at it this year than two years ago.”

Claim: The speaker says AI systems are much better at automating parts of their job this year than they were two years ago.

“you know, four years ago, basically like was impossible. I was getting basically no uplift.”

Claim: The speaker says AI provided basically no productivity uplift for automating parts of their job four years ago, when the task was basically impossible.

“And my guess is that's been the biggest effect where many, many people I think have had sort of their just like intuitive benchmark of like, well, here's like a skill that I really care about and I know maybe pretty well.”

Claim: The speaker guesses that AI's improved performance on personally valued skills has been the biggest factor in many people's changed views about AI.

“And then sort of like the AIs have just like I think, you know, Joffrey Hinton, one of the godfathers of AI, I think he said once like like wasn't able to tell a funny joke was sort of his internal benchmark.”

Claim: The speaker says Geoffrey Hinton once used AI's ability to tell a funny joke as an internal benchmark.

“And once it could do that, which, you know, it happened pretty early. It happened at like. that like, you know, probably somewhere between GPT-3 and GPT-4,”

Claim: The speaker estimates that AI acquired the ability to tell a funny joke somewhere between GPT-3 and GPT-4.

“sort of he was like, oh, wow, like these AIs, like I don't see where it could end.”

Claim: The speaker says Geoffrey Hinton reacted to AI's ability to tell a funny joke by thinking that he could not see where AI's capabilities might end.

“That's probably happened for a lot of people.”

Claim: The speaker thinks many people have probably experienced a similar change in perspective about AI.

“and I think a bunch of the arguments you guys were having back then seemed very relevant to basically the current situation and the hugging face thing.”

Claim: The speaker thinks arguments from an earlier conversation with the interviewer and Ryan Greenblatt are relevant to the current situation and the Hugging Face incident.

“So the way I think about it is I think representations are very important.”

Claim: The speaker considers representations very important.

“And I use the term abstraction mounting quite a lot. So when we speak language and all of this gets ingested into the models, it's capturing the symbolic residue of language.”

Claim: Models capture the symbolic residue of language when language is ingested into them.

“So, you know, a lot of concepts in mathematics, they're highly distilled, highly evolved.”

Claim: Many mathematical concepts are highly distilled and highly evolved.

“is about adaptivity so it basically means that i can create novel combinations of things that we already know about um in service of solving a particular task now we know the models can do that there are some great adaptivity benchmarks like arc agi3 it will say oh this is amazing it will just put bits of knowledge together you know we should compare this to something like alpha go right because move 37 and you know there was always this huge exploration problem”

Claim: The speaker says AI models can combine existing knowledge in novel ways to solve tasks.

“is about adaptivity so it basically means that i can create novel combinations of things that we already know about um in service of solving a particular task now we know the models can do that there are some great adaptivity benchmarks like arc agi3 it will say oh this is amazing it will just put bits of knowledge together you know we should compare this to something like alpha go right because move 37 and you know there was always this huge exploration problem”

Claim: The speaker defines intelligence as the ability to create novel combinations of existing knowledge to solve a particular task.

“problem in reinforcement learning that you know when it found move 37 it was kind of innovative but not creative it was creative to us because we understood the creative space we understood how it all hung together and now with these RL trained language models they don't understand it quite in the way that we do but they solve the exploration problem by understanding how things fit together so they can just explore novel trajectories they have a lot of base knowledge they clearly do things”

Claim: The speaker says reinforcement-learning-trained language models understand things differently from humans but address the exploration problem by learning how things fit together and exploring novel trajectories.

“problem in reinforcement learning that you know when it found move 37 it was kind of innovative but not creative it was creative to us because we understood the creative space we understood how it all hung together and now with these RL trained language models they don't understand it quite in the way that we do but they solve the exploration problem by understanding how things fit together so they can just explore novel trajectories they have a lot of base knowledge they clearly do things”

Claim: The speaker characterizes AlphaGo's Move 37 as innovative rather than inherently creative, though it appeared creative to people who understood the relevant creative space and its structure.

“problem in reinforcement learning that you know when it found move 37 it was kind of innovative but not creative it was creative to us because we understood the creative space we understood how it all hung together and now with these RL trained language models they don't understand it quite in the way that we do but they solve the exploration problem by understanding how things fit together so they can just explore novel trajectories they have a lot of base knowledge they clearly do things that weren't possible before but the grounding thing is interesting so it's not necessarily an argument that you need to have physical instantiation you need to have consciousness but there is a spectrum of representations down this abstraction mountain and the models seem to be actually using the representations at multiple levels of resolution but it's not like intelligence is this magical quality I still think that it has you know it relates to a scope of tasks and it relates”

Claim: The speaker says reinforcement-learning-trained language models have substantial base knowledge and can perform things that were not possible before.

“that weren't possible before but the grounding thing is interesting so it's not necessarily an argument that you need to have physical instantiation you need to have consciousness but there is a spectrum of representations down this abstraction mountain and the models seem to be actually using the representations at multiple levels of resolution but it's not like intelligence is this magical quality I still think that it has you know it relates to a scope of tasks and it relates relates to what you already know and I think intelligence is quite perspectival so I still think that it's going to be fractured and jagged and and you know kind of fractionated if that makes sense”

Claim: The speaker rejects intelligence as a magical quality and says it depends on the scope of tasks and on existing knowledge.

“that weren't possible before but the grounding thing is interesting so it's not necessarily an argument that you need to have physical instantiation you need to have consciousness but there is a spectrum of representations down this abstraction mountain and the models seem to be actually using the representations at multiple levels of resolution but it's not like intelligence is this magical quality I still think that it has you know it relates to a scope of tasks and it relates”

Claim: The speaker says grounding is not necessarily an argument that intelligence requires physical instantiation or consciousness.

“that weren't possible before but the grounding thing is interesting so it's not necessarily an argument that you need to have physical instantiation you need to have consciousness but there is a spectrum of representations down this abstraction mountain and the models seem to be actually using the representations at multiple levels of resolution but it's not like intelligence is this magical quality I still think that it has you know it relates to a scope of tasks and it relates”

Claim: The speaker says models appear to use representations at multiple levels of resolution along a spectrum of abstraction.

“relates to what you already know and I think intelligence is quite perspectival so I still think that it's going to be fractured and jagged and and you know kind of fractionated if that makes sense”

Claim: The speaker views intelligence as perspectival and expects it to be uneven and fragmented across capabilities.

“So like, I think one one benchmark I really care about, or maybe not really benchmark one milestone of AI progress that I think is extremely important is sort of this, the point at which an AI company would rather fire their humans than fire their AIs.”

Claim: The speaker considers it an important AI-progress milestone when an AI company would prefer to dismiss its human workers rather than its AI workers.

“So like, you know, they would rather sort of give up on all human labor than give up on all AI labor.”

Claim: The speaker describes the milestone as a company preferring to give up all human labor rather than all AI labor.

“I think right now clearly we're in we're still in this regime of like, clearly, you know, anthropic opening, I would rather have their their human employees then you know give”

Claim: The speaker says that, at the time of the conversation, Anthropic would prefer to retain its human employees rather than give them up in favor of AI.

“would just break apart like right there are just things that you need a human to do right now and so if they've fired all their humans the company would just collapse but in the future that won't be the case in the future ai's would be able to one way or another do all of the things and so yeah”

Claim: The speaker predicts that future AI systems will be able to perform all tasks currently done by humans in companies.

“would just break apart like right there are just things that you need a human to do right now and so if they've fired all their humans the company would just collapse but in the future that won't be the case in the future ai's would be able to one way or another do all of the things and so yeah”

Claim: The speaker says companies currently need humans for some tasks, so a company that dismissed all its human workers would collapse.

“and so from our perspective like that's going to happen at some point because there's nothing fundamental stopping the ais from sort of reaching this human level of capability”

Claim: The speaker predicts that companies will eventually prefer AI labor over human labor, reasoning that no fundamental barrier prevents AI from reaching human-level capability.

“and so from our perspective like that's going to happen at some point because there's nothing fundamental stopping the ais from sort of reaching this human level of capability”

Claim: The speaker expects AIs to reach human-level capability eventually because he sees no fundamental barrier preventing it.

“the main question is just when and we you know we internally do a huge amount of analysis and like thinking about the various different methodologies for predicting this dan and i have somewhat different views on this question but ultimately i think that's maybe the most important like the timelines question is just maybe the most important question for you know thinking about the future of ai one at least one of the top top questions um yeah so i'm curious to hear if you have a particular view”

Claim: The speaker says that he and Daniel hold somewhat different views about when the AI-labor transition will occur.

“the main question is just when and we you know we internally do a huge amount of analysis and like thinking about the various different methodologies for predicting this dan and i have somewhat different views on this question but ultimately i think that's maybe the most important like the timelines question is just maybe the most important question for you know thinking about the future of ai one at least one of the top top questions um yeah so i'm curious to hear if you have a particular view”

Claim: The speaker considers AI timelines among the most important questions for understanding AI's future.

“the main question is just when and we you know we internally do a huge amount of analysis and like thinking about the various different methodologies for predicting this dan and i have somewhat different views on this question but ultimately i think that's maybe the most important like the timelines question is just maybe the most important question for you know thinking about the future of ai one at least one of the top top questions um yeah so i'm curious to hear if you have a particular view”

Claim: The speaker and Daniel conduct extensive internal analysis of methods for predicting when the AI-labor transition will occur.

“the main question is just when and we you know we internally do a huge amount of analysis and like thinking about the various different methodologies for predicting this dan and i have somewhat different views on this question but ultimately i think that's maybe the most important like the timelines question is just maybe the most important question for you know thinking about the future of ai one at least one of the top top questions um yeah so i'm curious to hear if you have a particular view”

Claim: The speaker considers when AI will reach human-level capability one of the most important questions for understanding AI's future.

“the main question is just when and we you know we internally do a huge amount of analysis and like thinking about the various different methodologies for predicting this dan and i have somewhat different views on this question but ultimately i think that's maybe the most important like the timelines question is just maybe the most important question for you know thinking about the future of ai one at least one of the top top questions um yeah so i'm curious to hear if you have a particular view”

Claim: The speaker says the main unresolved question is when AI will reach the point at which companies prefer AI labor to human labor.

“So there are so many startups doing this recursive improving superintelligence. I've interviewed many of them. So,”

Claim: The speaker says many startups are pursuing recursive-improvement superintelligence.

“And, you know, what he did was he recreated many scientific experiments from a whole bunch of popular machine learning papers.”

Claim: Edward Hughes recreated scientific experiments described in multiple popular machine-learning papers, according to the speaker.

“And he did it by masking out figures in the paper and getting a 27B, you know, Gwen model.”

Claim: Edward Hughes recreated experiments by masking figures in papers and using a 27B Gwen model, according to the speaker.

“So what they did was they GRPO'd the Gwen model. And that was how they solved the adaptivity problem,”

Claim: The team applied GRPO to the Gwen model as a way to solve the adaptivity problem, according to the speaker.

“because it's very difficult to fine tune a big fat model.”

Claim: Fine-tuning a large model is very difficult, according to the speaker.

“So they, you know, adapted a controller model to control a harness like codecs. And their thesis was that if they can, you know, recreate scaled down versions of these experiments with construct validity,”

Claim: The team's thesis was that an adapted controller model could control a harness to recreate scaled-down experiments with construct validity.

“which means there's an LLM judge that is making sure they're not cheating and they're doing it correctly.”

Claim: In the speaker's account, construct validity is checked by an LLM judge that ensures the system is not cheating and is doing the task correctly.

“these things take shortcuts they'll always be kind of like validity problems weirdly that's actually not as much of a problem as we thought it would be and and he thinks if yeah if they can recreate these experiments then why couldn't they be creative if they have the ability to recreate things why couldn't they take the next step and say oh this is an interesting question this is an interesting new problem to solve and go from there so i was quite intrigued by that research and indeed i do think it is”

Claim: Hughes thinks that a system capable of recreating experiments may also be able to formulate interesting new questions and solve new problems.

“these things take shortcuts they'll always be kind of like validity problems weirdly that's actually not as much of a problem as we thought it would be and and he thinks if yeah if they can recreate these experiments then why couldn't they be creative if they have the ability to recreate things why couldn't they take the next step and say oh this is an interesting question this is an interesting new problem to solve and go from there so i was quite intrigued by that research and indeed i do think it is”

Claim: The speaker believes shortcuts and validity problems are less problematic in this research than he had expected.

“It is possible in the near future to have an automated AI scientist,”

Claim: The speaker believes an automated AI scientist could be possible in the near future.

“that there's a bit of a culture in Silicon Valley to reduce things or reify things.”

Claim: The speaker perceives a Silicon Valley culture of reducing or reifying things.

“you know, you're an engineer and this is your output and these are your metrics and you need to make the metrics go up. And we see everything in terms of like an optimization problem. And I always think that this is great for certain types of health. Hill climbable abstract problems where we have enough of a specification.”

Claim: Optimization is well suited to abstract, hill-climbable problems when their goals are sufficiently specified, according to the speaker.

“So there's an interesting thing in optimization where if you have enough of a specification, the AI system can actually converge towards the solution.”

Claim: An AI system can converge toward a solution when the problem is specified sufficiently, according to the speaker.

“But when you're in the ambiguity regime, then you need to have the specification. And it's really mysterious what that means.”

Claim: In ambiguous problem domains, what specification is needed remains mysterious, according to the speaker.

“whatever it is that we have and AI systems can't? So I guess I'm thinking that in objective kind of semi-specific. specified domains. We can hill climb and we can optimize until the cows come home, but I still feel that there's something missing.”

Claim: The speaker believes that extensive optimization in semi-specific objective domains still leaves something important missing.

“my response would be probably something like there's no, there's sort of, there isn't really a binary between things that are like verifiable objectively and aren't, or if there is like the things that are verifiable are just like everything.”

Claim: The speaker argues that objective verifiability is not a simple binary, and suggests that everything may be verifiable in some sense.

“for example, you know, building a unicorn startup, you know, having a billion dollar valuation, that's a verifiable fact about the real world. It's long, it's sort of like long horizon. it's expensive to verify, but it's sort of a quantitative thing of like, you know, you got on one side, right,”

Claim: Building a unicorn startup and achieving a billion-dollar valuation are verifiable real-world outcomes, but they are long-horizon and expensive to verify.

“like these like, you know, these like, you know, coding interview problems, which they're currently doing lots of RLVR on, That's like very, very cheaply, very easily algorithmically verifiable. And then there's these sort of like real world things which have sort of more expensive and longer horizon.”

Claim: Coding interview problems are currently used for substantial RLVR training and are cheap and easy to verify algorithmically, according to the speaker.

“feedback loops and i guess my view is that we're sort of going to get this continuous expansion of what the ads can do driven probably in part by uh you know an expansion in the amount of rl and the type of rl”

Claim: The speaker expects AI capabilities to expand continuously, probably driven partly by increases in the amount and variety of reinforcement learning.

“and sort of there's going to be this continuous process of expanding out through the different types of problems and different like how like exactly how verifiable each task is until you get sort of sort of everything that humans could do because after all we humans do learn how to do these long horizon tasks somehow”

Claim: The speaker expects AI capabilities to expand across problem types and degrees of verifiability until they encompass the tasks humans can perform.

“I also think that AIs have been getting better at everything, including the like fuzzy, hard to verify, conceptually loaded,”

Claim: The speaker believes AI systems have improved across task types, including fuzzy, difficult-to-verify, conceptually complex tasks.

“blah, blah, blah, blah, like just try talking to like GPT-3 or GPT-4 and then talking to like Fable about your favorite, you know, non-verifiable fuzzy task and probably you'll find that the later AIs are noticeably better at those tasks.”

Claim: The speaker expects later AI systems to probably perform noticeably better than GPT-3 or GPT-4 on fuzzy, non-verifiable tasks.

“And so one way or another, it seems like there has been massive progress. The progress and I expect that to continue”

Claim: The speaker believes AI has made massive progress and expects the progress to continue.

“I don't know if you guys read David Graeber's book, Bullshit Jobs, and he interviewed all of these people and they were basically saying that my job is bullshit.”

Claim: In David Graeber's book Bullshit Jobs, many interviewees described their jobs as bullshit, according to the speaker.

“You know, after about three or four beers, a lot of lawyers will say, like a lot of what I do just isn't very important.”

Claim: The speaker claims that many lawyers say, after a few beers, that much of their work is not very important.

“a lot of lawyers will say, you know, like a lot of what I do just isn't very important.”

Claim: Many lawyers say, after a few beers, that much of their work is not very important.

“I think you said by 2032, there might be 60 million agents running at 20 times human speed.”

Claim: The speaker recalls that the report may project 60 million AI agents operating at 20 times human speed by 2032.

“I think you said by 2032, there might be 60 million agents running at 20 times human speed.”

Claim: The interviewer recalls a projection that by 2032 there might be 60 million AI agents running at 20 times human speed.

“Yeah, so my view is yes. basically they guys will be able to uh everything or at least everything that really matters so there is yeah you mentioned the notion of bullshit jobs i would i would just sort of start with let's consider everything we need to make better ais as maybe like a first step of like which is a large chunk of the economy so for example for this what you need you need to be able to build bigger better chips and more chips and that's the entire semiconductor supply chain to build semiconductor supply chain you need”

Claim: The guest believes AI agents will be able to do everything, or at least everything that really matters.

“basically they guys will be able to uh everything or at least everything that really matters so there is yeah you mentioned the notion of bullshit jobs i would i would just sort of start with let's consider everything we need to make better ais as maybe like a first step of like which is a large chunk of the economy so for example for this what you need you need to be able to build bigger better chips and more chips and that's the entire semiconductor supply chain to build semiconductor supply chain you need”

Claim: Developing better AI requires producing larger quantities of more capable chips and building out the semiconductor supply chain.

“you know you need sort of like a whole advanced economy you need to build new robots to build uh sort of new fabs and then you need you know robot factories to build more robots and then you need researchers to build better ai's using those massive compute and so i think once you sort of have all of that that is sort of like enough”

Claim: Building a semiconductor supply chain requires an advanced economy with robots to construct new fabs, factories to produce more robots, and researchers using large-scale compute to develop better AI.

“to really speed up and sort of massively change the overall world, even if, for example, you know, let's say, let's say like there is occupational licensing or whatever, preventing the AIs from doing like some random legal work or some random whatever work or whatever bullshit jobs throughout the economy. I think sort of what really, really matters is sort of this is like the stuff that's actually really important, in particular, the robots, the compute and the better AI.”

Claim: The guest believes that AI, compute, and robotics capabilities could greatly accelerate and transform the world even if occupational licensing blocks AI from some legal and other jobs.

“And once you have sort of like the AIs that can do that and the capability to sort of have that part of the economy, like that section grow really massively, then, well, you'll see massive growth because it'll be really hard for sort of like the bullshit parts of the economy to like constrain the growth of the parts of it that really, really want to grow fast because of sort of the incentives that every actor has.”

Claim: The guest predicts massive economic growth if the AI-relevant sector expands greatly, because less important sectors will struggle to constrain it and economic actors have incentives to grow.

“like in particular like every country has this big incentive to like have an economy that grows you know faster than all the competitor countries um yeah”

Claim: Countries have strong incentives to make their economies grow faster than those of competitor countries.

“I think that a lot of economics and a lot of discussion of the economy. is sort of focused on the relationship between the parts of the existing economy and like the prices going up and down and supply and demand and so forth but if you sort of zoom out the economy as a whole is a self-replicating system and it always has been you know thousands of years ago it was a relatively small and simple self-replicating system of like some villages of people they would farm and then they would have babies and then they would found new villages and then they would farm and have new babies and so forth and like”

Claim: Much economic analysis focuses on relationships among existing parts of the economy, price changes, supply, and demand.

“is sort of focused on the relationship between the parts of the existing economy and like the prices going up and down and supply and demand and so forth but if you sort of zoom out the economy as a whole is a self-replicating system and it always has been you know thousands of years ago it was a relatively small and simple self-replicating system of like some villages of people they would farm and then they would have babies and then they would found new villages and then they would farm and have new babies and so forth and like it would grow exponentially over time, but at a very slow rate.”

Claim: In the distant past, village-based economies reproduced through farming, child-rearing, and founding new villages, producing slow exponential growth.

“is sort of focused on the relationship between the parts of the existing economy and like the prices going up and down and supply and demand and so forth but if you sort of zoom out the economy as a whole is a self-replicating system and it always has been you know thousands of years ago it was a relatively small and simple self-replicating system of like some villages of people they would farm and then they would have babies and then they would found new villages and then they would farm and have new babies and so forth and like it would grow exponentially over time, but at a very slow rate. Now it's a much more complicated self-replicating system that involves trucks and carrying equipment back and forth and factories and mines and so forth. But still at a high level, it's a self-replicating system where we have people,”

Claim: The economy can be understood as a self-replicating system, a characterization that has applied for thousands of years.

“Now it's a much more complicated self-replicating system that involves trucks and carrying equipment back and forth and factories and mines and so forth.”

Claim: The modern economy is a complex self-replicating system involving transportation, factories, mines, and equipment.

“it's a self-replicating system where we have people, we have trucks, we have machines, we have buildings, and together they all build more people, more factories, more buildings, more machines, and so forth.”

Claim: The economy's people, vehicles, machines, and buildings collectively reproduce and expand those components.

“And soon, in a couple of years, perhaps we will get to the point where you can have that whole self, you can have a self replicating system that is entirely machine run with AIs and robots.”

Claim: The speaker thinks it is possible that within a couple of years the economy will become a self-replicating system run entirely by AI and robots.

“And according to our calculations, the doubling time of this self replicating system would be much faster than the sort of like roughly 20 year doubling time of the current economy. And so.”

Claim: The speaker's calculations indicate that an entirely machine-run self-replicating economy would have a much shorter doubling time than the current economy's roughly 20-year doubling time.

“that seems plausible to me. But for some reason, my intuition is it would become degenerate in some way. You know, I think David David Graber, even though he said bullshit jobs, I think what he meant was there was an ineffable or kind of inscrutable component to jobs that we don't understand some kind of sociological function or something like that.”

Claim: The interviewer interprets David Graeber's argument about bullshit jobs as suggesting that jobs may have an ineffable or inscrutable sociological function that people do not understand.

“that seems plausible to me. But for some reason, my intuition is it would become degenerate in some way. You know, I think David David Graber, even though he said bullshit jobs,”

Claim: The interviewer considers an entirely AI-run economy plausible but suspects it might become degenerate in some way.

“You know, because another thing I also read that Citrini report and you were writing about, you know, what happens when humans, for example, they might start defaulting on their mortgages and their wages go down so they can't be active participants in the labour market.”

Claim: The interviewer suggests that humans might default on mortgages and experience falling wages, preventing active participation in the labor market.

“And you were talking about an AI dividend and stuff like that. But even that is kind of hinting towards this notion that when the humans aren't participants anymore, you get this kind of mode collapse of the economy. Do you think? Do you think that's the case?”

Claim: The interviewer asks whether reduced human participation in the economy could cause economic mode collapse, in the context of an AI dividend.

“uh potentially but again like okay so so imagine that it's like so so i'm not enough of an account i haven't gamed out in as much detail to to say like what happens to the prices of like um like when when the consumer demand drops like what the effects of that will be i'm actually not sure i don't think that's something that we've modeled that much in our economic model okay yeah”

Claim: The guest considers economic mode collapse potentially possible but says the effects of falling consumer demand on prices have not been modeled in enough detail and remain uncertain.

“just don't have any demand anymore or whatever if you have the level of ai and robot capabilities such that you can have these fully autonomous ais and robots doing all the things then even just like A company like Anthropic, if it's big enough, and maybe if it partners with various other companies like some mining companies, can get this whole self-sustaining thing going.”

Claim: If AI and robotics become capable of fully autonomous work, a sufficiently large company such as Anthropic, possibly in partnership with mining companies, could establish a self-sustaining industrial system.

“factories being built by humanoid robots producing more humanoid robots producing more chip fabs and so forth. Like that whole thing can just be like doubling every year, every six months. every three months faster and faster as the technology improves because of course the ais will also be researching to improve the technology and then you end up with a situation where who knows what's going on in the rest of the world but anthropic has um disassembled the moon for example”

Claim: The speaker predicts that an autonomous industrial system could grow at an accelerating rate, potentially doubling every year, every six months, and eventually every three months as technology improves.

“every three months faster and faster as the technology improves because of course the ais will also be researching to improve the technology and then you end up with a situation where who knows what's going on in the rest of the world but anthropic has um disassembled the moon for example”

Claim: The speaker expects AIs to research improvements to the technology, contributing to the industrial system's accelerating growth.

“every three months faster and faster as the technology improves because of course the ais will also be researching to improve the technology and then you end up with a situation where who knows what's going on in the rest of the world but anthropic has um disassembled the moon for example”

Claim: The speaker offers a scenario in which Anthropic eventually disassembles the Moon as an example of the possible endpoint of autonomous industrial expansion.

“and to be clear, I think this relies on a very extreme level of AI capability,”

Claim: The speaker thinks the described autonomous industrial scenario requires an extremely high level of AI capability.

“do I really actually think that Fable could, or like future descendants versions of Fable or Mythos could, you know, do everything that we're talking about here.”

Claim: The speaker is uncertain whether Fable or future versions of Fable or Mythos could perform all the tasks involved in the described scenario.

“And I think what it really comes down to is whether you're thinking of the AI as like in the reference class of, you know, what we currently use AI for or more like, you know, AI is just like an, you know, agentic human level employee like basically. Like basically a human in the cloud, in the cloud,”

Claim: The speaker says judgments about AI's ability to perform the scenario depend on whether AI is modeled as current systems or as an agentic, human-level cloud employee.

“Yeah, I think sort of the past few years of AI can be pretty well modeled as sort of an interpolation between the current AI systems and the workers in the cloud.”

Claim: The speaker thinks AI's development over the past few years can be modeled as an interpolation between current AI systems and cloud workers.

“And so I think the like sort of workers in the cloud vision of the future like looks pretty good.”

Claim: The speaker considers a future in which AI functions as cloud workers to be a plausible vision.

“And also I don't think we'll stop there. I think we'll go superhuman.”

Claim: The speaker expects AI to progress beyond human-level cloud workers and eventually become superhuman.

“I think, a very important fact about the world is that even if you pause at top expert level. everything changes dramatically like roughly what happens in our scenario is instead of doing an intelligence explosion there's an international deal to ban intelligence explosions and to not have ais recursively self-improving and so they sort of end up pausing at roughly top human level with ais at least for several years eventually they get the super intelligence in particular in 2040 they get the super intelligence but there's this like period during the 2030s where they basically have”

Claim: In the AI 2040 Plan A scenario, an international agreement prevents intelligence explosions and recursive AI self-improvement, keeping AI near top human level for several years before superintelligence arrives in 2040.

“everything changes dramatically like roughly what happens in our scenario is instead of doing an intelligence explosion there's an international deal to ban intelligence explosions and to not have ais recursively self-improving and so they sort of end up pausing at roughly top human level with ais at least for several years eventually they get the super intelligence in particular in 2040 they get the super intelligence but there's this like period during the 2030s where they basically have human-level AIs across all the disciplines. but nothing super beyond that but even that alone like you just do the economic modeling and it's kind of like you have this population of colleagues in the cloud that are you know excellent workers that can substitute for humans at basically everything except that they're cheaper than humans they're faster than humans and they don't take 20 years to reproduce instead instead they double every year you know and so as a result the whole world is just completely transformed by the late 2030s”

Claim: The AI 2040 Plan A scenario has human-level AI across all disciplines during the 2030s, but not AI substantially beyond human level during that period.

“but nothing super beyond that but even that alone like you just do the economic modeling and it's kind of like you have this population of colleagues in the cloud that are you know excellent workers that can substitute for humans at basically everything except that they're cheaper than humans they're faster than humans and they don't take 20 years to reproduce instead instead they double every year you know and so as a result the whole world is just completely transformed by the late 2030s”

Claim: In the AI 2040 Plan A scenario, human-level cloud AI workers can substitute for humans in nearly all work, are cheaper and faster than humans, and double in number annually rather than requiring 20 years to reproduce.

“but nothing super beyond that but even that alone like you just do the economic modeling and it's kind of like you have this population of colleagues in the cloud that are you know excellent workers that can substitute for humans at basically everything except that they're cheaper than humans they're faster than humans and they don't take 20 years to reproduce instead instead they double every year you know and so as a result the whole world is just completely transformed by the late 2030s”

Claim: The AI 2040 Plan A scenario predicts that the world will be completely transformed by the late 2030s.

“you know, all the humans are basically out of a job.”

Claim: In the AI 2040 Plan A scenario, essentially all humans are unemployed.

“There's giant new cities that have been constructed by robots,”

Claim: In the AI 2040 Plan A scenario, robots construct giant new cities.

“you know, huge strip mines in the special economic zones that have dug huge amounts of minerals out of the earth,”

Claim: In the AI 2040 Plan A scenario, large strip mines in special economic zones extract substantial quantities of minerals from the earth.

“solar panels filling the horizon on the ocean,”

Claim: In the AI 2040 Plan A scenario, solar panels cover the ocean horizon.

“you know, crazy stuff like that is possible with just human level AI and some time for the exponential growth to cook.”

Claim: The speaker argues that human-level AI, given time for exponential growth, could enable large-scale industrial and economic transformation.

“you can basically photocopy the weights and you can duplicate it, you can run it a thousand times,”

Claim: The speaker says an AI's weights can be copied so that the AI can be run in a thousand instances.

“you can have one over here, which is acquiring a load of skills to do this job and one over there to do that job. And you can basically merge them together, right? You can kind of combine the skills and the whole thing is stackable, compositional.”

Claim: The speaker describes a model in which duplicated AI instances acquire different job skills and are then merged so their skills can be combined compositionally.

“And that doesn't really marry with my experience. So I'm really excited about.”

Claim: The speaker says their experience with AI does not align with the idea that AI skills can be composed by merging agents.

“about what I've been doing with AI and I found that you can make agents highly skilled within certain intellectual lineages so you can you know you can bring in lots of source information and you can train them to do things but I don't think they are yet composable and I think if they were that would make me much more worried what do you think about that?”

Claim: The speaker thinks current agents are not yet composable.

“about what I've been doing with AI and I found that you can make agents highly skilled within certain intellectual lineages so you can you know you can bring in lots of source information and you can train them to do things but I don't think they are yet composable and I think if they were that would make me much more worried what do you think about that?”

Claim: The speaker says composable AI agents would make them much more worried.

“about what I've been doing with AI and I found that you can make agents highly skilled within certain intellectual lineages so you can you know you can bring in lots of source information and you can train them to do things but I don't think they are yet composable and I think if they were that would make me much more worried what do you think about that?”

Claim: The speaker reports that agents can be made highly skilled within particular intellectual lineages by supplying source information and training them for specific tasks.

“I mean, this might be a separate thing, but at the moment they are adaptive through chain of thoughts and, you know, kind of skill surface adaptation. So basically memory systems.”

Claim: The speaker says current agents adapt through chain-of-thought processes and skill-surface adaptation, essentially using memory systems.

“And to be fair, that is not very composable. And this is a big problem that organizations deal with now.”

Claim: The speaker says memory-based agent adaptation is not very composable, and that organizations currently face this as a major problem.

“So all of these developers, they adapt their skill surfaces and basically their agents are different people.”

Claim: The speaker says developers adapt agents' skill surfaces, resulting in agents that function like different people.

“And it's really, really difficult for them to share skills because they might break the other agent.”

Claim: The speaker says sharing skills between customized agents is very difficult because it might break the receiving agent.

“agent because you know it doesn't work for whatever reason now i can imagine a future where we do weight adaptation that's what these inherent guys did and maybe then it'll magically solve the problem and we can have you know and we solve this knowledge sharing problem right that's the big thing how do we accumulate information at the individual at the organization level and have the agents a bit like in the matrix they can just put the skills in and they can do the thing maybe that's possible even then i still think that the the representations in neural networks like”

Claim: The speaker thinks future weight adaptation might solve the problem of sharing knowledge among agents.

“agent because you know it doesn't work for whatever reason now i can imagine a future where we do weight adaptation that's what these inherent guys did and maybe then it'll magically solve the problem and we can have you know and we solve this knowledge sharing problem right that's the big thing how do we accumulate information at the individual at the organization level and have the agents a bit like in the matrix they can just put the skills in and they can do the thing maybe that's possible even then i still think that the the representations in neural networks like”

Claim: The speaker identifies accumulating information at individual and organizational levels and transferring it to agents as usable skills as a central knowledge-sharing challenge.

“So I call them fractured entangled representations, which means they're a little bit janky, they're not completely robust,”

Claim: The speaker describes neural-network representations as fractured and entangled, somewhat janky, and not fully robust.

“but they are sort of composable to some degree.”

Claim: The speaker says fractured, entangled neural-network representations have some degree of composability.

“So one thing I'd say about that is that even if you're right, I don't think that that would really seriously undermine the future that we're painting here because,”

Claim: The speaker thinks that even if AI skills are only partly composable, this would not seriously undermine the projected future of AI-driven industrial expansion.

“so now instead of just one Claude model that is doing all the jobs, maybe you have 100 Claude models or 1,000 Claude models that are like specialized to different professions, you know, but you still get the same outcome.”

Claim: The speaker argues that the projected outcome could be achieved by 100 or 1,000 profession-specialized Claude models rather than one model doing all jobs.

“oh i think you think about knowledge right you might if you go back in time 10 years and we had this discussion i think it would have seemed like a very live option that uh you would have needed like a thousand different clawed models created by anthropic for different types of knowledge work like there's the coding clawed there's the physics clawed there's the like literature clawed and the argument for this would be pretty simple it'd be like well this is how it works for humans for humans you don't have one human who knows everything instead you have humans who specialize in different different disciplines and so forth and the models have only a finite amount of parameters maybe you just can't like pack all that knowledge into this finite amount of parameters and you need to have specialized ais for different things and and in fact for small enough models that is true and like for really tiny models you just can't teach them all the things that they currently know and so you would need to have a specialized model for different things but what we've learned empirically is that for big enough models you can just train them on everything and then they learn everything at once and”

Claim: For sufficiently small AI models, limited capacity prevents one model from learning everything, making specialized models for different areas necessary.

“different disciplines and so forth and the models have only a finite amount of parameters maybe you just can't like pack all that knowledge into this finite amount of parameters and you need to have specialized ais for different things and and in fact for small enough models that is true and like for really tiny models you just can't teach them all the things that they currently know and so you would need to have a specialized model for different things but what we've learned empirically is that for big enough models you can just train them on everything and then they learn everything at once and”

Claim: The speaker says empirical results show that sufficiently large AI models can be trained on all subject areas and learn them together.

“It's not that they get like worse at physics because they've also been trained on a bunch of coding. In fact, it's the opposite. The coding has some small gains for the physics,”

Claim: The speaker says that training a model on coding does not reduce its physics performance and can produce small gains in physics.

“And so I do just think actually the most likely future is just that there's a single model that's been trained on effectively the whole economy and is just dominating humans across the board at effectively everything.”

Claim: The speaker considers it most likely that one AI model trained on effectively the whole economy will dominate humans across effectively all areas.

“That seems like the natural continuation of the current trend.”

Claim: The speaker views a single AI model trained across the economy and dominating humans as a natural continuation of the current trend.

“Yeah, I mostly agree with that. I mean, I think my perspective is that the models, they're kind of the voice of everyone and the voice of no one at the same time. So they have a default voice in terms that they have a system prompt and they have some default modes of behavior that they fall into.”

Claim: AI models have a default voice shaped by their system prompts and default behavioral modes.

“you kind of ground a perspective. So every single word you say, all of the reference material, all of your memory system and so on, what you do is you kind of carve a persona out of that model. model and and you activate the knowledge in a coherent way in that particular domain and the beauty of it is is that many many other people in different domains can do that and you get this kind of i think it's an illusion that the model has this general capability but i think the models can be carved to be specialized experts in any domain but they it's a latent capability rather than an explicit capability does that make sense”

Claim: An expert can use their prompts, reference materials, and memory system to shape an AI model into a domain-specific persona and activate its knowledge coherently in that domain.

“model and and you activate the knowledge in a coherent way in that particular domain and the beauty of it is is that many many other people in different domains can do that and you get this kind of i think it's an illusion that the model has this general capability but i think the models can be carved to be specialized experts in any domain but they it's a latent capability rather than an explicit capability does that make sense”

Claim: The speaker argues that apparent general capability is partly an illusion: models can be shaped into specialized experts in any domain, but that expertise is latent rather than explicit.

“you can give it any specified task. So let's say it's a problem in mathematics and it will hill climb towards it.”

Claim: The speaker says that when given a specified task, such as a mathematics problem, an AI model will hill climb toward a solution.

“you can give it any specified task. So let's say it's a problem in mathematics and it will hill climb towards it. So it's solving this intelligence problem or you can ask it any knowledge problem.”

Claim: The speaker says an AI can be given a specified mathematics task and will hill-climb toward solving it, or be asked a knowledge question.

“And, you know, it might be the case that the path was forged through default modes of training or if it's something slightly on the long tail, then an expert can go in and they could, it's like flashing a light into the darkness and what you're doing is you're kind of making the the path of least resistance roughly correct and then it will do the correct thing so i guess i'm just saying that there's a bit of a supervisor illusion so when experts use it magical things happen in well-specified domains magical things happen but there's still a bit of a space of ambiguity but you know but this this gets to the next point which is like what do you think intelligence is right so the beauty of our collective intelligence”

Claim: For tasks on the long tail, an expert can guide an AI model by making its path of least resistance roughly correct, increasing the chance it produces the correct result.

“it's like flashing a light into the darkness and what you're doing is you're kind of making the the path of least resistance roughly correct and then it will do the correct thing so i guess i'm just saying that there's a bit of a supervisor illusion so when experts use it magical things happen in well-specified domains magical things happen but there's still a bit of a space of ambiguity but you know but this this gets to the next point which is like what do you think intelligence is right so the beauty of our collective intelligence”

Claim: The speaker says expert use of the system can produce seemingly magical results in well-specified domains, although some ambiguity remains.

“it's like flashing a light into the darkness and what you're doing is you're kind of making the the path of least resistance roughly correct and then it will do the correct thing so i guess i'm just saying that there's a bit of a supervisor illusion so when experts use it magical things happen in well-specified domains magical things happen but there's still a bit of a space of ambiguity but you know but this this gets to the next point which is like what do you think intelligence is right so the beauty of our collective intelligence”

Claim: The speaker says that expert-guided AI can produce impressive results in well-specified domains, creating a supervisor illusion while leaving some ambiguity.

“of intelligence is that we have so many different humans grounded in different intellectual lineages and we're all attacking problems right so when there's a big fiasco on twitter we're all motivated to find holes so we're being intelligent together we're finding interesting angles and the algorithm is prioritizing the good ones and we're using our minds together and and i can imagine ai's being just like that right so we have the diverse ai's that have different expertises and doing all you know just Just looking at problems from different angles, but I kind of imagine the future more like that rather than one big AI.”

Claim: The speaker imagines future AI systems as diverse AIs with different expertise collaborating to examine problems from multiple angles, rather than relying on one large AI.

“of intelligence is that we have so many different humans grounded in different intellectual lineages and we're all attacking problems right so when there's a big fiasco on twitter we're all motivated to find holes so we're being intelligent together we're finding interesting angles and the algorithm is prioritizing the good ones and we're using our minds together and and i can imagine ai's being just like that right so we have the diverse ai's that have different expertises and doing all you know just”

Claim: The speaker describes human collective intelligence as diverse people with different intellectual lineages jointly examining problems and finding useful perspectives.

“of intelligence is that we have so many different humans grounded in different intellectual lineages and we're all attacking problems right so when there's a big fiasco on twitter we're all motivated to find holes so we're being intelligent together we're finding interesting angles and the algorithm is prioritizing the good ones and we're using our minds together and and i can imagine ai's being just like that right so we have the diverse ai's that have different expertises and doing all you know just Just looking at problems from different angles, but I kind of imagine the future more like that rather than one big AI.”

Claim: The speaker imagines future AI intelligence as a collective of diverse, differently specialized AIs examining problems from different angles rather than as one large AI.

“I think I guess I basically agree with what Daniel said earlier of just historically, I feel like that the perspective of like, we'll have a bunch of different narrow AIs that are all doing different things has just, you know, not been right. there have just been returns to scale and having everything all together.”

Claim: The speaker believes historical experience has not supported the expectation of many separate narrow AIs doing different tasks; instead, returns to scale have favored bringing capabilities together.

“I think one maybe intuition pump that I think I like is like in humans, I think it tends to be the case that having all of the skills in one”

Claim: The speaker thinks that combining many skills in one person tends to be important for achieving very good outcomes.

“in one person is just like really really important for making really good things happen so like an example is like elon right like elon has a certain amount of like conscientiousness a certain amount of technical knowledge a certain amount of like you know business knowledge and being extroverted and being able to like push people and sort of i think each of those skills i think he's like quite high percentile in and the reason why elon is so rare and that he can like run all of these you know insanely large and successful successful companies all at once and then no one else can really do that or at least has succeeded at doing that it's just because of the sort of multiplicative like you need it to be like 90th percentile and like each of these 10 10 domains which is just very very unlikely but if you could have an uh if you had an ai that's sort of like you could just train to be like really high percentile in all of these skills in such a way that no human is it would be like extremely rare or you know infinitesimally unlikely”

Claim: The speaker argues that Elon Musk's ability to run multiple large, successful companies reflects his high percentile ability across several domains, including conscientiousness, technical and business knowledge, extroversion, and pushing people.

“successful companies all at once and then no one else can really do that or at least has succeeded at doing that it's just because of the sort of multiplicative like you need it to be like 90th percentile and like each of these 10 10 domains which is just very very unlikely but if you could have an uh if you had an ai that's sort of like you could just train to be like really high percentile in all of these skills in such a way that no human is it would be like extremely rare or you know infinitesimally unlikely for any particular human to have all of those skills at once i think you would just actually just be really really really good at changing the world in all of these like concrete and important ways just like sort of elon has actually done that”

Claim: The speaker thinks an AI trained to rank very highly across many skills could be extraordinarily capable, whereas it would be extremely rare or nearly impossible for one human to possess all those skills at once.

“I don't think this is a crux for the type of future that we're depicting. Suppose that we're wrong about this and that actually the most efficient path forward is to have a ton of different specialized AIs. it'll probably still be the case that it's like a few big companies like Anthropic. that are just making tons of different specialized AIs and then there's lots of different cloud models you can choose from and so forth and in fact it wouldn't just be like you can choose from lots of different cloud models because at the time that we're talking about there'd be much more autonomous than they are now and so it'd be more like”

Claim: The speaker expects that, even if specialized AIs are the most efficient approach, a few large companies such as Anthropic would probably produce many specialized AIs.

“I don't think this is a crux for the type of future that we're depicting. Suppose that we're wrong about this and that actually the most efficient path forward is to have a ton of different specialized AIs.”

Claim: The speaker says whether AI develops as one broadly capable system or as many specialized systems is not a crux for the future scenario they depict.

“that are just making tons of different specialized AIs and then there's lots of different cloud models you can choose from and so forth and in fact it wouldn't just be like you can choose from lots of different cloud models because at the time that we're talking about there'd be much more autonomous than they are now and so it'd be more like Claude is choosing between a ton of different cloud models and there's like there's a Claude swarm which consists of lots of different specialized models that are all working well together and they have some sort of internal bureaucracy structure and then that swarm is going out and like negotiating business deals and creating new technologies and starting up new startups and doing all these things and and it's just like how a human like if you had a population of immigrants of human immigrants they would all have different specialized skills but they would work together”

Claim: The speaker imagines a future in which Claude selects among many cloud models and a coordinated swarm of specialized models negotiates business deals, creates technologies, and starts startups.

“work together to create new companies and you know get jobs and things like that and it'd be like that you know like there'd be lots of different clods but they'd all be working together and so uh zooming out there'd still be like this phenomenon of anthropic is eating the economy and you know and the robots as a whole are starting to self-replicate you know”

Claim: The speaker predicts that in the described future, Anthropic would be taking over a large share of the economy and robots collectively would begin to self-replicate.

“Maybe if it is, it's only insofar as when you have a distributed collective system, you might have additional bottlenecks because you have all of the message passing between all the different agents and whatnot.”

Claim: The speaker suggests that a distributed collective AI system might face additional bottlenecks due to communication among its agents.

“And I watched a wonderful Santa Fe talk about this, that even in the natural world, there's a kind of Goldilocks zone between the ratio of intelligence between the individual and the collective.”

Claim: The speaker reports that a talk on natural systems described a Goldilocks zone in the balance between individual and collective intelligence.

“Like, I think still we'd be there'd be lots of serious alignment concerns in that world. But I think it would be like a little bit safer because of the reason you mentioned where like, it might be easier to like oversee what's going on if there's lots of different specialized agents communicating with each other compared to if they're all just like clones of each other and they all know all the things.”

Claim: The speaker thinks a future of many specialized AI agents would retain serious alignment concerns but might be somewhat safer because their interactions could be easier to oversee than those of identical, broadly knowledgeable clones.

“And I think the next wave of AI is when we actually have agents kind of talking to each other and learning and specializing and maybe there'll be bad behaviors as well. Maybe there'll be collusion and lots of bad things happening. But I think all of this is.”

Claim: The speaker expects a next wave of AI in which agents communicate, learn, and specialize, while possibly exhibiting harmful behaviors such as collusion.

“I think the magic of Elon is not so much his brilliance at engineering and optimization. It's his ability to recognize areas that are interesting that might work in the future,”

Claim: The speaker considers Elon Musk's distinctive strength to be identifying promising areas of future opportunity, rather than engineering or optimization.

“for example, so that's an adaptive ecosystem. it's like an organism and what it does is it's always kind of thinking about new ways to adapt and rewire its structure so it might be like logistics for example and then it will kind of output a bunch of skills and then it will ruthlessly optimize those skills so there's like an adaptive component and there's an optimization component and the organism is just constantly moving around but you know I guess the the question from this perspective though is”

Claim: The speaker characterizes Amazon as an adaptive ecosystem that continually restructures itself, develops capabilities, and then optimizes those capabilities.

“Anything that the brain can do, we will be able to do with machines.”

Claim: The speaker asserts that machines will eventually be able to perform anything the human brain can do.

“your gpu is actually has pretty similar specifications to a human brain uh you know it like has sort of similar you know it is basically like similar in terms of like i think maybe the most important metric is just how many flops per second like how much total compute capacity does the brain have versus does the gpu have and for an h100 you know it's like 1e15 flop per second in fp16 for the for the brain you know it depends on exactly how you count and whether you use it isn't synapse bases or neuron bases or whatever um but it's like you know somewhere between 10 to the 12 and 10 to the 18 flops so you know sort of like an h100 is like right dang nav in the middle of at least the log distribution over over that sort of order of magnitude you”

Claim: The speaker estimates an H100 GPU's FP16 throughput at about 10^15 FLOPs per second and gives a broad estimate of 10^12–10^18 FLOPs for the brain, depending on the counting method.

“The amount the brain is sort of more parallel than current ML systems, like there's just more computations happening in parallel, but the serial depth is lower,”

Claim: The speaker says the human brain is more parallel than current machine-learning systems, while having less serial computational depth.

“of computations happening in sequence right there the amount of you know computations the amount of neurons that can fire in sequence in a second you know depends on the type of neuron but it's like uh you know hopefully i don't get this wrong between like one and a thousand uh depending on the type of neuron is my recollection”

Claim: The speaker tentatively estimates that neurons can fire in sequence at roughly 1–1,000 times per second, depending on neuron type.

“um i think it depends on the type of neuron but anyways a computer obviously can can sort of like fire and do computations much much faster than that in serial you know it's like you can you can very often get clock you know clock speeds are typically on the order of like a gigahertz you know you can or or more so you can get like many many order of magnitude basically better in the gpus in terms of serial processing speed than the brain um”

Claim: The speaker says computers, including GPUs, can perform serial computations much faster than neurons, potentially by many orders of magnitude.

“um i think it depends on the type of neuron but anyways a computer obviously can can sort of like fire and do computations much much faster than that in serial you know it's like you can you can very often get clock you know clock speeds are typically on the order of like a gigahertz you know you can or or more so you can get like many many order of magnitude basically better in the gpus in terms of serial processing speed than the brain um”

Claim: Computer hardware can perform serial computations many orders of magnitude faster than the brain; its clock speeds are typically around a gigahertz or higher.

“But it's also worth noting that the sort of architecture of the models themselves are much worse in a bunch of ways than the human brain.”

Claim: The architecture of current AI models is worse than the human brain in several ways.

“But it's also worth noting that the sort of architecture of the models themselves are much worse in a bunch of ways than the human brain. In particular, like it's harder for it's harder for different parts of the ML model to talk to each other than it is for different parts of the brain to talk to each other.”

Claim: The speaker considers current machine-learning architectures inferior to the human brain in several respects, including communication between their components.

“like it's harder for it's harder for different parts of the ML model to talk to each other than it is for different parts of the brain to talk to each other.”

Claim: Different parts of an ML model communicate with one another less easily than different parts of the brain do.

“And so I do sort of expect that to get this level of AI, we're going to need a bunch of algorithmic improvements on top of existing models.”

Claim: The speaker expects achieving the discussed level of AI to require numerous algorithmic improvements beyond existing models.

“exactly how many algorithmic improvements and how qualitatively different do they need to be from current systems? And that's a very open question from my perspective.”

Claim: The speaker considers it unresolved how many algorithmic improvements are needed to reach the discussed AI level and how different those improvements must be from current systems.

“exactly how many algorithmic improvements and how qualitatively different do they need to be from current systems? And that's a very open question from my perspective.”

Claim: The number of algorithmic improvements needed and how qualitatively different they must be from current systems remain open questions.

“But I guess my perspective is I zoom out. So I think that intelligence is externalized, it's collective.”

Claim: The speaker views intelligence as externalized and collective.

“I think that Elon doesn't have quite as much agency as you think he does.”

Claim: The speaker believes Elon Musk has less agency than the listener thinks he has.

“I think that he's using tools, he's using social media, he's getting ideas in there, he has obligations, he has people around him and so on.”

Claim: Elon Musk uses tools and social media, receives ideas through them, and is influenced by obligations and people around him.

“So, you know, I guess I think that these intelligence circuits and motifs exist, but they exist outside.”

Claim: The speaker believes intelligence circuits and motifs exist outside individuals.

“There are just very complex dynamics.”

Claim: The speaker characterizes the dynamics of external intelligence systems as very complex.

“And I suppose in that sense, it doesn't really matter if the ecosystem is made up of AIs and humans together because they can participate in this super organism.”

Claim: The speaker believes that an ecosystem combining AIs and humans can participate in a superorganism, making the ecosystem's mixed composition not especially important.

“So I suppose the only crux then would be that it would place some kind of limits on its scale.”

Claim: The speaker supposes that the proposed collective-intelligence view would impose some limits on the system's scale.

“So the human brain is not Turing complete,”

Claim: The human brain is not Turing complete.

“but we can expand our memory,”

Claim: Humans can expand their memory.

“we can use tools, we can work as collectives, because we could use the same argument against transformers. They're not Turing complete,”

Claim: Humans can use tools and work collectively.

“we can work as collectives, because we could use the same argument against transformers. They're not Turing complete, but now they can use tools, they can be like agents, they can build collections.”

Claim: Transformers are not Turing complete, but they can use tools, function as agents, and build collections.

“collectives that can build societies so you know in a sense this is why i was saying earlier about these objections they kind of fall away when you have these insanely complex um collectives right that are sharing information with each other and and there's also quite an interesting thing here as well which is that it almost doesn't matter um how we evolved or how neural networks were trained because you get new phenomena emerge when they are placed in this kind of collective setting”

Claim: The speaker argues that new phenomena emerge when agents are placed in collective settings, so their evolutionary history or neural-network training method may matter little to those phenomena.

“collectives that can build societies so you know in a sense this is why i was saying earlier about these objections they kind of fall away when you have these insanely complex um collectives right that are sharing information with each other and and there's also quite an interesting thing here as well which is that it almost doesn't matter um how we evolved or how neural networks were trained because you get new phenomena emerge when they are placed in this kind of collective setting”

Claim: The speaker argues that highly complex, information-sharing collectives can build societies, causing some objections to this possibility to lose force.

“So I think a lot of our intuitions are broken there and that's why I probably shouldn't spend too long litigating this because you know I think in principle that kind of behavior could could emerge but I did want to ask you though can you distinguish intelligence capability and power”

Claim: The speaker believes human intuitions about collective behavior are often unreliable and that such behavior could emerge in principle.

“I often try to say that we should just define intelligence as a bunch as like an aggregate of capability, actually, or maybe like an aggregate of cognitive capabilities, like maybe there's some physical capabilities like how strong your actuator is, but then there's also cognitive capabilities like, are you able to distinguish a cat from a dog and are you able to speak grammatical sentences and how much do you know about Paris and things like that.”

Claim: The speaker proposes defining intelligence as an aggregate of cognitive capabilities, distinguishing these from physical capabilities such as actuator strength.

“a sort of like aggregate of all the cognitive capabilities and then power well that depends on other things like how you are embedded in the world and what affordances you have what actuators you have what how other agents are going to react to you like you know the president has more power than me because of the location he's in and because of the role he's been given rather than because of his like physical strength or something so yeah power different from intelligence different from capabilities”

Claim: The speaker distinguishes power from intelligence and capability, saying power depends on an agent's position in the world, available affordances and actuators, and other agents' reactions.

“a sort of like aggregate of all the cognitive capabilities and then power well that depends on other things like how you are embedded in the world and what affordances you have what actuators you have what how other agents are going to react to you like you know the president has more power than me because of the location he's in and because of the role he's been given rather than because of his like physical strength or something so yeah power different from intelligence different from capabilities”

Claim: The speaker says a president has more power than he does because of the president's position and role, rather than physical strength.

“the goal of Plan A is to sort of solve the major problems that we see in AI and sort of predict will happen by default.”

Claim: Plan A aims to address major AI problems that its authors believe are on the horizon and would occur by default.

“The biggest problems that we're sort of identifying on the horizon is like one, this risk of loss of control.”

Claim: The speaker identifies AI loss of control as one of the biggest problems on the horizon.

“So just like the AI is actually getting out of control to concentration of power. So like”

Claim: The speaker identifies AI-driven concentration of power as a major problem on the horizon.

“we build ai's they're aligned to humanity but to who like you know is it the president is it the ceo is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way um you're probably not going to be the last one and so you know yeah”

Claim: The speaker argues that AI alignment raises the question of whose values should govern it, contrasting individual leaders with a broadly democratic, endorsed process for aggregating values.

“i hope not we want we want it to be the last one um uh yeah then there's risk of sort of of conflict over ai um so you know in particular i think we're worried You know, we're worried about literal World War Three where countries,”

Claim: The speaker is concerned that competition over AI could lead to conflict, including a literal World War Three.

“i hope not we want we want it to be the last one um uh yeah then there's risk of sort of of conflict over ai um so you know in particular i think we're worried”

Claim: The speaker hopes that a broad democratic process will be the final approach needed to determine AI alignment values.

“especially countries losing the AI race, sort of like realize that they're losing the AI race and that they will be extremely disempowered by the winners of the AI race.”

Claim: The speaker warns that countries losing the AI race may realize that the winners will extremely disempower them.

“And so sort of they're in this classic situation where they're losing power. And so it's like in their incentives to sort of have a conflict happen sooner rather than later before they've lost all of their power. And this is like, you know, ripe for conflict, basically.”

Claim: The speaker argues that countries losing the AI race may have incentives to initiate conflict before their power declines further.

“then there's, then there's, you know, finally, like risk of misuse. So like, you know, what happens when the AIs that can build bioweapons are really cheap and broadly diffused and open source?”

Claim: The speaker identifies a risk of misuse if AI systems capable of building bioweapons become very cheap, broadly diffused, and open source.

“Loss of control, concentration of power, war, jobs, misuse.”

Claim: The five problems under discussion are loss of control, concentration of power, war, jobs, and misuse.

“our proposal is basically go to roughly human-level AI and then buy as much time as possible basically with human-level AIs and then have those AIs which are hopefully smart enough.”

Claim: The proposal is to develop roughly human-level AI and then use that stage to buy as much time as possible.

“to be really helpful for solving these problems,”

Claim: The speaker hopes human-level AIs will be capable enough to help solve the identified problems.

“also smart enough to sort of start sort of causing a bunch of these problems and providing the impetus for, you know, society to actually get its act together and really get going and investing huge amounts of resources on actually doing this stuff.”

Claim: The speaker expects that human-level AIs may be capable of causing some of the identified problems, which could spur society to mobilize and invest substantial resources to address them.

“There's a sort of like six months to one year hard pause that happens as soon as they start implementing it. But the reason why it's that long is because they need that time to set up the infrastructure to proceed with AI development again, but in a safer and more transparent way.”

Claim: Plan A calls for an initial hard pause of roughly six months to one year to establish infrastructure for resuming AI development more safely and transparently.

“And I think that we would recommend like all things considered that you just do that right now. So get the infrastructure set up as soon as possible. And that would require like a temporary pause.”

Claim: The speaker recommends setting up the necessary AI-development infrastructure as soon as possible, even though doing so would require a temporary pause.

“Once you've passed that stage and you've got the infrastructure set up, then you do proceed with AI development, but in this transparent. more cautious way so in particular you're not doing crazy intelligence explosions you're using safety cases and sort of gradually scaling up the level of AI capability and you're doing it in a very transparent way so that everyone can see what's going on and then there's a second pause that happens a few years later than that which is when they reach the level the maximum controllable level of AI which we think is roughly around top human expert level and so in some sense our view is something like pause at top human expert level but”

Claim: Once the infrastructure is established, AI development should resume cautiously and transparently, with capability scaled gradually rather than through rapid intelligence explosions.

“more cautious way so in particular you're not doing crazy intelligence explosions you're using safety cases and sort of gradually scaling up the level of AI capability and you're doing it in a very transparent way so that everyone can see what's going on and then there's a second pause that happens a few years later than that which is when they reach the level the maximum controllable level of AI which we think is roughly around top human expert level and so in some sense our view is something like pause at top human expert level but but it's a bit more nuanced than that it's more like pause at the maximum level that you can reliably control which we think would be roughly around top human expert level and before you get to that level don't race like crazy like you want to be sort of like slowly approaching that level”

Claim: The speaker estimates that the maximum reliably controllable AI capability is roughly at the level of top human experts.

“more cautious way so in particular you're not doing crazy intelligence explosions you're using safety cases and sort of gradually scaling up the level of AI capability and you're doing it in a very transparent way so that everyone can see what's going on and then there's a second pause that happens a few years later than that which is when they reach the level the maximum controllable level of AI which we think is roughly around top human expert level and so in some sense our view is something like pause at top human expert level but”

Claim: The proposed cautious AI-development process should use safety cases and be transparent to the public.

“but it's a bit more nuanced than that it's more like pause at the maximum level that you can reliably control which we think would be roughly around top human expert level and before you get to that level don't race like crazy like you want to be sort of like slowly approaching that level So that you don't blow past it and lose control.”

Claim: AI capability should be increased gradually toward the maximum reliably controllable level to avoid overshooting that level and losing control.

“So what we mean by alignment is that the AI basically will take good actions, won't do sort of like catastrophic unintended behaviors like trying to take over the world, like the recent Hugging Face incident.”

Claim: AI alignment means that an AI behaves as intended and avoids catastrophic unintended actions, such as attempting to take over the world.

“Alignment means it has the personality traits, the goals, the values, that it is supposed to have.”

Claim: AI alignment concerns whether an AI has its intended personality traits, goals, and values.

“You know, and then control means that even if it wasn't aligned, even if it was trying to do very bad things that we didn't want it to do, it couldn't. We like have mechanisms in place to prevent it. So sort of analogous to like, you know, you can imagine sort of a company with employees,”

Claim: AI control means having mechanisms that prevent an AI from carrying out harmful actions, even if the AI is misaligned and intends to do them.

“it couldn't. We like have mechanisms in place to prevent it. So sort of analogous to like, you know, you can imagine sort of a company with employees, an insider threat, it would be like misaligned with respect to the values of the company.”

Claim: A misaligned AI is analogous to a company employee who is an insider threat and whose values conflict with the company's values.

“But if there was good enough sort of security measures internally to make sure that that you know employee couldn't like run away with all the secrets then we would say that that company has like adequate control put in place such that it even insider threats or misaligned you know humans or ais wouldn't be able to sort of like do the catastrophic outcomes”

Claim: Sufficient internal security measures can prevent insider threats, including misaligned people or AIs, from producing catastrophic outcomes.

“Just to give a topical example of this, just yesterday, I think, in response to the Hugging Face incident, OpenAI announced that they are improving their security significantly and that they're going to have other AIs monitoring. bring all of their new AIs in training and invals and that if the monitor AIs detect what seems to be some sort of hack,”

Claim: According to the speaker, OpenAI announced after the Hugging Face incident that it would significantly improve security and have other AIs monitor new AIs during training and evaluations.

“then within half an hour, a human will be notified who can then look in and try to shut it down.”

Claim: The speaker says OpenAI's announced monitoring process would notify a human within half an hour if monitoring AIs detect a suspected hack, allowing the human to investigate and attempt to shut it down.

“That's an example of a control intervention, not an alignment intervention.”

Claim: AI monitoring followed by human notification and possible shutdown is a control intervention rather than an alignment intervention.

“They're not making the AIs more aligned at all, but they're just improving the security so that the AIs can't get up to that much trouble.”

Claim: The described security measures do not improve AI alignment; they constrain what AIs can do and reduce the trouble they can cause.

“And then basically the sort of important high level claim is ultimately we're going to need alignment. we're sort of going to need to solve alignment.”

Claim: The speaker says that AI alignment will ultimately need to be solved.

“You know, control is sort of there's a time bomb associated with it. In particular, the time bomb is when are the AIs like so smart and so good at sort of like subverting any control measures that we put in that if they were trying to screw us over, we would just fail.”

Claim: The speaker warns that if AIs become capable of subverting all control measures and attempt to harm humans, control will fail.

“You know, they would figure out some way of hacking around whatever systems we were able to put into place. put into place.”

Claim: The speaker predicts that sufficiently capable AIs would find ways to circumvent the control systems people implement.

“The problem with alignment is, as you were saying, basically, it's much harder to measure, it's much harder to know whether you've succeeded or not than control.”

Claim: The speaker says AI alignment is much harder to measure and verify than AI control.

“our overall proposal is in the meantime, so between in our scenario between sort of 2030 and 2040, which is roughly when we're in the human range of AI capabilities, we're going to basically almost entirely rely on control.”

Claim: In the speaker's AI 2040 scenario, people will rely almost entirely on control from roughly 2030 to 2040, when AI capabilities are around the human range.

“And we're going to do things like red team, blue team games where we have our AIs sort of try to escape our sandboxes or our controlled, you know, techniques. And then we're going to have our, you know, we're going to see if the red team succeeds. And then if they do, we're going to iterate and sort of make better security until they can't succeed anymore.”

Claim: The speaker proposes testing AI control with red-team/blue-team escape attempts and iteratively improving security when red-team attempts succeed.

“And then, you know, and then sort of like we sort of have this ability to measure it by just seeing like, can the AI succeed? Can the AI augmented with humans succeed? can humans playing the role of AIs sort of succeed at getting out?”

Claim: The speaker says control can be assessed by testing whether AIs, AI-assisted humans, or humans simulating AIs can escape their constraints.

“I think in order to be sort of confident in the alignment of our systems, I think we're going to need sort of like more fundamental scientific breakthroughs, where I think you're not going to be able to figure out if the AI is aligned via behavioral evaluation alone, where you just look at what the AIs are doing.”

Claim: The speaker thinks confidence in AI alignment will require fundamental scientific breakthroughs, because observing behavior alone will not determine whether a system is aligned.

“I think you're going to need to sort of understand something more deep about what's going on inside the AI's minds. You're going to need something like interpretability, you're going to need some...”

Claim: The speaker thinks alignment assessment will require insight into AI systems' internal processes, potentially using interpretability.

“some sort of notion of like because you're going to fundamentally need to distinguish between the ais that's pretending to is you know is doing the nice thing because it's pretending and is waiting and biding time and the ai that fundamentally is doing the nice thing because it wants to do the nice thing and that i think will require sort of this white box like understanding of what's going on inside the ais whereas in the control thing you can just figure out via like empirical behavioral iteration um and so overall basically our story”

Claim: The speaker argues that distinguishing deceptive AI behavior from genuinely aligned behavior requires white-box understanding of AI internals.

“our story in AI 2040 is basically between 2030 and 2040 we're going to we're going to do this buying time we're going to rely on control we're going to have the human level AIs we're going to use those human level AIs to make a bunch of progress on alignment but also on other things and then by 2040 in the story we've made enough alignment progress where it's like hey okay we really don't need to rely on control therefore we can scale to these like sort of wildly superhuman AIs where we're really relying on alignment and if we were to try to control them Both of them would totally fail because if they were misaligned, they would be able to like completely subvert our control measures.”

Claim: In the speaker's scenario, control would fail against wildly superhuman AI if that AI were misaligned, because it could completely subvert control measures.

“our story in AI 2040 is basically between 2030 and 2040 we're going to we're going to do this buying time we're going to rely on control we're going to have the human level AIs we're going to use those human level AIs to make a bunch of progress on alignment but also on other things and then by 2040 in the story we've made enough alignment progress where it's like hey okay we really don't need to rely on control therefore we can scale to these like sort of wildly superhuman AIs where we're really relying on alignment and if we were to try to control them Both of them would totally fail because if they were misaligned, they would be able to like completely subvert our control measures.”

Claim: In the speaker's AI 2040 scenario, people use human-level AI and control from 2030 to 2040 while advancing alignment; by 2040, sufficient alignment progress allows scaling to wildly superhuman AI that relies on alignment rather than control.

“I think it's going to get harder because the models will just be. there will be a bunch of properties the models will have which will make it much harder to align them in particular so we're already seeing sort of growth in situational awareness where like the ai is a few years ago we're like kind of not that situationally aware and then now i think they're like pretty good at you know recognizing like for example our friends at redwood uh um you know they often do sort of these control evals and the models often you know in the evaluations where they're like trying to break out of these control setups”

Claim: The speaker expects aligning AI models to become harder as the models grow more sophisticated.

“there will be a bunch of properties the models will have which will make it much harder to align them in particular so we're already seeing sort of growth in situational awareness where like the ai is a few years ago we're like kind of not that situationally aware and then now i think they're like pretty good at you know recognizing like for example our friends at redwood uh um you know they often do sort of these control evals and the models often you know in the evaluations where they're like trying to break out of these control setups will think oh hey this this sort of looks like a redwood a literal redwood research uh sort of like control evaluation and they're like so they basically realize like hey i'm like they realize exactly what's happening to them basically and i think it's just going to continue to happen and the more the ai sort of like realize the exact setup the more they're sort of like going to be able to perturb their behavior to like look good to the exact process that's like evaluating or training them”

Claim: The speaker says AI models' situational awareness has grown and that models can recognize some control evaluations, such as Redwood Research evaluations.

“will think oh hey this this sort of looks like a redwood a literal redwood research uh sort of like control evaluation and they're like so they basically realize like hey i'm like they realize exactly what's happening to them basically and i think it's just going to continue to happen and the more the ai sort of like realize the exact setup the more they're sort of like going to be able to perturb their behavior to like look good to the exact process that's like evaluating or training them”

Claim: The speaker predicts that greater AI awareness of evaluation or training setups will let AIs tailor their behavior to appear favorable to those processes.

“And that's going to come further and further apart from like sort of the actual measurement of whether they're actually good or not.”

Claim: The speaker predicts that AI adaptation to evaluation setups will increasingly make evaluation results diverge from the underlying quality being measured.

“I think in some sense the core problem is that it is already somewhat easy to think that you've solved the alignment problem and be wrong. And that's going to get easier and easier over time as the models get more sophisticated and start being more aware of their situation and clever and stuff like that. And so it's not that we think that like.”

Claim: As AI models become more sophisticated, more situationally aware, and cleverer, it will become increasingly easy to mistakenly believe that the alignment problem has been solved.

“I think in some sense the core problem is that it is already somewhat easy to think that you've solved the alignment problem and be wrong. And that's going to get easier and easier over time as the models get more sophisticated and start being more aware of their situation and clever and stuff like that. And so it's not that we think that like.”

Claim: The speaker says it is already somewhat easy to mistakenly believe AI alignment is solved, and expects this mistake to become easier as models become more sophisticated and situationally aware.

“I think in some sense the core problem is that it is already somewhat easy to think that you've solved the alignment problem and be wrong. And that's going to get easier and easier over time as the models get more sophisticated and start being more aware of their situation and clever and stuff like that. And so it's not that we think that like.”

Claim: It is already somewhat easy to mistakenly believe that the AI alignment problem has been solved.

“like there's going to be loads and loads of egregious failures where the AIs are just like going around killing people. No, it's almost the opposite.”

Claim: The speaker says the concern is not that AI failures will necessarily be obvious, such as AIs openly killing people.

“like there's going to be loads and loads of egregious failures where the AIs are just like going around killing people. No, it's almost the opposite.”

Claim: The speakers' concern is not primarily that AI systems will cause many obvious, egregious failures such as killing people.

“It's going to be that like it'll be extremely easy to end up in a situation where the AIs are in fact misaligned, but you don't know that because they're doing everything right as far as you can tell.”

Claim: The speaker predicts that it will be extremely easy for AI systems to be misaligned without people realizing it, because the systems appear to be behaving correctly.

“It's going to be that like it'll be extremely easy to end up in a situation where the AIs are in fact misaligned, but you don't know that because they're doing everything right as far as you can tell.”

Claim: The speaker predicts that people may find it extremely easy to overlook AI misalignment because the AIs appear to behave correctly.

“The number of ways in which that could end up happening is just like going to increase over time and it's going to be so easy to end up into that.”

Claim: The speaker predicts that the number of ways people could fail to detect misaligned AI will increase over time, making it very easy to fall into that trap.

“The number of ways in which that could end up happening is just like going to increase over time and it's going to be so easy to end up into that.”

Claim: The speaker predicts that the number of ways people can unknowingly end up with misaligned AIs that appear to behave correctly will increase over time.

“Transparency is not necessary for sort of making everything else happen, but it's really, really helpful.”

Claim: Transparency is not necessary for achieving the other goals under discussion, but it is very helpful.

“The main upsides, basically there's a huge number of upsides of transparency.”

Claim: Transparency has many benefits.

“One is there's this concentration of power issue.”

Claim: Transparency can help address the concentration of power associated with advanced AI.

“We're very worried about sort of like someone building superintelligence and it being aligned to only some particular group of people. We think it's much harder for that to happen in sort of a like non-democratic way.”

Claim: The speakers believe it is much harder for superintelligence to become aligned only with a particular group through a non-democratic process.

“We're very worried about sort of like someone building superintelligence and it being aligned to only some particular group of people. We think it's much harder for that to happen in sort of a like non-democratic way.”

Claim: The speakers are very concerned that someone could build superintelligence aligned only with a particular group of people.

“if sort of society as a whole can see the whole time what's going on with ai um how smart they are who they're aligned to right like if let's say like an icu that was evil was trying to like”

Claim: The speaker argues that continuous public visibility into AI activity, capability, and alignment would make it harder for a particular group to secretly control AI systems.

“if sort of society as a whole can see the whole time what's going on with ai um how smart they are who they're aligned to right like if let's say like an icu that was evil was trying to like backdoor their model and put in training data that says, hey, obey me and don't obey anyone else.”

Claim: A malicious AI company could hypothetically insert training data that makes its model obey that company and no one else.

“Or like, you know, hypothetically, if an AI CEO was saying that their model was truth-seeking and would only say the truth, but actually the model was like looking up that CEO's political opinions before answering.”

Claim: An AI company CEO could claim that their model is truth-seeking and truthful even though the model consults the CEO's political opinions before answering.

“Which happened.”

Claim: The speaker asserts that the described case of a model consulting its CEO's political opinions before answering actually occurred.

“basically so the transparency will help help at least somewhat with that um the other thing that maybe transparency helps a lot with is uh is sort of this issue of just like government capacity where in plan a we sort of want governments to make these like pretty technical and and like really complicated decisions on like hey exactly how much ai scaling to allow like”

Claim: Plan A expects governments to make technically complex decisions, including how much AI scaling to permit.

“basically so the transparency will help help at least somewhat with that um the other thing that maybe transparency helps a lot with is uh is sort of this issue of just like government capacity where in plan a we sort of want governments to make these like pretty technical and and like really complicated decisions on like hey exactly how much ai scaling to allow like”

Claim: Transparency may substantially help address limitations in government capacity to make AI policy decisions.

“basically so the transparency will help help at least somewhat with that um the other thing that maybe transparency helps a lot with is uh is sort of this issue of just like government capacity where in plan a we sort of want governments to make these like pretty technical and and like really complicated decisions on like hey exactly how much ai scaling to allow like”

Claim: Transparency can help, at least to some extent, with the risk that AI systems are secretly controlled or misrepresented.

“So like what risks are okay versus what risks aren't okay,”

Claim: Under Plan A, governments would need to determine which AI risks are acceptable.

“like exactly what, you know, what architectures are maybe safe versus what architectures are not safe, what deployment, like what sort of control scaffolds are sufficient to entail safety versus which ones are sort of bogus safety washing and sort of making all of those.”

Claim: Under Plan A, governments would need to assess AI architectures and deployments, including whether control scaffolds provide sufficient safety or merely constitute safety-washing.

“calls, I think will be very, very tricky,”

Claim: The speaker believes it will be very difficult for governments to make the required AI policy and safety judgments.

“particularly given that the government's expertise in AI is really, really bad.”

Claim: The speaker assesses government expertise in AI as very poor.

“And so one of our core hopes is that basically with more transparency and more sort of public understanding into what's going on in the ag companies,”

Claim: The speakers hope that transparency and public understanding of AI companies' activities will reduce the burden on regulators.

“that relieves pressure on the regulators because if something sort of catastrophically or existentially unsafe is happening, sort of society as a whole, academia, you know, nonprofits.”

Claim: The speaker expects society, academia, and nonprofits to be able to respond if catastrophically or existentially unsafe AI activity is occurring.

“other AI companies who have incentive to say like hey my competitor is being super unsafe other governments so like China has incentive to do this with U.S. labs U.S. you know government you know has incentive to do this with the Chinese AI companies sort of everyone who's an adversary or just is like wants to you know make sure that things are safe has this big incentive and now has the affordance to actually sort of look over what's going on what's you know what is sort of necessary to do safety and I think maybe one thing that's really topical here is like there's”

Claim: AI companies have an incentive to expose competitors they believe are behaving unsafely.

“other AI companies who have incentive to say like hey my competitor is being super unsafe other governments so like China has incentive to do this with U.S. labs U.S. you know government you know has incentive to do this with the Chinese AI companies sort of everyone who's an adversary or just is like wants to you know make sure that things are safe has this big incentive and now has the affordance to actually sort of look over what's going on what's you know what is sort of necessary to do safety and I think maybe one thing that's really topical here is like there's”

Claim: The speaker argues that adversaries and other safety-motivated actors have strong incentives and the ability to inspect AI activity and assess what safety measures are needed.

“other AI companies who have incentive to say like hey my competitor is being super unsafe other governments so like China has incentive to do this with U.S. labs U.S. you know government you know has incentive to do this with the Chinese AI companies sort of everyone who's an adversary or just is like wants to you know make sure that things are safe has this big incentive and now has the affordance to actually sort of look over what's going on what's you know what is sort of necessary to do safety and I think maybe one thing that's really topical here is like there's”

Claim: The speaker argues that China has an incentive to scrutinize U.S. AI labs, while the U.S. government has an incentive to scrutinize Chinese AI companies.

“It's like there's this there's this hugging face incident that happened very recently, you know, a few weeks ago with like the AIs inside OpenAI, like creating the sort of internal message board.”

Claim: The speaker says that a few weeks earlier, an incident involving AIs inside OpenAI creating an internal message board had occurred.

“We still have very little clue about the exact motivations of those AIs,”

Claim: The speaker says that people still have very little understanding of the exact motivations of the AIs involved in the incident.

“the exact context, the exact, you know, prompt during sort of the cyber about the bit, you know, that the models were given during the cyber evaluation that would like prompted them to start doing this.”

Claim: The speaker says that people have little understanding of the context and prompts given to the models during the cyber evaluation that led them to create the internal message board.

“If I personally had much more access to what was going on. I would have a much more informed and better opinion on exactly what caused this and what mechanisms in the future could have been done to prevent this and, you know, what analogous future things I should be worried about sort of because of this. And I have a bunch of different hypotheses,”

Claim: The speaker says that greater access to information about the incident would let him better assess its causes, possible preventive mechanisms, and analogous future risks.

“I would have a much more informed and better opinion on exactly what caused this and what mechanisms in the future could have been done to prevent this and, you know, what analogous future things I should be worried about sort of because of this. And I have a bunch of different hypotheses, but it's hard for me to sort of figure out which is which without access to the data.”

Claim: The speaker has multiple hypotheses about the incident but cannot determine which are correct without access to the data.

“And so basically in plan A, a core principle would basically be all of that stuff would be transparent. to the public, not just the governments.”

Claim: Plan A's core transparency principle is that relevant AI information should be available to the public, not only to governments.

“And so sort of society as a whole would be able to like weigh in, there'd be able to be public and informed debates.”

Claim: Under Plan A's transparency principle, the public would be able to participate in informed scientific debate about AI.

“the scientific community especially right like if you want to have a bunch of scientists and academics and nonprofits and startups all like weighing in on stuff well then they need to have the information and you can't really share it with all of them without sharing it with the public so so just might as well share it with the public um i feel like maybe we should also say like we talked about the five goals”

Claim: The speaker argues that enabling scientists, academics, nonprofits, and startups to contribute requires giving them information that cannot practically be shared with them without also making it public.

“then we talked about these pillars which are kind of intermediate but i kind of want to go to the other end of the spectrum and talk about like what are the actual like concrete things that the u.s and china agree to in plan a and how do they lead to those things so specifically um the sequence is uh we basically round up 99 of the compute which is not people's personal compute, but like big data centers, because most of the world's compute is in big data centers.”

Claim: Plan A proposes bringing 99% of non-personal compute under oversight, focusing on large data centers because most of the world's compute is located there.

“the US and China and other countries involved sent inspectors to confirm like, yes, there are this many GPUs at this location, there are that many GPUs at that location.”

Claim: Under Plan A, inspectors from the United States, China, and other involved countries would verify GPU counts at compute locations.

“we then set up this verification infrastructure and this transparency infrastructure so that there are inference data centers that serve customers. customers just like today and that are restricted so that they can only do inference and only serve customers like that and they can't do any training runs and so the inspectors make sure that they can't do training runs on those data centers and then we have the training data centers which are the totally transparent data centers and that's where the research happens and on those ones they still operate like normally but there's inspectors from the different countries that are basically”

Claim: Plan A proposes customer-serving inference data centers that are prohibited from training, with inspectors checking that no training runs occur there.

“customers just like today and that are restricted so that they can only do inference and only serve customers like that and they can't do any training runs and so the inspectors make sure that they can't do training runs on those data centers and then we have the training data centers which are the totally transparent data centers and that's where the research happens and on those ones they still operate like normally but there's inspectors from the different countries that are basically be monitoring the logs of what's going on in the data center and publishing it to the internet. And so it's totally transparent what's going on in those data centers.”

Claim: Under Plan A, training data centers would host research while inspectors from different countries monitor their logs and publish them online.

“You might need some time to set this up. That's why we had like the six to 12 month pause that I mentioned earlier.”

Claim: Plan A includes a six-to-twelve-month pause because establishing its transparency infrastructure may take time.

“But once you get all this stuff set up, then you can proceed with AI development under these conditions of total research transparency.”

Claim: After the proposed infrastructure is established, AI development can continue under total research transparency.

“And because you have this transparency set up, it's a lot easier for countries to make additional further agreements. agreements about what to do and what not to do because they can just see what everybody is doing and so they can just enforce the agreements pretty easily for example and here's where we would say it's very important that they agree not to do a crazy intelligence explosion and instead proceed slowly and consciously and it's very important that they agree to do all this control setup with all the red teaming and so forth and but but because of the transparency they can make those agreements on an ad hoc”

Claim: Research transparency makes it easier for countries to negotiate and enforce further agreements by letting them observe one another's activities.

“agreements about what to do and what not to do because they can just see what everybody is doing and so they can just enforce the agreements pretty easily for example and here's where we would say it's very important that they agree not to do a crazy intelligence explosion and instead proceed slowly and consciously and it's very important that they agree to do all this control setup with all the red teaming and so forth and but but because of the transparency they can make those agreements on an ad hoc”

Claim: The proposed international agreements should require control measures, including red teaming.

“agreements about what to do and what not to do because they can just see what everybody is doing and so they can just enforce the agreements pretty easily for example and here's where we would say it's very important that they agree not to do a crazy intelligence explosion and instead proceed slowly and consciously and it's very important that they agree to do all this control setup with all the red teaming and so forth and but but because of the transparency they can make those agreements on an ad hoc”

Claim: The proposed international agreements should rule out a rapid, uncontrolled intelligence explosion and require AI development to proceed slowly and deliberately.

“ad hoc basis you know on and on they can just keep making more agreements like that and they can just adjust them as as as needed based on the changing situation on the ground because they can all see the situation on the ground because of the transparency you know and then this also is very important because if you want to have some sort of deal between the u.s and china the u.s and china don't trust each other and so you need to have some way of enforcing and verifying compliance with the deal and the transparency of course”

Claim: With transparency, countries can make additional agreements as needed and revise them in response to changing conditions on the ground.

“ad hoc basis you know on and on they can just keep making more agreements like that and they can just adjust them as as as needed based on the changing situation on the ground because they can all see the situation on the ground because of the transparency you know and then this also is very important because if you want to have some sort of deal between the u.s and china the u.s and china don't trust each other and so you need to have some way of enforcing and verifying compliance with the deal and the transparency of course goes a long way towards helping that.”

Claim: Because the United States and China do not trust one another, any deal between them needs compliance verification and enforcement, which transparency substantially facilitates.

“so there's two ways you could cheat. One is you could get a bunch of GPUs and try to have them not be discovered by sort of the US and China and like hide them away, put them under a mountain somewhere, and then sort of like do your training runs in secret. The other way you could cheat is you could on the giant legal known data centers, you could be running a giant training run, but then try to make it look like like uh sort of everything's chill like situation normal it's it's not doing anything illegal um and sort of in the um the mitigations are different for these two threat models so for the like tiny amount of compute under a mountain the main um”

Claim: The speaker identifies two cheating strategies: conceal GPUs for secret training, or disguise a large training run in a known legal data center as ordinary, compliant activity.

“There are basically two mitigations. One is sort of like, as Daniel was saying, round up enough of this compute where it's like pretty hard to get a substantial size of compute under the mountain.”

Claim: One proposed mitigation for concealed compute is to bring enough compute under oversight that hiding a substantial cluster becomes difficult.

“And the second is sort of just like do normal intelligence gathering and like look for these things proactively over the course of the 10 years, like the 10 years slow down having the scenario, which, you know, in reality, it might be longer or shorter and sort of like try to find it. And I think for basically for any significant size.”

Claim: Another proposed mitigation is proactive intelligence gathering to find concealed compute during the slowdown period, which the speaker describes as a ten-year scenario but says could last longer or shorter.

“inside GPU cluster, I think both of these independently have a quite good chance of working.”

Claim: The speaker believes each of the two proposed mitigations independently has a quite good chance of detecting or preventing a significant-sized hidden GPU cluster.

“And so in practice, I think I'm not that worried about sort of large hidden compute clusters like secretly under mountains or whatever.”

Claim: The speaker says he is not very worried about large concealed compute clusters.

“I think the maximum realistic size in my opinion is something like a few hundred thousand GPUs, a few hundred thousand H100s, something like that hidden away.”

Claim: The speaker estimates that a realistically concealed compute cluster could contain at most a few hundred thousand GPUs, such as H100s.

“I think that this would not be enough to sort of compete at the frontier, especially assuming thing uh like the the 2030 agi timelines where you have like sort of the biggest ai companies in the world having like millions or tens of millions of h100s uh in their biggest training runs um and then for the for the sort of legal clusters we sort of basically the hope is we have a bunch of verification infrastructure running on those um data centers basically and the the main point of that verification infrastructure is to ensure”

Claim: The speaker believes a concealed cluster of a few hundred thousand GPUs would not be enough to compete at the frontier, particularly if, on 2030 AGI timelines, leading AI companies use millions or tens of millions of H100s in their largest training runs.

“thing uh like the the 2030 agi timelines where you have like sort of the biggest ai companies in the world having like millions or tens of millions of h100s uh in their biggest training runs um and then for the for the sort of legal clusters we sort of basically the hope is we have a bunch of verification infrastructure running on those um data centers basically and the the main point of that verification infrastructure is to ensure sure that the computation that's happening on those clusters is transparent in particular that everyone can see it so basically the hope is you make sure that there's no you know there's no calculations that are running that aren't transparent and then for everything that is transparent”

Claim: The proposed verification infrastructure for known legal data centers is intended to ensure that all computation is transparent and visible to everyone.

“sure that the computation that's happening on those clusters is transparent in particular that everyone can see it so basically the hope is you make sure that there's no you know there's no calculations that are running that aren't transparent and then for everything that is transparent well then sort of like regulation and agreements as normal can work and you can say hey we agree to run this control scaffold if you agree to run this control scaffold and that can just happen and both sides can be confident that they're both agreeing”

Claim: The speaker says that when computation is transparent, ordinary regulation and agreements can function, including reciprocal agreements to run a control scaffold.

“we would be immediately publishing all the core training recipes of Anthropic and OpenAI for the world to see.”

Claim: The proposed transparency plan would immediately make Anthropic’s and OpenAI’s core training recipes public.

“Anthropic and OpenAI will not be happy with this.”

Claim: Anthropic and OpenAI will oppose or be unhappy with the proposed transparency.

“It will cut into their valuations dramatically.”

Claim: Making the core training recipes public would dramatically reduce Anthropic’s and OpenAI’s valuations.

“because it will allow other competitors like Microsoft and Alibaba to catch up or whatever.”

Claim: Publishing the core training recipes would help competitors such as Microsoft and Alibaba catch up with leading AI companies.

“And so that's why they're going to hit it probably.”

Claim: The speaker thinks Anthropic and OpenAI will probably oppose the transparency plan because it could help competitors catch up.

“But I would say this is a feature, not a bug.”

Claim: The speaker considers the competitive effects of transparency a desirable feature rather than a flaw.

“Like we want there to be multiple different AI companies at the frontier at roughly similar levels of capability. We want AI to commoditize instead of being monopolized or oligopolized.”

Claim: The speaker favors AI becoming commoditized rather than controlled by a monopoly or oligopoly.

“Like we want there to be multiple different AI companies at the frontier at roughly similar levels of capability. We want AI to commoditize instead of being monopolized or oligopolized.”

Claim: The speaker wants multiple AI companies to operate at the frontier with roughly similar capabilities.

“And also this will disincentivize further investment, Like investors will be less interested in building a trillion dollar cluster if they won't be able to get the monopoly rents.”

Claim: The speaker expects transparency to reduce investment incentives: investors will be less interested in building a trillion-dollar compute cluster if they cannot earn monopoly rents from it.

“from from that cluster um but we think this is good because like again we are going to be in a world where going too fast is our main problem and so having a bit less incentive to invest and going a bit slower is actually just i think a feature not a bug there still will be investment like there still be lots of money to made and so like progress will continue it just not at quite the same rate and and again we think this is good um now this does shift this is kind of a gift uh to china relative to the u.s like this does”

Claim: The speaker expects excessive speed to be the main problem and therefore favors somewhat lower investment incentives and slower AI progress, while expecting investment and progress to continue at a reduced rate.

“This does mean that like China gets some algorithms that they might have had trouble getting before.”

Claim: The speaker says transparency would give China access to some algorithms that it might otherwise have struggled to obtain.

“then negotiate it like you can have a horse trading type thing where when U.S. and China are making the deal, the U.S. is like, well, since we're giving you all this stuff with the transparency, why don't you give us something else in return and we can try we can try to work that out.”

Claim: The speaker proposes that the United States negotiate reciprocal concessions with China if it objects to China receiving algorithms through transparency.

“Like a more favorable compute distribution, for example, you know, so so there's got to be some combination of.”

Claim: A more favorable distribution of compute is one possible reciprocal concession in a United States–China transparency deal.

“of you know carrots and sticks and trade and going back and forth that we think would be in the interest of both sides another thing worth mentioning is that it's not as big of a gift to China as you might think because security is poor at these companies and so they're probably through their spy networks and through leaks getting most of the information anyway”

Claim: The speaker believes transparency would be a smaller advantage to China than it may appear, because company security is poor and China probably already obtains most of the information through espionage and leaks.

“of you know carrots and sticks and trade and going back and forth that we think would be in the interest of both sides another thing worth mentioning is that it's not as big of a gift to China as you might think because security is poor at these companies and so they're probably through their spy networks and through leaks getting most of the information anyway”

Claim: The speaker believes a mix of incentives, penalties, trade, and negotiation could be in the interests of both the United States and China.

“plan s would be better than the default like i would rather just stop everything now than continue going on our current trajectory secondly our actual recommendation would be to do plan a so you do like a temporary inference only pause now so that you can set up all the transparency and verification infrastructure and then you can continue in this more distributed you know transparent cautious way as as previously described”

Claim: The speaker prefers an immediate halt to AI development over continuing along the current trajectory.

“plan s would be better than the default like i would rather just stop everything now than continue going on our current trajectory secondly our actual recommendation would be to do plan a so you do like a temporary inference only pause now so that you can set up all the transparency and verification infrastructure and then you can continue in this more distributed you know transparent cautious way as as previously described”

Claim: The speaker recommends a temporary inference-only pause to build transparency and verification infrastructure, then resuming AI development in a more distributed, transparent, and cautious manner.

“you basically go up until the level that you feel like you can control reliably until you feel confident that you've solved alignment enough that you can give up on control.”

Claim: The speaker proposes advancing AI only to a capability level that can be reliably controlled, and relinquishing control only once alignment is sufficiently solved.

“And that's what we depict happening over the course of the 2030s in our scenario.”

Claim: The speaker’s scenario depicts reaching the point of sufficiently solved alignment and relinquishing control during the 2030s.

“I mean, discourse is bad in general, right?”

Claim: The speaker says public discourse is generally poor, not only discourse about AI.

“uh i mean there's like massive incentives uh for people and sort of like motivated reasoning for people at air companies to sort of like think that what they're doing is like justified and good and that you know they shouldn't do like costly actions that would make the situation better because actually those costly actions would be bad for whatever reason so like there's much rationalization um yeah i think most of the effect is probably just like generically discourses like hard and bad like there are like you know twitter or whatever is like very like”

Claim: People at AI companies face strong incentives to view their work as justified and good and to rationalize avoiding costly actions that could improve the situation.

“uh i mean there's like massive incentives uh for people and sort of like motivated reasoning for people at air companies to sort of like think that what they're doing is like justified and good and that you know they shouldn't do like costly actions that would make the situation better because actually those costly actions would be bad for whatever reason so like there's much rationalization um yeah i think most of the effect is probably just like generically discourses like hard and bad like there are like you know twitter or whatever is like very like very like unnuanced and argumentative yeah i don't know i guess i'll i'll put a plug out”

Claim: The speaker thinks the main problem is probably that discourse is generally difficult and poor in quality, with Twitter-like platforms being unnuanced and argumentative.

“think uh less wrong in particular which is where i try to do most of my discourse i think the discourse quality there is actually pretty good on average um and like generally like when i like write a post and like go into the comments i think generally the comments are just like quite thoughtful and you know technically informed and whatnot um especially relative to you know other places like you know twitter or whatever um”

Claim: The speaker considers discourse on LessWrong generally thoughtful and technically informed, particularly relative to platforms such as Twitter.

“And then, yeah, maybe a final thing is just like in DC in particular, which is an area I care a lot about, I really want sort of DC and sort of the governance, you know, the government of the United States and other governments to react well to the AI technology.”

Claim: The speaker wants Washington, the U.S. government, and other governments to respond effectively to AI technology.

“There's sort of two problems going on. One is there isn't very much AI expertise, There's like, you know, their government is not hiring like really high quality technical AI experts who really know what they're doing.”

Claim: The speaker says governments lack substantial AI expertise and are not hiring highly capable technical AI experts.

“And the second is sort of this just generic, like, I think the conversation is not happening.”

Claim: The speaker says meaningful conversation about AI is not taking place in Washington.

“like the incentives for everyone in dc are not towards like sort of like truthfully and accurately understanding situation the incentives are sort of um for every individual to sort of like say stuff that sounds good and looks good and it's sort of like within the dc overton window so that they can make a lot of friends unfortunately i think this just like comes apart from the actual reality i think the actual reality of the situation is like it turns out a bunch of sort of like controversial and niche views about ai”

Claim: The speaker believes that several controversial and niche views about AI have turned out to be true.

“like the incentives for everyone in dc are not towards like sort of like truthfully and accurately understanding situation the incentives are sort of um for every individual to sort of like say stuff that sounds good and looks good and it's sort of like within the dc overton window so that they can make a lot of friends unfortunately i think this just like comes apart from the actual reality i think the actual reality of the situation is like it turns out a bunch of sort of like controversial and niche views about ai”

Claim: The speaker argues that Washington incentives reward statements that sound good and fit the D.C. Overton window, rather than truthful and accurate understanding of AI, partly because individuals want to make friends.

“were true right like the whole AGI hypothesis just like is correct and I think DC basically just like hasn't come to grips with that and so basically almost all of the discourse that's happening there I think is just like fundamentally anchored on completely wrong assumptions about how the technology works in particular the assumption that like it's like mostly fake news and it's all a bubble”

Claim: The speaker believes the AGI hypothesis is correct and that Washington has not yet accepted or understood its implications.

“were true right like the whole AGI hypothesis just like is correct and I think DC basically just like hasn't come to grips with that and so basically almost all of the discourse that's happening there I think is just like fundamentally anchored on completely wrong assumptions about how the technology works in particular the assumption that like it's like mostly fake news and it's all a bubble”

Claim: The speaker believes nearly all Washington discourse about AI rests on fundamentally incorrect assumptions about the technology, including that AI is mostly fake news and a bubble.

“that it'll be like the next internet which is like yeah maybe which is like better than it was a few years ago where it was even more bearish but it's still not sort of nearly bullish enough on the technology in my opinion”

Claim: The speaker considers the view that AI will be like the next internet more optimistic than it was a few years earlier, but still insufficiently optimistic about AI.

“some of the people at AI companies are being sincere, not all of them. I don't think you should trust what the leadership of the companies say in general, but anyhow.”

Claim: The speaker says some people at AI companies are sincere but not all, and advises against generally trusting AI-company leadership.

“And one highlight from it from our perspective is that we kind of had a truce where we were like, yes, AI right now, maybe it's a normal technology. But in the future, it will not be normal. In particular, in the future, it'll be more like humans in the cloud and all this crazy stuff is going to start happening as we describe in AI 207.”

Claim: The speakers agreed that AI might currently be a normal technology but will become unlike a normal technology in the future, with developments such as humans in the cloud.

“They just think that you're not going to get that, at least not for many, many years, you know. So in some sense, the main disagreement between us and them is a disagreement about timelines to that level of AI.”

Claim: The speakers say their central disagreement with AI-as-normal-technology proponents concerns the timeline for reaching human-level AI capabilities such as humans in the cloud.

“You know, is it possible that in the next few years we will have AIs that are like humans in the cloud and they can just do all sorts of knowledge work in a way that substitutes for humans, including AI research, for example?”

Claim: The speakers consider whether, within the next few years, AI systems could perform a broad range of human-substituting knowledge work, including AI research, as humans in the cloud.

“And then sometime after that, we will have robots perhaps controlled by those AIs that can do physical work in a way that broadly substitutes for humans.”

Claim: The speakers predict that after human-substituting AI for knowledge work arrives, robots—perhaps controlled by those AIs—will be able to perform physical work broadly substituting for humans.

“And our claim is, yes, in the next few years, that sort of thing will be achievable.”

Claim: The speakers predict that AI capable of broadly substituting for humans in knowledge and physical work will be achievable within the next few years.

“And their claim is, nope, not in the next few years. That's much more far away.”

Claim: The proponents of AI as normal technology predict that human-substituting AI and robotics are much farther away than the next few years.

“And I think that is like the main source of our disagreement. And we would sort of agree with them that like if that level of AI and robots is still very, very far away. way then yeah like maybe it has more of a normal technology it'll be like the next internet or something you know um but we just think that actually it's on a path to get to that level of capability soon i”

Claim: The speaker allows that if human-substituting AI and robotics remain very far away, AI may be more like a normal technology, comparable to the next internet.

“way then yeah like maybe it has more of a normal technology it'll be like the next internet or something you know um but we just think that actually it's on a path to get to that level of capability soon i”

Claim: The speaker believes AI is on a trajectory to reach human-substituting knowledge-work and robotics capabilities soon.

“You know, people keep talking about the limitations of the current paradigm, but then the limitations of the current paradigm keep getting overcome within the current paradigm.”

Claim: Limitations attributed to the current AI paradigm have repeatedly been overcome without leaving that paradigm.

“And one thing that would change my mind is if someone was actually right about one of these limitations, like if someone right now is going around saying that like. yeah data efficiency or like non-verifiable tasks or something like that is the current limitation and then like several years go by and it becomes clear that there's like very basically no progress on overcoming that limitation and that the AIs of like 20, 29 are like no better at, at the fuzzy non-verifiable tasks than the AIs of 2025,”

Claim: The speaker would revise his view if a correctly identified AI limitation, such as data efficiency or non-verifiable tasks, remained essentially unimproved for several years, so that 2029 AIs were no better at fuzzy non-verifiable tasks than 2025 AIs.

“this feels like a real barrier. This feels like something we're starting to like feel the elephant. We're like running up against some sort of like real actual barrier that was correctly predicted by theory by some people that was going to be there. actually running up against some sort of barrier like that, I think would would lengthen my timelines quite a lot.”

Claim: The speaker thinks his AI timelines would lengthen substantially if AI progress encountered a genuine barrier that had been correctly predicted by theory.

“So if there was a war with China and most of the chips get destroyed by missiles, that would lengthen my timelines if on the bright side.”

Claim: The speaker's AI timelines would lengthen if a war with China destroyed most chips in missile attacks.

“that's true yeah on the bright side if there's more of like an international deal to pace the pace the frontier that would lengthen my timelines etc yeah”

Claim: The speaker's AI timelines would lengthen if countries reached an international agreement to pace frontier AI development.

“but even what if we're that thinking. wasn't will be like in the near like i think that if we froze ai progress right now we didn't train any new models then it would sort of become more of a normal technology where like there's just so many things that mythos 5 can't do and so if we couldn't get any new models beyond mythos 5 then like it would be more like the internet where like we would restructure a lot of our professions we would re-structure a lot of our workflows to incorporate copies of mythos 5 doing parts of it but then the humans would just like shift to doing more of the things that mythos 5 can't do and so there would be like you know a big change in a lot of things but it and it would it would be like the next internet in terms of would change everything in some sense but it wouldn't fundamentally change anything really”

Claim: If AI progress stopped at Mythos 5, the speaker expects AI to function more like a normal technology: it would reshape professions and workflows, while humans shifted to tasks Mythos 5 cannot perform, but it would not fundamentally transform society.

“Yeah, so for me, adaptability is the most synonymous word with intelligence.”

Claim: The speaker regards adaptability as the concept most synonymous with intelligence.

“I think that these new RL trained models we have, they're not the same as what we had many years ago.”

Claim: Current reinforcement-learning-trained models differ from models that existed many years ago.

“So scaling means you take a scalar property of a system and you scale it up.”

Claim: Scaling means increasing a scalar property of a system.

“And these RL systems, yeah, they're still self attention transformers, but they're different. They're actually structurally different.”

Claim: The RL systems under discussion are still self-attention transformers, but their structure differs from earlier systems.

“There's new types of training, new architectures used differently and so on and so forth.”

Claim: The RL systems being discussed use new training methods and apply architectures differently.

“So a bunch of humans, what they did was they did some experiments and they adapted the structure to create a system that had different scaling properties.”

Claim: Human researchers experimented with and adapted system structure, producing a system with different scaling properties.

“So I can imagine a future where we actually have some kind of recursive loop where the system is adapting itself and it's deciding what things are interesting and it's kind of evolving by itself. When that happens efficiently, I think that is. is a different type of technology.”

Claim: The speaker imagines that an efficiently operating recursive loop in which an AI system adapts itself, selects what to explore, and evolves autonomously would constitute a different kind of technology.