Three Red Lines We're About to Cross Toward AGI (Daniel Kokotajlo, Gary Marcus, Dan Hendrycks)

“Having, say for instance the United States, disrupt China's ability to develop a superintelligence, the main way in which they would develop it is if they get the ability to automate AI research and development fully and take the human out of the loop. Then you go from human speed to machine speed, we've had people such as Dario for instance at Sheo Anthropik talk about how such a recursive process like that could lead to an intelligence explosion.”

Claim: Dario has discussed the possibility that recursively automated AI research and development could lead to an intelligence explosion.

“Having, say for instance the United States, disrupt China's ability to develop a superintelligence, the main way in which they would develop it is if they get the ability to automate AI research and development fully and take the human out of the loop. Then you go from human speed to machine speed, we've had people such as Dario for instance at Sheo Anthropik talk about how such a recursive process like that could lead to an intelligence explosion.”

Claim: Fully automating AI research and development and removing humans from the loop could shift the pace of development from human speed to machine speed.

“of this explosion and that would lead to a durable edge where nobody will be able to catch up and last week Sam Altman discussed how this process could telescope a decade's worth of AI development in a year or potentially a month.”

Claim: An intelligence explosion could create a durable advantage that other actors cannot catch up to.

“of this explosion and that would lead to a durable edge where nobody will be able to catch up and last week Sam Altman discussed how this process could telescope a decade's worth of AI development in a year or potentially a month.”

Claim: Sam Altman has said that an AI-development process could compress a decade's worth of development into one year or possibly one month.

“There's been this very seductive argument that has appealed to all of these people, which is basically well it's probably gonna happen anyway, if we don't do it someone else will. Basically all of these people sort of trust themselves more than they trust everyone else.”

Claim: People who use the argument that someone else will build advanced AI tend to trust themselves more than they trust others.

“There's been this very seductive argument that has appealed to all of these people, which is basically well it's probably gonna happen anyway, if we don't do it someone else will. Basically all of these people sort of trust themselves more than they trust everyone else.”

Claim: Some people argue that advanced AI will probably be developed regardless, because if one group does not build it, another will.

“else and have therefore convinced themselves that even though these risks are real, the best way to deal with them is for them to go as fast as possible and win the race. Why did they make OpenAI? Well they were worried that they didn't trust Demis to handle all that power responsibly when he was in charge of the AI project and all the fate of the world rested in his hands. So they wanted to create OpenAI to be this countervailing force that could do it right and distribute it to everybody and not concentrate power so much in Demis' hands.”

Claim: OpenAI was created as a countervailing force because its founders worried that Demis would not responsibly handle the power of leading the AI project.

“else and have therefore convinced themselves that even though these risks are real, the best way to deal with them is for them to go as fast as possible and win the race. Why did they make OpenAI? Well they were worried that they didn't trust Demis to handle all that power responsibly when he was in charge of the AI project and all the fate of the world rested in his hands. So they wanted to create OpenAI to be this countervailing force that could do it right and distribute it to everybody and not concentrate power so much in Demis' hands.”

Claim: OpenAI's founders intended the organization to distribute AI broadly and prevent power from being concentrated in Demis's hands.

“else and have therefore convinced themselves that even though these risks are real, the best way to deal with them is for them to go as fast as possible and win the race. Why did they make OpenAI? Well they were worried that they didn't trust Demis to handle all that power responsibly when he was in charge of the AI project and all the fate of the world rested in his hands. So they wanted to create OpenAI to be this countervailing force that could do it right and distribute it to everybody and not concentrate power so much in Demis' hands.”

Claim: Some people acknowledge that AI risks are real but believe that moving as fast as possible and winning the race is the best way to address them.

“And in fact, the theme also came up in the lawsuit. They were talking about how they were worried that Demis would become dictator, using a G_I_”

Claim: People involved in the lawsuit discussed fears that Demis would become a dictator by using AGI.

“On the proto A_S_I_ things, I think that they follow the instruct instructions fairly reasonably. There are some other parts of it”

Claim: The proto-ASI systems follow instructions fairly reasonably.

“You know when these things are still in our hands kind of, that's okay, but if you have them guiding weapons or something like that, there are many circumstances where fairly reasonably is not good enough.”

Claim: Fairly reasonable behavior is insufficient in many circumstances where AI systems guide weapons.

“All right, so I'm Gary Marcus. I'm a cognitive scientist, an entrepreneur.”

Claim: Gary Marcus is a cognitive scientist and entrepreneur.

“I've written six books, most recently Taming Silicon Valley.”

Claim: Gary Marcus has written six books, with Taming Silicon Valley as his most recent book.

“And I think we all want what is best for humanity and hope that AI can be positive for humanity and not negative. I think the three of us are maybe not known for agreeing on everything, but we actually have a lot of. A lot of shared values around that, and I'm looking forward to the conversation.”

Claim: Gary Marcus believes the three participants share values around wanting AI to benefit humanity rather than harm it.

“My name is Daniel Crocatello. I'm the Executive Director of the AI Futures Project.”

Claim: Daniel Kokotajlo is the executive director of the AI Futures Project.

“We are the people who made AI 2027, which is a comprehensive detailed scenario forecast of the future of AI.”

Claim: The AI Futures Project created AI 2027, a comprehensive and detailed scenario forecast of AI's future.

“More recently, I've done evaluations to evaluate capabilities such as MMLU and the math benchmark and humanity's last exam. I'm the director of the Center for AI Safety.”

Claim: Dan Hendrycks is the director of the Center for AI Safety.

“I also advise scale AI and XAI.”

Claim: Dan advises Scale AI and xAI.

“I also have done research in machine learning. like I coined the Jell-O and the C-Loo, a common activation function.”

Claim: Dan says he conducted machine-learning research and coined GELU and SiLU, which he describes as a common activation function.

“More recently, I've done evaluations to evaluate capabilities such as MMLU and the math benchmark and humanity's last exam.”

Claim: Dan says he has recently evaluated AI capabilities using MMLU, a math benchmark, and Humanity's Last Exam.

“And I've been focusing on trying to measure aspects of intelligence and also AI systems' safety properties for mostly the course of my whole career.”

Claim: Dan says that for most of his career he has focused on measuring aspects of intelligence and the safety properties of AI systems.

“You want to start with that? Yeah, so first of all, I actually think it's quite reasonable to basically stand in front of the bulldozer and say stop.”

Claim: Daniel Kokotajlo considers it reasonable to call for stopping AI development.

“I think that eventually we want to build AI because it is possible to get it right and it is possible to make it in such a way that's, you know, beneficial to everyone and massively beneficial to everyone in fact.”

Claim: Daniel Kokotajlo believes AI should eventually be built because it may be possible to develop it correctly and make it beneficial to everyone, potentially to a great extent.

“If we are currently not on a track to getting it right and we're currently on a track to make it in a way that's going to be horrible, then it makes sense to just sort of stop until we figure out a better way.”

Claim: If AI development is on a path toward a horrible outcome rather than being done correctly, it makes sense to pause development until a better approach is found.

“AI 2027 has the slowdown ending in which things go really well for almost everybody.”

Claim: AI 2027's slowdown ending depicts an outcome that goes very well for almost everyone.

“I think more people probably read the darker. scenario than the positive scenario.”

Claim: Gary Marcus believes more people probably read AI 2027's darker scenario than its positive scenario.

“Sure. So in the in the slowdown ending of AI 2027, they. manage to devote just barely enough effort and time and resources towards the technical alignment problem that they solve it just barely in time so that they can still beat China as they have wanted to do. But by they, I mean, you know, the leader of the leading AI company and the president and maybe some other companies, right, that group of people.”

Claim: In the slowdown ending of AI 2027, the leading AI company’s leader, the president, and possibly other companies devote just enough effort, time, and resources to solve the technical alignment problem barely in time, while still beating China.

“You know, the slowdown ending is not our recommendation for what we should actually aim for.”

Claim: The speaker and their collaborators do not recommend aiming for the slowdown ending of AI 2027.

“We think to aim for this sort of scenario would be to expose the world to incredible amounts of risk of various types.”

Claim: The speaker believes that aiming for the AI 2027 slowdown scenario would expose the world to very large risks of various kinds.

“But nevertheless, it's a coherent, plausible scenario for how things could sort of muddle through and end up pretty well for everybody.”

Claim: The speaker considers the AI 2027 slowdown scenario coherent and plausible, with a possible outcome in which events muddle through and things end up pretty well for everyone.

“When you get the AIs that are super intelligent, so what I mean by that is better than the best humans at everything, and also faster and cheaper,”

Claim: The speaker defines superintelligent AI as outperforming the best humans at everything while also operating faster and at lower cost.

“you can just completely transform the economy,”

Claim: The speaker says superintelligent AI could completely transform the economy.

“you know, super intelligent designed robot factories constructed in record speed producing all these amazing robots and all these amazing new industrial equipment which then are used to construct new types of factories and new types of laboratories to do all sorts of experiments to build all sorts of new technology.”

Claim: The speaker describes a possible cycle in which superintelligent AI designs rapidly constructed robot factories that produce robots and industrial equipment, which are then used to build more factories and laboratories for experiments and technology development.

“new technologies and blah blah blah blah blah eventually you get and by eventually i mean in only a few years uh you get to a completely automated wonderful economy of all sorts of new technologies that have been iterated on and designed by super intelligences material needs are basically just met for everybody you know there's just an incredible abundance of wealth uh to distribute and uh you know things like curing all sorts of diseases”

Claim: The speaker predicts that within only a few years, superintelligences could design and iteratively improve technologies that enable a completely automated economy, meet nearly everyone’s material needs, and create abundant wealth to distribute.

“new technologies and blah blah blah blah blah eventually you get and by eventually i mean in only a few years uh you get to a completely automated wonderful economy of all sorts of new technologies that have been iterated on and designed by super intelligences material needs are basically just met for everybody you know there's just an incredible abundance of wealth uh to distribute and uh you know things like curing all sorts of diseases putting up new settlements on Mars and so forth, all that stuff becomes possible.”

Claim: The speaker says that in the envisioned superintelligence-driven future, curing many diseases and establishing new settlements on Mars could become possible.

“So that's the sort of like potential upside. And then, of course, there's the question of can we actually achieve that, right? And if we do have the technology to achieve that, who's in control of the technology? And do they actually use their power over the army of superintelligences to make that sort of broadly distributed good future for everybody? Or do they do something that's more dystopian, you know?”

Claim: The speaker sees a broadly beneficial, widely shared future as a potential upside of superintelligence, but questions whether it is achievable, who would control the technology, and whether controllers would use it for that future or for a more dystopian outcome.

“Yeah. So I think there are two concepts. There's AGI and there's ASI, the artificial general intelligence and artificial super intelligence. I think AGI doesn't have a clear definition, so it's difficult for me to say for all definitions of that that would be worth standing in front of.”

Claim: The speaker believes AGI lacks a clear definition, so they cannot say that every system described as AGI would be worth physically resisting to stop.

“Some people would say we have AGI already.”

Claim: The speaker reports that some people believe AGI already exists.

“So I primarily think about stopping superintelligence. The main reason for that is because that is much more implementable, geopolitically compatible with existing incentives, and so on. So it's something that I can actually... foresee.”

Claim: The speaker primarily focuses on stopping superintelligence because they consider that goal more implementable and more compatible with existing geopolitical incentives.

“I think things stopping tomorrow is not something I can as easily foresee. So I don't really think about that.”

Claim: The speaker finds an immediate halt to AI progress difficult to foresee and therefore does not focus much on that scenario.

“I couldn't conceive of how we would coordinate now to do that.”

Claim: SPEAKER_02 could not conceive of how people could coordinate at present to prevent the development of superintelligence.

“the United States disrupt China's ability to develop a superintelligence. The main way in which they would develop it is if they get the ability to automate AI research and development fully and take the human out of the loop.”

Claim: SPEAKER_02 says the main route by which China would develop superintelligence is fully automating AI research and development, removing humans from the loop.

“loop. Then you go from machine speed or from human speed to machine speed.”

Claim: SPEAKER_02 says fully automated AI research and development would shift the pace of development from human speed to machine speed.

“And we've had people such as Dario, for instance, the CEO of Anthropoc talk about how that will give the USA or such a such a recursive process like that could lead to an intelligence explosion and that would lead to a durable edge where nobody will be able to catch up. And last week, Sam Altman discussed how this process could.”

Claim: SPEAKER_02 reports that Dario, Anthropic's CEO, has said recursive AI development could cause an intelligence explosion and create a durable lead that others could not catch up to.

“the CEO of Anthropoc talk about how that will give the USA or such a such a recursive process like that could lead to an intelligence explosion and that would lead to a durable edge where nobody will be able to catch up. And last week, Sam Altman discussed how this process could. telescope a decade's worth of AI development in a year or potentially a month.”

Claim: SPEAKER_02 reports that Sam Altman said recursive AI development could compress ten years of AI development into one year or possibly one month.

“And I think that's extraordinarily destabilizing for two reasons.”

Claim: SPEAKER_02 considers a very fast automated AI R&D loop extraordinarily destabilizing.

“One, if they control it then, or if a state controls it, such as say China, then all the other countries are at substantial risk because that superintelligence could be weaponized and used to crush other countries.”

Claim: SPEAKER_02 says that if a state such as China controls superintelligence, other countries face substantial risk because it could be weaponized against them.

“And if they don't control it, which I think would be I think would be fairly likely by a nearly unsupervised, extremely fast-moving process.”

Claim: SPEAKER_02 thinks it would be fairly likely that a state would not control the rapid AI development process, which would be nearly unsupervised.

“then everybody's survival is also threatened.”

Claim: SPEAKER_02 says that if a state does not control the rapid AI development process, everyone's survival is threatened.

“So either way, this very fast automated AI R&D loop is quite destabilizing whether a state controls it or not.”

Claim: SPEAKER_02 considers a very fast automated AI R&D loop destabilizing whether or not a state controls it.

“And so I think it makes sense not just because AI is scary or in some vague sense, but I think that there are very strong geopolitical incentives for a state's self-preservation to prevent that.”

Claim: SPEAKER_02 argues that states have strong geopolitical incentives to prevent destabilizing rapid automated AI development in order to preserve themselves.

“I think people generally think that if we have AI, it will necessarily hollow out all other all values or that will have an extreme pressure toward one of them.”

Claim: SPEAKER_02 says people generally believe AI will inevitably displace other values or exert extreme pressure toward one value.

“That will be zoo animals or we'll be all pleasured out and just, you know, in our VR experiences constantly and have superficial experiences.”

Claim: SPEAKER_02 describes possible outcomes of AI-driven value displacement as people becoming like zoo animals or living constantly in VR with superficial experiences.

“I think you could set a society up so that people have the autonomy to choose between different ways that they would want to live their life, given the resources that AI increasing GDP could provide.”

Claim: SPEAKER_02 thinks society could preserve people's autonomy to choose their way of life using resources generated by AI-driven GDP growth.

“So I think that there's a way you could have a list of objective goods being met in society. So I don't think there's no equilibria where things can work out.”

Claim: SPEAKER_02 believes society can meet objective goods and that workable equilibria exist.

“I think there's a way of achieving a variety of goods and still preserving human autonomy and all that like sort of what we have now.”

Claim: SPEAKER_02 believes it is possible to achieve a variety of social goods while preserving human autonomy.

“right that's right and there's there's other sorts of longer term questions about how you might do that like how are you going to distribute power maybe that would be with uh compute slices that people would rent out for instance where they would have the unique cryptographic key uh to to activating that compute slice so that you're not just distributing wealth but you're also distributing um ways of of generating that wealth”

Claim: SPEAKER_02 suggests distributing power through rentable compute slices activated by individual cryptographic keys, thereby distributing both wealth and wealth-generating capacity.

“You know, there's a lot in this deep future that one could pull out, but I think there are some paths where things work out well.”

Claim: SPEAKER_02 believes some possible deep-future paths lead to good outcomes.

“I heard him talking about universal basic income. And, you know, I've always thought that that was an important part of the equation.”

Claim: SPEAKER_07 has long considered universal basic income an important part of addressing the effects of AI and economic change.

“And it seems to me that the more that data has driven things, the more acquisitive the companies, and it's not just open AI, have been, the less realistic it has seemed to me that we're going to have anything like a universal basic income. I think there is some scenario.”

Claim: SPEAKER_07 believes that as data has become more influential, companies have become more acquisitive and universal basic income has seemed less realistic.

“in which there's so much wealth that everybody's subsistence gets met I don't expect that the wealthy people are going to give up the beachfront property under any circumstance or their power and I think the dynamics that we've seen in Washington lately have made me though I know our politics may not be the same have made me less optimistic about any kind of relatively equal distribution or even just not a relatively extreme distribution I think that that”

Claim: SPEAKER_07 does not expect wealthy people to relinquish beachfront property or power under any circumstances.

“in which there's so much wealth that everybody's subsistence gets met I don't expect that the wealthy people are going to give up the beachfront property under any circumstance or their power and I think the dynamics that we've seen in Washington lately have made me though I know our politics may not be the same have made me less optimistic about any kind of relatively equal distribution or even just not a relatively extreme distribution I think that that”

Claim: SPEAKER_07 allows that a scenario may arise in which abundant wealth meets everyone's subsistence needs.

“in which there's so much wealth that everybody's subsistence gets met I don't expect that the wealthy people are going to give up the beachfront property under any circumstance or their power and I think the dynamics that we've seen in Washington lately have made me though I know our politics may not be the same have made me less optimistic about any kind of relatively equal distribution or even just not a relatively extreme distribution I think that that”

Claim: SPEAKER_07 says recent political dynamics in Washington have reduced their optimism about achieving relatively equal distribution or avoiding extreme inequality.

“positive outcomes do depend on getting some answers there to what would be the incentives, the mechanisms, the dynamics such that things are well distributed.”

Claim: SPEAKER_07 believes positive outcomes depend on identifying incentives, mechanisms, and dynamics that produce well-distributed resources.

“And I guess I'm sort of making this up as I go, but I think there's an argument here for stopping the train if we can't see any solution to the political thing.”

Claim: SPEAKER_07 argues that AI development should be stopped if no solution to the political problem can be envisioned.

“Most of my own personal concerns are really about technical alignment, which we'll talk about soon enough. But if we don't... have an even envisionable political solution that does make me nervous and I could see an argument for let's stop the train now because we just don't see how to do it and going back to something you said at the very beginning there's an interesting argument about delaying the train versus stopping the train and like I think all of us signed the pause letter maybe”

Claim: SPEAKER_07 is concerned that no political solution is currently envisionable and sees a case for stopping AI development now if there is no way to solve the problem.

“the the part of the pause letter that made me sign it that i like the best was it said let's pause this particular thing that we know is problematic in certain ways i mean it was really about delaying the development of gpt5 which ironically still doesn't exist two and a half years after some of us signed the pause letter um but the notion was we would pause the development of gpt5 because we Because we knew that GPT-4 had certain kinds of problems around alignment and that we would spend that time instead working on safety.”

Claim: The pause letter proposed delaying GPT-5 development because GPT-4 had alignment problems, using the delay to work on safety.

“So it was explicitly constructed as a delaying tactic. It wasn't saying never build AI. It wasn't saying don't do any more AI research.”

Claim: The pause letter was a delaying tactic; it did not call for permanently stopping AI development or all AI research.

“It was, in fact, saying do more AI safety research and wait. And that does still seem like maybe not a politically viable thing in this moment, but at least a strategy one might.”

Claim: The pause proposal advocated more AI safety research and waiting, although the speaker thought this might not be politically viable at the time of the conversation.

“one might consider is, you know, maybe we should be constantly updating our estimates on, you know, how likely do you get the positive outcomes versus negative outcomes, and how much would that change as a function, for example, of putting more resources into safety research as opposed to capabilities research, etc.”

Claim: A strategy worth considering is to continually update estimates of positive and negative AI outcomes, including how those estimates change when resources shift from capabilities research to safety research.

“I like the pause say I framing more than the stop AI framing for the reasons you just mentioned.”

Claim: The speaker prefers framing the proposal as pausing AI rather than stopping AI.

“And then there's a lot more to say about more nuanced proposals for what is to be done.”

Claim: The speaker believes there are many more nuanced proposals for addressing AI-related concerns.

“I think making time for technical research, I am not expecting much of a return on investment from that. Um I don't think um most success strategies part um in the foreseeable future particularly go through uh”

Claim: The speaker does not expect technical research to yield much return on investment.

“I think making time for technical research, I am not expecting much of a return on investment from that. Um I don't think um most success strategies part um in the foreseeable future particularly go through uh AI is being fully controllable.”

Claim: The speaker does not expect most successful strategies in the foreseeable future to achieve full controllability of AI.

“Like an extreme version of the argument I just placed comes from the cognitive psychologist and evolutionary psychologist Jeffrey Miller, who had a tweet that I really liked, which was, we should wait until we can build this stuff safely, even if that takes, I forget what he said, I'll say 250 years. um it was an interesting framing because like everybody's thinking like what should we do next week or should we sign this pause letter and it was kind of deliberately extreme like i think it was maybe even 500 years like if it takes us 500 years we should just wait and you know”

Claim: Jeffrey Miller argued that AI development should wait until AI can be built safely, even if achieving that takes hundreds of years.

“So I think that the process that I described earlier, that sort of recursive loop, I guess you could call it an intelligence recursion, which if it goes fast enough is an intelligence explosion,”

Claim: A recursive intelligence-improvement loop could become an intelligence explosion if it proceeds fast enough.

“that is not something you can research your way out of. You can't just write an eight-page paper and then we've solved it. In the news tomorrow, we figured out how to fully de-risk.”

Claim: The speaker believes research alone cannot solve the risks posed by an intelligence explosion.

“There's some extremely fast moving process that we've never done before and all of its unknown unknowns have been anticipated.”

Claim: The speaker characterizes fully automated intelligence recursion as an unprecedented, extremely fast-moving process whose unknown unknowns cannot all be anticipated.

“Well, I mean, we could politically, I mean, it would take a lot of willpower and it's probably not likely, but we could, you know, try to have a global treaty. Don't go there. Don't work on these kinds of things. Let's report it if you do.”

Claim: The speaker thinks a global treaty prohibiting work on these kinds of AI systems, with reporting requirements, would require substantial political will and is probably unlikely.

“Right. I mean, you could at least imagine that kind of scenario. I don't see any technical way. of coping necessarily with that set of problems right now but if we were”

Claim: The speaker does not currently see a technical solution for coping with the problems posed by intelligence recursion.

“We will make treaties around it and we just shouldn't do it until we have answers to them.”

Claim: The speaker proposes treaties establishing AI red lines and avoiding crossing them until the relevant questions have answers.

“So I think it would be worth states clarifying that we don't want anybody doing that type of fully automated intelligence recursion because it'd be destabilizing.”

Claim: States should clarify their opposition to fully automated intelligence recursion because the speaker considers it destabilizing.

“But you can still have various forms of coordination through deterrence without anything like a treaty, just as we've had strategic stability on multiple issues without treaties necessarily. So it's something that is potentially a later stage thing, but you have to have the conversation advance for it.”

Claim: States can coordinate through deterrence without a treaty; the speaker cites strategic stability on multiple issues as precedent for this possibility.

“Yeah, so I think three red lines would be no recursion where that's fully automated, not just some AI-assisted one, but one that has that explosive potential.”

Claim: Daniel Kokotajlo identifies fully automated recursive self-improvement with explosive potential—not merely AI-assisted self-improvement—as a red line.

“No AI agents with expert level virology skills or cyber offensive skills made accessible without some safeguards,”

Claim: Daniel Kokotajlo proposes that AI agents with expert-level virology or cyber-offensive capabilities should not be accessible without safeguards.

“and model weights need to have some capability level need to have some good information security for containing them and making sure that they're not exfiltrated or stolen by rogue actors.”

Claim: Daniel Kokotajlo argues that model weights above a capability threshold need strong information security to prevent containment failures, exfiltration, or theft by rogue actors.

“here the first, I don't know about the second two, the other two, but definitely the first one.”

Claim: Gary Marcus is uncertain about the second and third proposed red lines but definitely supports the first red line.

“I think that if somehow we could coordinate on that, that'd be great. I'm not sure if that's the best red line to coordinate on, but it's an excellent place to start at least.”

Claim: Gary Marcus favors coordinating on the first red line, while remaining unsure whether it is the best line for coordination; he considers it an excellent starting point.

“I think that the type of thing that I'm probably going to end up advocating for is going to be be more of a like Rather than like here's a line that we're all not going to cross, something more like we are going to gradually develop AIs with these capabilities, but we're going to do it in a way that's like mutually transparent to each other and that proceeds sort of slowly and cautiously where we all like debate whether it's safe to go to the next level and then after we get there we study it for a little bit and then debate whether it's safe to go to the next level and so forth. So the sort of thing that we're probably going to end up advocating for is going to look something like this.”

Claim: Gary Marcus expects to advocate for gradual, mutually transparent AI capability development, with cautious staged progress, safety debates before each stage, and study of each stage before deciding whether to proceed.

“something more like that rather than uh but but yeah in terms in terms of like the thing that you really need to like stop from happening in the short term uh that sort of recursive self-improvement thing is i would say the number one thing”

Claim: Gary Marcus identifies recursive self-improvement as the main development that needs to be prevented in the short term.

“People are already pushing against them. They may not be breaking through them.”

Claim: The interviewer says that people are already pushing against the proposed AI red lines, although they may not yet have crossed them.

“Like depending on your definition of recursive self-improvement, I might give you different estimates. Mine might be a little longer than yours.”

Claim: The interviewer says estimates for recursive self-improvement depend on its definition and suggests his own timeline may be longer than the guest's.

“I mean like OpenAI was open originally. It is not open anymore.”

Claim: The interviewer says OpenAI was originally open but is no longer open.

“So, you know, there are elements still pushing for transparency, but there are certainly elements pushing against.”

Claim: The interviewer says that some actors continue to advocate for transparency while others oppose it.

“So I'm so more optimistic about increased transparency and more situational awareness from governments.”

Claim: Daniel Kokotajlo is relatively optimistic that transparency and government situational awareness will increase.

“A reason for that is I think it's incentive compatible for the U.S. to be more transparent about what's going on at the frontier.”

Claim: Daniel Kokotajlo believes that greater U.S. transparency about frontier AI developments is incentive-compatible.

“Because China already knows. Meanwhile, for PLA or People's Liberation Army developments, there's somewhat less transparency there.”

Claim: Daniel Kokotajlo says China already knows about frontier developments, whereas information about People's Liberation Army developments is somewhat less transparent.

“So the people to gain more from information would probably be the rest of the world.”

Claim: Daniel Kokotajlo thinks the rest of the world would probably benefit more than China from access to information about frontier developments.

“So China has somewhat more of an advantage there, and this sort of levels things. And that having high transparency into the frontier is very useful for making more credible deterrence.”

Claim: Daniel Kokotajlo argues that China has somewhat greater information advantage, and that high transparency about frontier AI can help level the information imbalance and make deterrence more credible.

“I think the notion of a durable advantage in LLMs, if that's what the technology is, is a myth.”

Claim: The interviewer considers the idea of a durable advantage in large language models to be a myth, assuming LLMs are the relevant technology.

“Like if we stay on LLMs, nothing's going to be durable.”

Claim: The interviewer predicts that no durable advantage will persist if AI development remains based on LLMs.

“I think with the current paradigm. I am expecting them to continue to leapfrog each other.”

Claim: Daniel Kokotajlo expects AI developers operating under the current paradigm to continue leapfrogging one another.

“So there could be a different technology, you know, my personal favorite would be neurosymbolic AI,”

Claim: The interviewer says neurosymbolic AI is his personal favorite among possible alternative AI technologies.

“but you know, this could be espionage, people are going to share ideas or whatever. But if the paradigm stays roughly like it is, I don't see anybody getting a durable advantage.”

Claim: The interviewer does not expect any actor to gain a durable advantage if the AI paradigm remains roughly unchanged.

“LLMs are just a subset of this overall AI research, you-hmm. know, and not the only possible AI design.”

Claim: Daniel Kokotajlo says large language models are one subset of AI research and are not the only possible AI design.

“I mean, I would say, like, in some sense, arguably, we're already seeing a move away from LLMs with things like these so-called reasoning agents that have access to tools and can write code and so forth.”

Claim: Reasoning agents that use tools and can write code arguably represent a move away from LLMs.

“So there's going to be this continuous shift towards, I would say, rather than like discrete paradigm shifts, it'll be more.”

Claim: AI development will proceed through a continuous shift rather than discrete paradigm shifts.

“So it'll be more like a continuous paradigm shift and the AIs of 2027 are just going to be like quite different from the AIs of 2023,”

Claim: AI systems in 2027 will be quite different from AI systems in 2023.

“But taken as a whole, the AIs, I do think that the United States could potentially end up having a durable advantage over. China and one particular tech company within the United States could end up having a durable advantage.”

Claim: The United States could potentially develop a durable overall AI advantage over China.

“China and one particular tech company within the United States could end up having a durable advantage.”

Claim: One particular U.S. technology company could potentially develop a durable advantage.

“By durable advantage, I don't mean mean um they never figure out what you already figured out i mean that by the time they catch up you've already moved ahead so that you it's like you know you keep running as fast as they're running so that even though they're only six months behind they can never quite catch up because by the time they do you're six months ahead again”

Claim: A durable advantage means maintaining a lead by advancing again before competitors can catch up, not that competitors never learn the same things.

“Right now, it doesn't seem huge, but uh um if you are the first to trigger a recursion, uh ha that can matter a lot.”

Claim: A six-month lead can matter greatly if it enables an actor to trigger recursive AI improvement first.

“the plan. Yeah, it's still highly A_I_ assisted, and doesn't there's nobody could t try and spin this off uh spin this up right now and have it have that be foreseeably going to lead to something substantial,”

Claim: Current attempts at recursive AI improvement remain highly AI-assisted, and starting such an effort now is not foreseeably expected to produce something substantial.

“you could do your hyper parameter search faster or something like that and you could call that recursion if you want but that's not really what we're talking about here we're talking about like a system finds a fundamentally new idea that heretofore would only come from humans and that leveraging”

Claim: The relevant form of AI recursion involves a system generating fundamentally new ideas previously produced only by humans and carrying out the broader development process.

“the whole else stack of not only coming up with the ideas, but testing them, validating them, working out the tweaks, implementing them, scaling them.”

Claim: The full AI research-and-development process includes generating, testing, and validating ideas, refining and implementing them, and scaling the results.

“think that's exactly what people like Sam and Dario are now hinting that they're going to be able to do soon. I'm not really buying those claims,”

Claim: Sam and Dario have hinted that they will soon be able to automate the full AI idea-to-implementation-and-scaling process, but the speaker is skeptical of those claims.

“but I think that's what they want to do now.”

Claim: Sam and Dario appear to want to automate the full AI research-and-development process now.

“Or let's say potentially destabilize it. I mean, if it's just hype and they can't really do it, it's not destabilizing. But if if it opens up one arms. of them achieves it. If it were technically feasible, that would be destabilizing, yeah.”

Claim: Automating the full AI research-and-development process would be destabilizing if technically feasible, but not if the claims are merely hype and the capability cannot be achieved.

“And I mean, I think we can agree it probably is. is technically achievable. It's just a question of whether you can do it with current technology or later.”

Claim: The speaker believes it is probably technically achievable to automate the full AI research-and-development process, though it may require technology beyond what is currently available.

“no surly argument that this can't be done that you need.”

Claim: The speaker sees no sound argument that automating the full AI research-and-development process is impossible.

“I mean, I think we all think that's going to happen.”

Claim: The speaker expects the discussed AI capability to eventually be achieved.

“And so when advantage for transparency then is if there were very high visibility as to what's going on at the frontier, if they're basically triggering this sort of process and And the public is kept reasonably informed as it's happening. I think the world would be very much freaking out.”

Claim: If frontier AI activity were highly visible and the public were kept reasonably informed as recursive development unfolded, the world would react with intense alarm.

“So I think that possibly sunlight might be a way of providing substantial pressure to prevent that.”

Claim: Public transparency might exert substantial pressure to prevent recursive frontier AI development.

“but American people are pretty worried. It's maybe not at the top of their set of worries, which might be economic. and so forth but you know you look at these polls and like 75 percent of american public is already worried and i expect that that worry will increase not decrease at least i don't see any active thing that is going to decrease it in time”

Claim: The speaker says polls indicate that 75% of Americans are already worried about AI and expects public concern to rise, seeing no active measure likely to reduce it in time.

“thing that might decrease it is propaganda from the tech companies, I think.”

Claim: The speaker suggests that technology-company propaganda might reduce public concern about AI.

“But at least my experience at OpenAI was that there was more pressure to like not talk about the risks in public and to like, you know, sort of downplay that sort of thing than there's no pressure to play it up, at least not as far as I could tell.”

Claim: The speaker's experience at OpenAI was that internal pressure favored avoiding public discussion of AI risks and downplaying them, rather than playing them up.

“And, you know, if you look at the public messaging by Sam over the years, he certainly started to like talk about it less and less and less.”

Claim: Sam's public messaging has addressed AI risks progressively less over the years.

“he didn't answer the question, what were you most worried about? I mean, I made Blumenthal come back from, Blumenthal says it's jobs, and Sam gives his explanation why he's not that worried about jobs, and I say to Blumenthal, you should really ask him a question. And Sam there said, my biggest worry is that we do, I can't remember the exact words, but substantial harm to humanity.”

Claim: At a Senate appearance, Sam described substantial harm to humanity as his biggest worry.

“and when he was back at the senate a few weeks ago his biggest harm was like in so many words like we don't make all the money and extract all the value that we could or”

Claim: The speaker characterizes Sam's biggest concern at a Senate appearance a few weeks earlier as the company not making all the money or extracting all the value it could.

“like that right so he i mean i think it's true that he he used to talk more about this kind of risk stuff and talks less now”

Claim: Sam discussed AI risks more often in the past than he does now.

“Yeah. So so first of all, yeah, loads of people have these views who aren't conflicted and trying to hype up the companies, you know.”

Claim: Many people who are not conflicted and are not trying to promote AI companies nevertheless hold views about AI risks.

“But then but then. To your point about the hype, my take on what's been happening is that DeepMind, OpenAI, and Anthropic have been full of people who were thinking about superintelligence from the very beginning at the highest levels of leadership,”

Claim: According to the speaker, DeepMind, OpenAI, and Anthropic had senior leaders thinking about superintelligence from the outset.

“and therefore they have been considering... at least somewhat, both the loss of control risk and the concentration of power risk, you know, who gets the control of the AIs from the beginning.”

Claim: The speaker says leaders at DeepMind, OpenAI, and Anthropic had considered, at least to some extent, both AI loss-of-control risks and risks from concentrating power over AI.

“And this is documented in all sorts of ways. You can look at their old writings and so forth, some of the leaked emails.”

Claim: The speaker says old writings and leaked emails document that people at these AI companies considered AI risks.

“So loss of control is what if we don't solve the alignment problem in time and the AIs take over?”

Claim: The speaker describes AI loss-of-control risk as the possibility that AIs take over if the alignment problem is not solved in time.

“And then concentration of power is if we do solve the alignment problem, like who controls the AIs? What goals do we put in them?”

Claim: The speaker says that even if AI alignment is solved, concentration-of-power concerns remain about who controls AIs and what goals they are given.

“You know, do we risk becoming a dictatorship or some sort of crazy oligarchy or whatever?”

Claim: The speaker warns that concentrated control of AI could risk leading to a dictatorship or oligarchy.

“So you can find writings from people at these companies, both senior researchers and also the CEOs and so forth, going back decades talking about these things.”

Claim: The speaker says writings by senior researchers and CEOs at the companies have discussed AI loss-of-control and concentration-of-power concerns for decades.

“The key here is, I guess I would say. In the single word, it'd be rationalization.”

Claim: The speaker characterizes the reasoning used to justify building advanced AI despite recognized risks as rationalization.

“So there's been this very seductive argument that has appealed to all of these people, which is basically, well, it's probably going to happen anyway.”

Claim: The speaker says a persuasive justification for building advanced AI is that it will probably be developed regardless.

“If we don't do it, someone else will.”

Claim: The speaker says a common justification for building advanced AI is that another actor will build it if one's own group does not.

“And it's better for us to do it first than for someone else to do it because we're the responsible good guys who will wisely solve all the safety issues and then also beneficently, you know, give UBI or whatever.”

Claim: The speaker says proponents justify racing to build advanced AI by believing their group is more responsible, can solve safety issues, and will provide benefits such as universal basic income.

“to make sure that everything works out well so basically all of these people sort of trust themselves more than they trust everyone else and have therefore convinced themselves that even though these risks are real the best way to deal with them is for them to go as fast as possible and win the race and this is you know deep mind's plan so to speak demis sabas his plan was basically be there get there first with this big corporation google and then because you have such a lead over everybody else you can sort of slow down and get all the safety stuff right and sort of make sure that everything goes well before everybody else catches up.”

Claim: The speaker says DeepMind's strategy, associated with Demis Hassabis, was to reach advanced AI first with Google's backing, then use its lead to slow down and address safety before competitors caught up.

“to make sure that everything works out well so basically all of these people sort of trust themselves more than they trust everyone else and have therefore convinced themselves that even though these risks are real the best way to deal with them is for them to go as fast as possible and win the race and this is you know deep mind's plan so to speak demis sabas his plan was basically be there get there first with this big corporation google and then”

Claim: The speaker says AI developers who trust themselves more than others have concluded that, despite real risks, they should move as fast as possible and win the race to build AI.

“That plan was torpedoed when you know Elon and Sam and Ilya made OpenAI.”

Claim: The speaker says DeepMind's first-mover plan was undermined when Elon Musk, Sam Altman, and Ilya Sutskever founded OpenAI.

“Well they were worried that they didn't trust Demis to handle all that power responsibly.”

Claim: The speaker says OpenAI's founders created the organization partly because they distrusted Demis Hassabis's ability to handle concentrated AI power responsibly.

“when he was in charge of the AI project you know the only you know when all the fate of the world rested in his hands so they wanted to create open AI to be this countervailing force that could do it right and make it you know distributed to everybody and not concentrate power so much in Demis's hands and in fact they were the leaked emails or the emails that came up in the lawsuit they were talking about how they were worried that Demis would become dictator using AGI”

Claim: The speaker says OpenAI was created to counterbalance Demis Hassabis's AI project, distribute AI more broadly, and avoid concentrating power in his hands.

“when he was in charge of the AI project you know the only you know when all the fate of the world rested in his hands so they wanted to create open AI to be this countervailing force that could do it right and make it you know distributed to everybody and not concentrate power so much in Demis's hands and in fact they were the leaked emails or the emails that came up in the lawsuit they were talking about how they were worried that Demis would become dictator using AGI”

Claim: The speaker says emails disclosed in a lawsuit expressed concern that Demis Hassabis might use AGI to become a dictator.

“we can see how well that's worked out, you know, all the anthropic people basically split off from OpenAI because they didn't think OpenAI was going to handle the safety stuff responsibly.”

Claim: The speaker says Anthropic's founders or early members left OpenAI because they doubted OpenAI would manage AI safety responsibly.

“So then they're claiming that like they have the technical talent and they'll be able to like sort out the alignment issues better than everyone else, you know, it's a mess.”

Claim: The speaker says Anthropic claims its technical talent will enable it to solve AI alignment issues better than other organizations.

“definitely should not.”

Claim: The speaker says people definitely should not trust any of the AI labs under discussion.

“I think pushing for things like transparency as well as the government having people whose job it is to be keeping track of these, coming up with contingency plans, interviewing these labs for their plans and coming up with an eternal assessment to their probability of success, all of that seems useful, and I think at least some of the players here would probably be willing to push for that type of stuff, and I think others wouldn't.”

Claim: The speaker considers transparency, government oversight and contingency planning, interviews with AI labs about their plans, and ongoing assessment of their likelihood of success useful measures.

“all of that seems useful, and I think at least some of the players here would probably be willing to push for that type of stuff, and I think others wouldn't.”

Claim: The speaker thinks some AI developers would probably support transparency and government oversight measures, while others would not.

“And so I think it's are they willing to help solve these collective action problems or are they going to continue defecting and”

Claim: The speaker says a key question is whether AI developers will cooperate to address collective-action problems or continue to defect.

“when these companies were fresh and young and idealistic their their founding mythology was basically yes the risks are real and that's why you should come work at our company because we're the good guys and so when that was still fresh and still like you know the main thing they were saying they were talking about it a lot but then now when their their sort of like founding myth is sort of like kind of laughable and it's like very much not something they can say with a straight it's eroded it's”

Claim: Early AI companies presented their founding narrative as acknowledging real risks and inviting people to join because the companies were the good guys.

“when these companies were fresh and young and idealistic their their founding mythology was basically yes the risks are real and that's why you should come work at our company because we're the good guys and so when that was still fresh and still like you know the main thing they were saying they were talking about it a lot but then now when their their sort of like founding myth is sort of like kind of laughable and it's like very much not something they can say with a straight it's eroded it's coded some it's not something they can really save with a straight face so much anymore and also there are lots of under lots of political pressure you know to get investors and to ward off regulation and stuff like that so so the rationalization wheels are continuing to turn and they're coming up with new narratives to justify what they're doing you”

Claim: AI companies' founding narrative that they were the good guys addressing real risks has eroded and is no longer as credible for them to present publicly.

“coded some it's not something they can really save with a straight face so much anymore and also there are lots of under lots of political pressure you know to get investors and to ward off regulation and stuff like that so so the rationalization wheels are continuing to turn and they're coming up with new narratives to justify what they're doing you”

Claim: AI companies face political pressure to attract investors and avoid regulation, prompting them to develop new narratives that justify their actions.

“So maybe we will maybe won't. The mechanisms of open sourcing or at least open weight models, which means that, you know, essentially anybody can get in this game to some degree.”

Claim: Open-weight models allow essentially anyone to participate in AI development to some degree.

“well, like hypothetically, suppose that we achieved AGI, someone achieved AGI and immediately opened the weights for everybody. It's not going to happen like that.”

Claim: The speaker considers it unlikely that an organization would release AGI model weights to everyone immediately upon achieving AGI.

“But suppose it did happen like that. Then for a brief glorious moment, anybody with enough GPUs could be able to run their own AGI right at the frontier of capability.”

Claim: In the hypothetical case that AGI weights were released to everyone, anyone with sufficient GPUs could briefly run AGI at the frontier of capability.

“AGI isn't like, you know, there's going to be AGI plus and AGI plus plus and so forth. And whoever's going to get to AGI plus is going to be the one who had the most GPUs so they can run the AGIs to do the research fastest,”

Claim: In an intelligence-explosion scenario, the actor with the most GPUs would be expected to advance beyond AGI first by using more AGI instances to conduct research faster.

“gave everybody exactly the same starting point at the same level of capability, the people who had more GPUs would pull ahead slowly but surely over everybody else,”

Claim: If all actors began with equal AI capability, actors with more GPUs would gradually gain an advantage over the others.

“So I think that there's this unfortunate sort of inherent I don't know if I want to say winner take all effect, but there's like this turn to scale sort of thing inherent in the dynamics of an intelligence explosion.”

Claim: The dynamics of an intelligence explosion inherently favor increased scale, though the speaker is unsure whether to characterize the effect as winner-take-all.

“And that's part of what makes this so scary.”

Claim: The inherent pressure toward scale in an intelligence explosion is one reason the speaker finds the scenario frightening.

“all the actors are incented to push it as fast as possible and to have the maximal resources in order to do so.”

Claim: AI actors have incentives to accelerate development and secure as many resources as possible for that purpose.

“And they're also incentivized not to open it up and, you know, to not be transparent about it and so forth, unfortunately.”

Claim: AI actors have incentives to keep their systems closed and avoid transparency.

“I think you're a little bit more optimistic and I'm with Daniel and being a little bit less optimistic about transparency. I'm especially... Especially less optimistic about transparency as we get closer to having actual AGI or let's say differentiated AI.”

Claim: The speaker expects transparency to be less likely as AI systems approach AGI or become meaningfully differentiated.

“So right now I would say that what's there is not very differentiated. Everybody's kind of using LLMs with reasoning models and so forth.”

Claim: At the time of the conversation, AI systems were not very differentiated, with most actors using LLMs and reasoning models.

“There's some differentiation in how people do their RL and what their data sets are. But I think there's not so much value in transparency until somebody has something that. Something that really is unique that they think other people aren't going to just reconstruct very rapidly. Um and as one gets to that point of having some unique piece of intellectual property, which I think will happen, um there's even more reason to be less transparent.”

Claim: The speaker believes transparency has limited value to an AI organization until it develops a unique capability that competitors cannot rapidly reproduce.

“There's some differentiation in how people do their RL and what their data sets are. But I think there's not so much value in transparency until somebody has something that.”

Claim: AI organizations differ to some extent in their reinforcement-learning methods and datasets.

“Something that really is unique that they think other people aren't going to just reconstruct very rapidly. Um and as one gets to that point of having some unique piece of intellectual property, which I think will happen, um there's even more reason to be less transparent.”

Claim: The speaker expects AI organizations to develop unique intellectual property, increasing their incentive to limit transparency.

“So I I I think I'd be optimistic about being transparent about the numbers of the best models internally, not necessarily the methods that were used to create them. or the weights for them.”

Claim: The speaker is optimistic that AI organizations may disclose their best models' performance numbers internally while withholding the methods and model weights used to produce them.

“But at least the public knowing uh uh what's going on um the peak capabilities that the models are exhibiting and then being aware of that. Um I I think it's good and that there's more tractability for this than a lot of other asks.”

Claim: The speaker considers it desirable for the public to be informed about the peak capabilities demonstrated by AI models.

“Um I I think it's good and that there's more tractability for this than a lot of other asks.”

Claim: The speaker believes that obtaining public disclosure of AI models' peak capabilities is more feasible than many other policy requests.

“But I'm just saying that they're not going to do it by default. Like if nobody asked them to be transparent about these things, then”

Claim: AI companies are unlikely to disclose model capability information voluntarily unless someone asks them to do so.

“So, I mean, we can agree that like voluntary regs, self-regulation is probably not enough to get to that point.”

Claim: The speaker considers voluntary regulation and industry self-regulation probably insufficient to ensure disclosure of AI model capabilities.

“I will make the argument that we've made some progress towards AGI and very little towards alignment, and we can see if we agree on that.”

Claim: AI has made some progress toward AGI and very little progress toward alignment, according to the speaker.

“which is to say that you could make a simple prediction. You could say, I think AGI will be here in 2027 or 2039 or whatever. But I think we all understand that that's the unsophisticated thing to do. You can certainly say your best guess for which particular year you think it might. might come but as people who are either scientists or know something about science we know there's what we call confidence interval around that so you know it might be this plus or minus that the most sophisticated thing to do is to actually draw out a curve and say you know i think some of the probability mass will come you know before 2027 and some of it will come before 2037 and some of it will come after in your i guess it's sort of an appendix to your ai 2027”

Claim: A more sophisticated AGI forecast represents probability across multiple possible arrival years rather than naming only one predicted year.

“But for those things, let's say for super intelligence, I was thinking 50%. 50% chance by the end of 2027. And that's why AI 2027 depicts it happening at the end of 2027 is because I was sort of illustrating my median projection.”

Claim: While writing AI 2027, the speaker assigned a 50% probability to superintelligence arriving by the end of 2027 as his median projection.

“Now the other people at AI Futures Project tended to be somewhat more optimistic than me. They tended to think it would take longer to get to superintelligence. They thought it would take longer, like 2029, 2031, something like that.”

Claim: Other AI Futures Project members generally forecast superintelligence later than the speaker, with estimates around 2029 or 2031.

“by the time we actually published it, I had lengthened my timeline somewhat. So these days, I would say 50% by end of 2028.”

Claim: At the time of the conversation, the speaker's median forecast gave superintelligence a 50% chance of arriving by the end of 2028, after he lengthened his timeline before AI 2027 was published.

“But in terms of what the shape of our distributions look like, they tend to have a sort of hump in the next five years and then like a long tail.”

Claim: The speaker's forecasts for AI timelines tend to place a concentration of probability in the next five years, followed by a long tail.

“the pace of AI progress has been quite fast over the last 15 years.”

Claim: AI progress has been quite fast over the 15 years preceding the conversation.

“And we understand something about the reasons for why it's been so fast and basically the reason is scale.”

Claim: The speaker attributes the rapid pace of AI progress primarily to scale.

“So they've been scaling up compute, they've been scaling up data, they've been drawing in ever more researchers into the field, especially compute. is probably the most important input that they're scaling up.”

Claim: AI developers have scaled compute and data and drawn more researchers into the field; the speaker considers compute probably the most important input being scaled.

“And that's sort of turbocharged progress.”

Claim: The speaker says scaling has substantially accelerated AI progress.

“But they simply won't, they the companies, simply won't be able to scale things up at the same pace after a couple years of that.”

Claim: The speaker predicts that AI companies will be unable to maintain their current pace of scaling after a couple of years.

“It won't be like a sharp cutoff, but it'll be a couple things. So partly it'll be you know power supplies partly it'll just be um compute production like much of the world's even after building new fabs and even after converting much of the world's chip production into uh ai chips uh they won't they'll have to like produce 10 times more fabs in order to scale up by 10 times right whereas previously they could just take chips designed for gaming and repurpose them for ai so so in a bunch of little ways that are going to add up there's going to be all these friction friction”

Claim: The speaker expects power-supply limits and compute-production constraints to impede scaling: after repurposing available chip production and building new fabs, a tenfold increase in scale would require ten times more fabs.

“you know power supplies partly it'll just be um compute production like much of the world's even after building new fabs and even after converting much of the world's chip production into uh ai chips uh they won't they'll have to like produce 10 times more fabs in order to scale up by 10 times right whereas previously they could just take chips designed for gaming and repurpose them for ai so so in a bunch of little ways that are going to add up there's going to be all these friction friction options that's going to start to bite that will make it harder for them to continue the crazy exponential rate of scale up”

Claim: The speaker expects multiple accumulating frictions to make it harder for AI companies to sustain an extremely rapid exponential rate of scaling.

“I don't know the actual numbers, but let's say that GPT-2 used maybe 10% of the internet or something like that, or 5% or something like that.”

Claim: The speaker estimated, with uncertainty, that GPT-2 used about 5%–10% of the internet.

“Maybe you guys know the actual numbers. And GPT-3 used a significantly larger fraction.”

Claim: GPT-3 used a significantly larger fraction of the internet than GPT-2.

“GPT-4 used most of the internet, including... transcriptions of videos and stuff like that.”

Claim: GPT-4 used most of the internet, including video transcriptions.

“And so you can't just keep 100x-ing that because there just isn't enough data.”

Claim: The speaker argued that current AI approaches cannot keep increasing their data by 100 times because the available data is insufficient.

“There's new data generated every day, you know, so you can always eke out a little more and people are turning to augmented data as well they should,”

Claim: Because new data is generated every day, researchers can continue to obtain somewhat more data.

“but that's not a kind of universal solvent. It works better for things like math where you can verify that the augmented data are better.”

Claim: Augmented data is not a universal solution; it works better for tasks such as mathematics, where the quality of the augmented data can be verified.

“And so I think we're already running against. and that kind of like bottleneck on one of the, let's say, raw resources that go in at least into the current approaches.”

Claim: The speaker thinks current AI approaches are already encountering a bottleneck in at least one of their raw resources.

“And another thing I would add is that you can also just think about money, which is which can use to buy many of these things.”

Claim: Money can be used to purchase many of the resources needed for AI development.

“And you can say, well, they've been scaling up the amount of money that they're spending on AI research and on training runs in particular over the last decades, but they it'll be hard for them to continue scaling at the same pace. You know, they're probably already doing something like.”

Claim: Although spending on AI research and training runs has increased over the last decades, the speaker thinks it will probably be difficult to keep increasing it at the same pace.

“And you can say, well, they've been scaling up the amount of money that they're spending on AI research and on training runs in particular over the last decades, but they it'll be hard for them to continue scaling at the same pace. You know, they're probably already doing something like. thing like billion dollar training runs,”

Claim: The speaker estimated that AI companies are probably already conducting training runs costing around one billion dollars.

“you know if it goes at the same pace what was the biggest training run in in 2020 was like what like three million dollars something like that you know four five million dollars so so they've gone up by like two and a half orders of magnitude in five years there was another two and a half orders of magnitude we're doing a 500 billion dollar training run in 2030 like like”

Claim: The speaker suggested that if training-run costs continued rising at the same pace, they could reach $500 billion by 2030, following an increase of about 2.5 orders of magnitude over five years.

“there's just not enough money in the world you know like the tech companies just won't be able to afford it even if they've grown bigger than they are today”

Claim: The speaker argued that the world does not have enough money for technology companies to afford training-run spending at the projected scale, even if those companies grow larger than they are today.

“the economy just won't be able to afford it and so so that's why we predict that like”

Claim: The speaker argued that the economy will not be able to afford AI training-run spending at the projected scale.

“If you don't get to some sort of radical transformation, if you don't get to some sort of crazy AI-powered automation of the economy by the end of this decade, then there's going to be a bit of an AI winter. There's going to be, at the very least, a sort of tapering off of the pace of progress.”

Claim: The speaker predicted that without a radical transformation—such as AI-powered automation of the economy by the end of this decade—AI progress will at least slow, and may enter an AI winter.

“And then that sort of stretches out a lot of probability mass into the future because that's a very different world. It's like, you know, it could take forever, not forever, it could take a quite long time to get to AGI once you're in that regime.”

Claim: The speaker said that a slowdown in AI progress would shift substantial probability toward later AGI timelines, and that reaching AGI in that regime could take quite a long time.

“we sort of ran into that bottleneck, I think maybe two or so years ago.”

Claim: The speaker thinks AI development may have encountered a data bottleneck roughly two years ago.

“And the main things that have been continuing the pace would be this, you know, thinking mode type of stuff outside of the trends that were existing previously. So I think it's sort of picking up slack in some ways for the. the fact that most of the internet's been trained on.”

Claim: The speaker thinks thinking-mode approaches have helped sustain AI progress beyond earlier trends, partly compensating for the fact that most of the internet has already been used for training.

“these curves represent our subjective judgment, which is very uncertain. It's just our opinions,”

Claim: The speaker said the forecasting curves reflect subjective, highly uncertain opinions.

“I guess the way I would like to say it should be done is rather than just sort of pulling a number out of your ass, so to speak, you should come up with models and look at trends and then, you know, have little calculations that attempt to give numbers.”

Claim: The speaker recommends grounding forecasts in models, trends, and calculations instead of selecting a number without analysis.

“And then you should stare at all of that and then pull a number out of your ass based on all that stuff that you've just looked at, you know. And so that's what we did.”

Claim: The speaker said their forecasting process involved reviewing models, trends, and calculations before making a judgment about the final number.

“And the main arguments and pieces of evidence that we found moving to inform our overall estimates were what we would call the benchmark and plus gaps argument.”

Claim: The speaker said the “benchmark and plus gaps” argument was among the main evidence and arguments informing their overall estimates.

“It's called bioinkers because it references the human brain and I think that part's actually the less exciting and plausible part of it.”

Claim: The speaker considers the BioAnchors framework's human-brain reference less exciting and less plausible than its core idea.

“The part that I think is more robust and more worth using is this core idea that you can think of this trade-off between more time to come up with new ideas and do AI research and more compute with which to do the AI research. And you can sort of think...”

Claim: The speaker considers the BioAnchors framework's more robust and useful core idea to be a trade-off between additional time for AI research and additional compute for AI research.

“You can make a big two-dimensional plot and you can imagine, 10 more years, 20 more years, 30 more years, how does the probability that we get to AGI go up with more time?”

Claim: The speaker proposed assessing how the probability of reaching AGI changes with 10, 20, or 30 more years.

“But you can also imagine... Not more time, but just more compute. Could we get to AGI today if we had, you know, five orders of magnitude more compute, 10 orders of magnitude more compute, 30 orders of magnitude more compute, right?”

Claim: The speaker proposed assessing whether AGI could be achieved today with five, 10, or 30 orders of magnitude more compute.

“Like, for example, if you had 10 to the 45 floating point operations, you could do a training run that's basically just simulating the entire planet Earth and all life evolving on it. on it for a billion years you know with that amount of compute and and and the thought there is that you don't really need to understand how intelligence works at all if you're building it with that type of training run because there's no insight coming from you it's you're just sort of letting nature do its thing and letting evolution take its course and so so the thought is that we can make those not guaranteed but like a soft upper bound at something like 10 to the 45 and then you can make other sort of soft upper bounds you can think”

Claim: A training run using 10^45 floating-point operations could simulate Earth and all life evolving on it for one billion years.

“on it for a billion years you know with that amount of compute and and and the thought there is that you don't really need to understand how intelligence works at all if you're building it with that type of training run because there's no insight coming from you it's you're just sort of letting nature do its thing and letting evolution take its course and so so the thought is that we can make those not guaranteed but like a soft upper bound at something like 10 to the 45 and then you can make other sort of soft upper bounds you can think”

Claim: A training run that simulates Earth and life evolving for a billion years would not require its designers to understand how intelligence works, because it would let nature and evolution take their course.

“on it for a billion years you know with that amount of compute and and and the thought there is that you don't really need to understand how intelligence works at all if you're building it with that type of training run because there's no insight coming from you it's you're just sort of letting nature do its thing and letting evolution take its course and so so the thought is that we can make those not guaranteed but like a soft upper bound at something like 10 to the 45 and then you can make other sort of soft upper bounds you can think”

Claim: The speaker considers 10^45 floating-point operations a possible soft upper bound, not a guarantee, for the compute needed to produce intelligence through a nature-like training run.

“well what could we do with 10 to the 36 and you can lay I wrote a blog post about this in 2021 you know suppose we had 10 to the 36 flop what are some like really huge types of training runs we could do and then like what's our guess as to how likely that is to work and what you can do is you can sort of”

Claim: In a 2021 blog post, the speaker considered possible large training runs using 10^36 floating-point operations and estimated their likelihood of success.

“You can start to smear out your probability mass over this dimension of compute. And so you sort of have a soft upper bound, and then you have, of course, a lower bound, which is the amount that we already have done. We clearly haven't done it right now with this amount of compute. And so that gives you this smeared probability distribution over compute.”

Claim: The speaker's framework represents uncertainty about the compute required with a probability distribution across compute levels, bounded above softly and below by compute already used.

“but now we're also going to get new ideas. And so as new ideas come along, we're going to be able to come up with more efficient methods that allow us to train it with less compute. So you can think of.”

Claim: The speaker expects future ideas to produce more efficient training methods that require less compute.

“And I think this is the right sort of like basic framework for calculating these sorts of timelines. But it's a sort of relatively abstract, like low information framework that doesn't really look at the details of the technology today and the details of the benchmarks. So I think it's like...”

Claim: The speaker considers a compute-based probability framework a useful basic prior for calculating AI timelines, while noting that it is abstract, low-information, and does not account for current technology or benchmark details.

“a good way to get your prior so to speak but then you should update based on actual trends on the benchmarks and so forth which is what I'm about to get to but the the reason why I mentioned this prior process is that 10 to the 45 floating point operations isn't actually that far away from where we are right now right now we're at like what like 10 to the 26 or something for the training runs and we're going to be crossing a few orders of magnitude”

Claim: The speaker recommends using the abstract compute framework as a prior and updating it with observed benchmark trends.

“a good way to get your prior so to speak but then you should update based on actual trends on the benchmarks and so forth which is what I'm about to get to but the the reason why I mentioned this prior process is that 10 to the 45 floating point operations isn't actually that far away from where we are right now right now we're at like what like 10 to the 26 or something for the training runs and we're going to be crossing a few orders of magnitude in the next couple years and so even if you just had like a an even if you just smeared out your probability mass with maximum uncertainty across the like”

Claim: The speaker estimates that current training runs use about 10^26 floating-point operations and predicts that they will increase by a few orders of magnitude within the next couple of years.

“orders of magnitude from where we are now to 10 to the 45 there'd be like you know a non-negligible amount that it's going to happen in the next few years and so like even on priors you should think it like decently plausible that it could happen by the end of the decade and then you should update your prior based on the actual evidence which i'll now get to so the actual evidence i would say let's look at agentic coding benchmarks that seems to me to be um”

Claim: Even under a maximally uncertain prior spread from current compute levels to 10^45 floating-point operations, the speaker considers reaching 10^45 in the next few years a non-negligible possibility and reaching it by the end of the decade decently plausible.

“The most informative thing to look at, and the reason for that is because I don't think that the fastest way to get to super intelligence is in a single leap where humans come up with the new paradigms in their own brains. I think rather it's going to be this more gradual process where humans automate more of the AI research process and then that gets us to the new paradigms fast.”

Claim: The speaker expects the fastest route to superintelligence to be gradual automation of more of the AI research process, rather than a single leap in which humans invent new paradigms unaided; he expects this automation to accelerate the arrival of new paradigms.

“The most informative thing to look at, and the reason for that is because I don't think that the fastest way to get to super intelligence is in a single leap where humans come up with the new paradigms in their own brains.”

Claim: The speaker considers agentic coding benchmarks the most informative evidence for assessing progress toward superintelligence.

“And so I think that the lowest hanging fruit as far as the AI research process is concerned that's going to automate. first is the coding.”

Claim: The speaker expects coding to be the first and easiest part of the AI research process to automate.

“So I'm looking to see when will we get to the point where the coding is basically all handled by LLM-like AI assistance.”

Claim: The speaker is assessing when LLM-like AI assistance will handle nearly all coding work.

“Places like Meter, METR, have been building these little coding environments, doing all these coding tasks. The companies themselves have been doing this, of course, because they are racing as fast as they can to get to this automated coder milestone.”

Claim: METR and AI companies have developed coding-task environments or benchmarks, with companies pursuing an automated-coder milestone in a race to advance quickly.

“So we look at those and we extrapolate trends on them and we forecast that, well, in the next couple of years, they're basically going to saturate. You know, we're going to have AIs that can just crush all of these coding tasks.”

Claim: The speaker forecasts that coding benchmarks will be nearly saturated within the next couple of years, with AI systems able to perform all the benchmark tasks very well.

“something to scoff at they're not just multiple choice questions they're like the sort of tasks that would take a human like four hours to do or eight hours to do but that's not the same thing as complete automated encoding so first we extrapolate so when they saturate the benchmarks and then we try to make our guess as to what the gap is between the first system that can completely saturate these benchmarks and the first system that can actually automate the coding and that's probably the more speculative part”

Claim: The speaker distinguishes benchmark saturation from complete coding automation and says estimating the time between the first system to saturate the benchmarks and the first system to automate coding is more speculative.

“something to scoff at they're not just multiple choice questions they're like the sort of tasks that would take a human like four hours to do or eight hours to do but that's not the same thing as complete automated encoding so first we extrapolate so when they saturate the benchmarks and then we try to make our guess as to what the gap is between the first system that can completely saturate these benchmarks and the first system that can actually automate the coding and that's probably the more speculative part”

Claim: The coding benchmark tasks are substantial programming tasks that could take a human four to eight hours, rather than simple multiple-choice questions.

“And my approach to this is more from the cognitive science.”

Claim: The speaker assesses AI progress primarily through cognitive-science problems rather than quantitative progress.

“I see a set of problems that a cognitive creature must solve, many of which I wrote in my 2001 book, The Algebraic Mind. And I don't feel like we've solved any of those problems despite the quantitative progress that we've made.”

Claim: Despite quantitative progress in AI, the speaker believes that none of the key problems a cognitive creature must solve have been solved.

“need. And those include generalizing outside the distribution, which I think still remains a huge problem. I think the apple paper was a good um, there are actually two apple papers I discovered today. Um but the apple paper with the tower of Hanoi stuff I think is an example of problems with distribution shift. I think we've seen many of them over the years. We see that these systems have trouble multipl doing multiplication with large numbers unless they ca”

Claim: The speaker believes that generalization outside the training distribution remains a major problem for current AI systems.

“which I think still remains a huge problem. I think the apple paper was a good um, there are actually two apple papers I discovered today. Um but the apple paper with the tower of Hanoi stuff I think is an example of problems with distribution shift. I think we've seen many of them over the years. We see that these systems have trouble multipl doing multiplication with large numbers unless they ca”

Claim: The speaker regards an Apple paper involving Tower of Hanoi as an example of AI systems struggling with distribution shift.

“which I think still remains a huge problem. I think the apple paper was a good um, there are actually two apple papers I discovered today. Um but the apple paper with the tower of Hanoi stuff I think is an example of problems with distribution shift. I think we've seen many of them over the years. We see that these systems have trouble multipl doing multiplication with large numbers unless they ca to call on tools and so forth. I think there's lots of evidence for that.”

Claim: The speaker says current AI systems struggle to multiply large numbers unless they can use tools.

“I think that there's a problem of distinguishing types and tokens that leads to bleed through when you're representing multiple individuals from some category that leads to hallucinations. So I wrote an essay recently about the hallucinations that I think it was ChatGPT made about my friend Harry Shearer who's a”

Claim: The speaker believes that AI systems' failure to distinguish types from tokens can cause details about individuals in the same category to blend together and produce hallucinations.

“well-known actor and it misnamed the roles of characters that he played in the movie Spinal Tap and said that he was British when he's American and so forth. And I think this blurring together that we see of hallucinations remains a problem. And I could go on with a list of others. I think there are several having to do with reasoning planning etc.”

Claim: The speaker believes that hallucinations caused by information blending remain a problem for AI systems, alongside unresolved problems in reasoning and planning.

“well-known actor and it misnamed the roles of characters that he played in the movie Spinal Tap and said that he was British when he's American and so forth. And I think this blurring together that we see of hallucinations remains a problem. And I could go on with a list of others. I think there are several having to do with reasoning planning etc.”

Claim: The speaker says ChatGPT hallucinated that Harry Shearer was British and misidentified the roles Shearer played in Spinal Tap, although Shearer is American.

“What would AI look like two years before we achieved AGI or ASI or something like that? Certainly two years before we achieved ASI, we would have full solutions to all of those things.”

Claim: The speaker predicts that two years before ASI is achieved, AI systems will have full solutions to all the cognitive problems he listed.

“If we specified an algorithm for something,”

Claim: The speaker expects an AI system to follow a specified algorithm for a task.

“we would expect the system to be able to follow it. Current systems can't even play chess reliably according to the rules. So, you know, O3 will not, sorry, O3 will. sometimes make illegal moves.”

Claim: The speaker says current AI systems, including O3, do not reliably follow chess rules and sometimes make illegal moves.

“It can't avoid illegal moves.”

Claim: The speaker says O3 is unable to avoid making illegal chess moves.

“Another thing I would expect is that current systems, when we're close, would be basically the equivalent of their domain-specific counterparts, or at least be close,”

Claim: The speaker expects AI systems near AGI to perform about as well as, or at least approach, domain-specific systems.

“AGI means artificial general intelligence.”

Claim: AGI is an abbreviation for artificial general intelligence.

“And I would say the reality is that domain-specific systems are actually much better than the general ones right now.”

Claim: The speaker says domain-specific AI systems currently perform much better than general-purpose AI systems.

“The only general ones we have are LLM-based.”

Claim: The speaker says that the only general-purpose AI systems currently available are based on large language models.

“AlphaFold is a very carefully engineered hybrid neurosymbolic system that far outperforms what you could get from a pure chatbot or something like that.”

Claim: The speaker describes AlphaFold as a carefully engineered hybrid neurosymbolic system that greatly outperforms a pure chatbot.

“So even sometimes very old systems will beat the modern domain general ones. On Tower of Hanoi...”

Claim: The speaker says that some very old AI systems can outperform modern general-purpose AI systems.

“uh Herb Simon solved it in 1957 with a classical technique that generalizes to arbitrary length whereas the LLMs do not generalize to arbitrary length and face problems and so I could go through more but the gist of it is I don't see the qualitative problems that I think need to be solved um and I'll just go to your 1045 10 to the 45th because it's really interesting to me um I wrote a piece once with Christophe”

Claim: The speaker does not see current AI systems solving the qualitative cognitive problems he considers necessary to address.

“uh Herb Simon solved it in 1957 with a classical technique that generalizes to arbitrary length whereas the LLMs do not generalize to arbitrary length and face problems and so I could go through more but the gist of it is I don't see the qualitative problems that I think need to be solved um and I'll just go to your 1045 10 to the 45th because it's really interesting to me um I wrote a piece once with Christophe”

Claim: The speaker says Herb Simon's 1957 classical technique for Tower of Hanoi generalizes to arbitrary lengths, while LLMs do not.

“stuff cook i don't know if you guys know him the neuroscientist um we wrote a science fiction essay it's the only published science fiction essay i've ever written and it was in a book that i wrote called the future of the brain which i guess we wrote in 2050 i was the editor and we wrote the epilogue to it in call it 2015 and so it was set in something like 2055 I think it was 2045 and the notion was there's a book about neuroscience that by that point we would actually have created an entire simulation of the brain but it would run slower than the human brain and would still not have taught us anything about how the brain really works. So we'd have this simulation,”

Claim: In a science-fiction scenario set around 2045, the speaker imagined that a complete brain simulation would run slower than a human brain and fail to explain how the brain works.

“but we wouldn't understand the principles of it. You know, we'd have a neuron by neuron simulation, maybe even a protein by protein simulation,”

Claim: The speaker's imagined brain simulation could reproduce the brain neuron by neuron and possibly protein by protein.

“And I do wonder with a 10 to 45, even if you sort of train on everything, would you have solved the distribution shift and would you have abstracted? principles that allow you to run efficiently and effectively and usefully and in new new domains and so forth i think it's a really interesting question i'd never thought of the 1045 even though i read at least some of your paper i admit i didn't get that level of detail”

Claim: The speaker questions whether training at a scale of 10^45, even on everything, would solve distribution shift and yield principles for efficient and useful performance in new domains.

“like there's one way to extrapolate on the basis of things like compute. like nobody can really do this in a closed form way.”

Claim: Extrapolations based on factors such as compute cannot be made in a fully closed-form way.

“like there's one way to extrapolate on the basis of things like compute. And I think you've done a masterful job of doing that as well as can be done and acknowledging that there's still an element of pulling things out of one's behind, which is true on, you know, any account, like nobody can really do this in a closed form way.”

Claim: The speaker believes that AI progress forecasting cannot be done in a fully closed-form way.

“I think Jan LeCun, who I often disagree about many things with, actually would be closer to mine. He would probably give a different set of litmus tests that he's looking for.”

Claim: The speaker thinks Jan LeCun's approach is closer to his own, though LeCun would probably use different litmus tests.

“We would both emphasize world models. We have slightly different ideas about world models,”

Claim: The speaker and Jan LeCun both emphasize world models, but have somewhat different ideas about what world models are.

“but he would say, I don't think we're close because we don't really have world models.”

Claim: According to the speaker's characterization of Jan LeCun's view, current AI is not close to the goal because it lacks genuine world models.

“And neither of us, I think, are satisfied with the current thing that some people call reasoning, but neither of us think is, you know, robust enough.”

Claim: The speaker and Jan LeCun are dissatisfied with current AI capabilities described as reasoning because they consider them insufficiently robust.

“And neither of us, I think, are satisfied with the current thing that some people call reasoning, but neither of us think is, you know, robust enough.”

Claim: The speaker says that he and Yann LeCun consider current AI systems' purported reasoning insufficiently robust.

“I think he and I both take an architectural approach or a cognitive approach.”

Claim: The speaker characterizes himself and Yann LeCun as taking architectural or cognitive approaches to AI.

“I think he and I both take an architectural approach or a cognitive approach.”

Claim: The speaker and Jan LeCun both approach AI through architecture or cognition.

“10 to the 45 scenario where you just sort of brute force evolve intelligence, you would not understand how it works at all.”

Claim: In a hypothetical scenario where intelligence is brute-force evolved using 10^45 units of effort, the resulting system would not be understood by its creators.

“you would have it and you could sort of take those evolved creatures out of their simulated environment. and then start you know plugging them into like chat products and stuff and using them in your economy um and they would be smart you know they evolved they built their own civilization in there you know so they're pretty smart and they're pretty good at generalizing and so forth um so you wouldn't understand how it works but you would still nevertheless have the ai system and and indeed that's kind of what i think is happening today for us like we don't understand how these ai systems work very well uh”

Claim: In the hypothetical, evolved simulated creatures could be deployed in chat products and the economy, and would be intelligent and capable of generalization after evolving and building a civilization in their simulated environment.

“and then start you know plugging them into like chat products and stuff and using them in your economy um and they would be smart you know they evolved they built their own civilization in there you know so they're pretty smart and they're pretty good at generalizing and so forth um so you wouldn't understand how it works but you would still nevertheless have the ai system and and indeed that's kind of what i think is happening today for us like we don't understand how these ai systems work very well uh”

Claim: The speaker thinks current AI development resembles the hypothetical process: people use AI systems without understanding very well how they work.

“just throwing giant blobs of compute at giant data sets and training environments and then toying out or playing around with them afterwards and seeing what they're good at you know um but”

Claim: Current AI development involves applying large amounts of compute to large datasets and training environments, then experimenting with the resulting systems to assess their capabilities.

“I think part of my darkness about it is I think there's too much alchemy and not enough principles.”

Claim: The speaker's pessimism about AI stems partly from believing AI development relies too much on trial-and-error alchemy and too little on principles.

“100 and this is and this is part of why the like alignment issue feels so looming to me is that like we don't even know what we're doing how are we supposed to how are we supposed to craft a mind that has you know, the right virtues and the right principles and so forth, when we don't even like, right, anyhow,”

Claim: The speaker considers AI alignment a looming problem because humans do not understand what they are doing well enough to reliably create minds with the desired virtues and principles.

“you know, I also can't solve the Tower of Hanoi, and I also can't do large math problems in my head. I need cools. I need to be able to like programme a little bit or like, you do it, use a computer.”

Claim: The speaker cannot solve the Tower of Hanoi or perform large mathematical problems mentally without tools, and relies on programming or a computer for such tasks.

“One is that, you know, relatively young children can actually do Tower of Hanoi really well if they care about it. So um”

Claim: Relatively young children can perform well on the Tower of Hanoi when they are motivated to do it.

“Even pretty big ones. I mean, there's a video, I think, of a kid doing seven discs lightning fast, like in two minutes on YouTube. A kid who enjoys it. He's probably 12 or 15 or whatever. I'm sure he can do eight discs if he wants to because it's a recursive algorithm.”

Claim: The speaker recalls a video of a child completing a seven-disc Tower of Hanoi puzzle in about two minutes and believes the child could probably complete eight discs because the solution is recursive.

“We humans do get into memory limitations.”

Claim: Humans have limitations in memory.

“You shouldn't expect that AGI should.”

Claim: The speaker argues that AGI should not be expected to share human memory limitations.

“We should pause for a moment on definition of AGI, It has had varied definitions.”

Claim: The term AGI has been defined in different ways.

“The one that I always imagine is it should be at least as good as humans in a bunch of things and better in certain ones. So I would not be satisfied with an AGI that can't do arithmetic.”

Claim: The speaker's preferred conception of AGI is a system at least as capable as humans across many tasks and superior in some areas.

“The one that I always imagine is it should be at least as good as humans in a bunch of things and better in certain ones. So I would not be satisfied with an AGI that can't do arithmetic.”

Claim: The speaker would not consider a system that cannot perform arithmetic to be satisfactory as AGI.

“it's equivalent to people in this respect, but I would actually expect more and especially if we're talking about the risks that... we're concerning. I think we're really talking about a form of AGI that probably can do all the short-term memory things that people can't and has the versatility and flexibility of humans.”

Claim: The speaker thinks the form of AGI relevant to the risks under discussion would probably exceed humans on short-term-memory tasks while matching human versatility and flexibility.

“You can argue about that. And, you know, I would say that like the weakest AGI would be it's as good as Joe Sixpack, who's really not very good at reasoning, is full of confirmation bias,”

Claim: The speaker suggests that the weakest plausible AGI could perform at the level of an ordinary person, who may reason poorly and exhibit substantial confirmation bias.

“You know, critical reasoning, you'd say, okay, that's AGI because it does what Joe Sixpack does. I think most of the safety arguments are really around AGI that's at least as smart as like, you know, most smart people.”

Claim: The speaker thinks most AI safety arguments focus on AGI that is at least as intelligent as most smart people.

“Smart people can, in fact, do things like Tower of Hanoi. And certainly, you know, another example I gave you was chess, right? Six-year-olds can learn to follow the rules of chess.”

Claim: The speaker says that smart people can solve tasks such as the Tower of Hanoi and that six-year-olds can learn chess rules.

“03 will do things like have a queen jump over a knight, which is not possible.”

Claim: The speaker says GPT-3 has produced an illegal chess move in which a queen jumps over a knight.

“03 will do things like have a queen jump over a knight, which is not possible. And so, you know, the failures there are quite striking and not something you need even an expert to do.”

Claim: The speaker considers GPT-3's chess-rule failures striking and detectable without expert knowledge.

“Yeah, so I guess I would say it feels to me like we're making progress on these things, or at least that these barriers are not going to be... uh barriers for long uh i think for example that maybe the”

Claim: The speaker feels AI is progressing on the discussed limitations and expects those limitations not to remain barriers for long.

“well let me tell you more let me tell you why i think this so so i think that yeah maybe like the transformer by itself might have trouble doing a lot of this but you should think about the transformer plus the system of tools and plugins that you can build around it like claude plus code interpreter and things like that um uh and i think that that system could solve tarif and i and things like that because claude”

Claim: The speaker thinks a transformer alone might struggle with the discussed tasks, while a system combining a transformer with tools and plugins could solve them.

“well let me tell you more let me tell you why i think this so so i think that yeah maybe like the transformer by itself might have trouble doing a lot of this but you should think about the transformer plus the system of tools and plugins that you can build around it like claude plus code interpreter and things like that um uh and i think that that system could solve tarif and i and things like that because claude club might be smart enough to to look up the algorithm and then implement the algorithm and then you know do it um and so so i actually i'm curious for your immediate reaction to that point”

Claim: A transformer combined with tools and plugins, such as Claude with a code interpreter, might solve tasks like Tariff and I even if a transformer alone struggles with them.

“club might be smart enough to to look up the algorithm and then implement the algorithm and then you know do it um and so so i actually i'm curious for your immediate reaction to that point”

Claim: Claude might be capable of finding an algorithm, implementing it, and executing it.

“club might be smart enough to to look up the algorithm and then implement the algorithm and then you know do it um and so so i actually i'm curious for your immediate reaction to that point”

Claim: The speaker thinks Claude might be capable of finding an algorithm and implementing it to solve the discussed tasks.

“I've always advocated neurosymbolic AI,”

Claim: The speaker has advocated neurosymbolic AI since at least 2001.

“and I think that that is actually a species of neurosymbolic AI. I don't think it's the right one,”

Claim: The speaker regards language models using tools as a form of neurosymbolic AI, though not the right form.

“the arguments that I made going back to 2001 was. that you need symbols in the system to do abstract operations over variables. There were several other arguments, but that was the core argument. And what you're doing when you have Claude call an interpreter is you are doing operations over variables in the Python that it creates. And so you're you know the you're moving into a different part of the system. So the pure neural networks don't have operations over variables and fail on all of these things.”

Claim: The speaker argued by 2001 that AI systems need symbols to perform abstract operations over variables.

“that you need symbols in the system to do abstract operations over variables. There were several other arguments, but that was the core argument. And what you're doing when you have Claude call an interpreter is you are doing operations over variables in the Python that it creates. And so you're you know the you're moving into a different part of the system. So the pure neural networks don't have operations over variables and fail on all of these things.”

Claim: Pure neural networks lack operations over variables and fail at tasks that require such operations.

“that you need symbols in the system to do abstract operations over variables. There were several other arguments, but that was the core argument. And what you're doing when you have Claude call an interpreter is you are doing operations over variables in the Python that it creates. And so you're you know the you're moving into a different part of the system. So the pure neural networks don't have operations over variables and fail on all of these things.”

Claim: When Claude uses an interpreter, the resulting system performs operations over variables in the Python code Claude generates.

“things. Symbolic systems can do them fine.”

Claim: Symbolic systems can perform operations over variables that pure neural networks fail to perform.

“And here you're using the neural network to create the symbolic system that you need in order to solve that particular problem. So it's absolutely a neurosymbolic solution.”

Claim: A system in which a neural network creates the symbolic tools needed for a problem is a neurosymbolic solution.

“I think that the rate limiting step is that they don't always call the right code that they need if if you could make that solid enough that would be great”

Claim: A key bottleneck for tool-using AI systems is that they do not always call the appropriate code.

“you could think i'll just say a couple more sentences there um one of the first attempts at this strategy was to put an llm into wolfram alpha as the which is a totally symbolic system right um and not wolfram alpha the intimate mathematica um and the results were were you know hit or miss right and the problem was on the interface getting to the tools if you can really reliably get to the tools then you have a neuro symbolic system that works if you can have the neural networks reliably call the tools that they want um or that they to the problem i should say um then you're golden i think empirically it is”

Claim: An early attempt to combine an LLM with a symbolic system by integrating it into Wolfram Alpha produced inconsistent results.

“you know hit or miss right and the problem was on the interface getting to the tools if you can really reliably get to the tools then you have a neuro symbolic system that works if you can have the neural networks reliably call the tools that they want um or that they should be calling to the problem i should say um then you're golden i think empirically it is”

Claim: The early LLM–Wolfram Alpha approach was limited by the interface through which the model accessed tools.

“you know hit or miss right and the problem was on the interface getting to the tools if you can really reliably get to the tools then you have a neuro symbolic system that works if you can have the neural networks reliably call the tools that they want um or that they should be calling to the problem i should say um then you're golden i think empirically it is”

Claim: A neurosymbolic system can work if its neural networks reliably call the tools appropriate to the problem.

“It would be hard to get the tools to work reliably.”

Claim: Reliably getting AI systems to use tools is difficult.

“so there I would say it does seem to me like the AIs have been getting more reliable over the last couple of years.”

Claim: AI systems appear to have become more reliable over the past couple of years.

“so so the way I would interpret this graph and I'm sure many of our audience has already seen this but they have this you know they they have this suite of agentic coding tasks uh that are sort of organized from uh from shortest to longest in terms of like how long it takes to human beings to complete the tasks and then they note that you know the AIs of 2023 could generally speaking do the tasks from here to here but not do the tasks above this length but then each year like the crossover point has been lengthening and a very natural interpretation of what's going on here is that the AIs are getting more reliable you know if they've ever if they have a chance of getting into some sort of catastrophic error”

Claim: The agentic coding-task graph shows the task duration at which AI performance crosses over from success to failure increasing each year since 2023.

“from shortest to longest in terms of like how long it takes to human beings to complete the tasks and then they note that you know the AIs of 2023 could generally speaking do the tasks from here to here but not do the tasks above this length but then each year like the crossover point has been lengthening and a very natural interpretation of what's going on here is that the AIs are getting more reliable you know if they've ever if they have a chance of getting into some sort of catastrophic error”

Claim: The increasing task-duration crossover in the graph is evidence that AI systems are becoming more reliable.

“at any given point then you know if it's like a one percent chance per second then after like 50 seconds they're going to get into an error but then if that goes down by an order of magnitude then they can go for more seconds and so forth and so so the thought here i would say this is evidence that they are just getting generally speaking more reliable better at not only not making mistakes but recovering from the mistakes they make not infinitely better they're still less reliable than humans but”

Claim: AI systems are still less reliable than humans, even though their reliability is improving.

“at any given point then you know if it's like a one percent chance per second then after like 50 seconds they're going to get into an error but then if that goes down by an order of magnitude then they can go for more seconds and so forth and so so the thought here i would say this is evidence that they are just getting generally speaking more reliable better at not only not making mistakes but recovering from the mistakes they make not infinitely better they're still less reliable than humans but”

Claim: For an AI with a persistent per-second risk of catastrophic error, reducing that risk by an order of magnitude increases the time it can operate before an error occurs.

“at any given point then you know if it's like a one percent chance per second then after like 50 seconds they're going to get into an error but then if that goes down by an order of magnitude then they can go for more seconds and so forth and so so the thought here i would say this is evidence that they are just getting generally speaking more reliable better at not only not making mistakes but recovering from the mistakes they make not infinitely better they're still less reliable than humans but”

Claim: The graph is evidence, in the speaker's view, that AI systems are improving at avoiding errors and recovering from errors.

“but there's substantial progress being made year over year.”

Claim: AI reliability is improving substantially from year to year.

“with the particular graph, I think there's a lot of problems and some are actually relevant to this argument.”

Claim: The speaker believes the graph has several flaws, including flaws relevant to the argument about AI reliability.

“One is it was all relative to coding tasks. It wasn't tasks in general.”

Claim: The graph's benchmark covered coding tasks rather than tasks in general.

“I'm very concerned from a scientific perspective that we don't know how much data contamination there is, we don't know how much augmentation is done relative to those benchmarks and so forth, which leads me to my next point about it, which is, I can't, I'm blanking on the guy's name from 80,000 hours,”

Claim: The speaker is concerned that the extent of data contamination and augmentation in the coding benchmarks is unknown.

“looking at agents. And so the y-axis for the coding thing was sort of minutes to hours to days or whatever. And for agents it was like seconds to tens of seconds or something like that. So it was a totally different y-axis.”

Claim: A separate agent benchmark graph used a seconds-to-tens-of-seconds time scale, unlike the coding-task graph's minutes-to-hours-or-days scale.

“was some more, that was maybe like web browsing agents or something”

Claim: The speaker tentatively identifies the agents in the separate graph as possibly web-browsing agents.

“The argument was that you're getting the same kind of curve, but the scaling was completely different on the y-axis.”

Claim: The reliability-over-time curve uses a substantially different y-axis scale.

“What's there for me is the performance is still pretty poor on those agent things.”

Claim: Performance on agent tasks is still poor.

“More generally, I don't like the axis at all. I think that it's very arbitrary.”

Claim: The speaker considers the graph's axis very arbitrary.

“we give a bunch of examples, but like, how long does it take this task? And also, it's all to the point of 50% performance is just a very weird measure. And you can find many things that, you know, a human can actually do in three seconds that according to the graph,”

Claim: The speaker considers 50% performance an unusual and problematic measure for interpreting the graph.

“which is choose a legal move on a chessboard, You know, this takes a grandmaster. I don't know, 100 milliseconds or something like that.”

Claim: The speaker estimates that choosing a legal chess move takes a human about 100 milliseconds, while expressing uncertainty about the estimate.

“But O3 still can't do that task with 100% reliability.”

Claim: O3 cannot choose a legal chess move with 100% reliability.

“It can do maybe 80% reliability or something like that.”

Claim: The speaker estimates that O3 chooses a legal chess move with roughly 80% reliability.

“So it's just, it's weird to read anything off of this graph.”

Claim: The speaker considers the graph difficult to interpret reliably.

“One other thing I was going to mention, which is a new topic sort of, is that I think in a lot of these cases, my defense of the AI is that, well, they never were trained on this task.”

Claim: The speaker's defense of AI systems in many cases is that they were not trained on the task being evaluated.

“they fail on things that they're not trained on I mean that's an oversimplification but if we're talking about AI risk and”

Claim: The speaker cautions that explaining AI failures as failures on tasks the systems were not trained on is an oversimplification, while discussing AI risk.

“I understand that the really dangerous AIs of the future have to be able to do stuff without having trained on it.”

Claim: The speaker believes that genuinely dangerous future AI systems must be capable of performing tasks without having been trained on those tasks.

“At least I would say to the same extent that humans can do things without having trained on it. So to take your Grandmaster example,”

Claim: The speaker argues that dangerous future AI systems should generalize to untrained tasks at least as well as humans do.

“Grandmasters can do that task in less than three seconds with greater than 80% reliability.”

Claim: Grandmasters can choose a legal chess move in under three seconds with greater than 80% reliability.

“also they've trained on it a lot like they've done lots of that and whereas o3 possible i don't know that we don't know the details of o3's training process but it's possible that there was not a single playing of a game of chess at all in the training process”

Claim: The speaker says grandmasters have extensive chess practice, while O3 may have had no chess games in its training data; he notes that O3's training process is not known to him.

“So there's a case, may have seen by transcripts the way, they've of games of chess seen transcripts, they've seen the explicit rules in Wiki and know they've been trained on Wikipedia. They've probably read like Bobby Fischer played chess, which is how I learned. learn to play chess so they have books on chess and so they will be related read everything and what is the famous phrase publicly and privately available sources or whatever that wasn't quite the famous phrase but you get the idea so I think they probably actually have a lot of data on chess and just as a side note on transparency from the scientific perspective it's very hard to really evaluate these systems because we don't know what's in the training set either”

Claim: The speaker thinks AI systems probably received substantial chess-related training data from publicly available sources.

“they will be related read everything and what is the famous phrase publicly and privately available sources or whatever that wasn't quite the famous phrase but you get the idea so I think they probably actually have a lot of data on chess and just as a side note on transparency from the scientific perspective it's very hard to really evaluate these systems because we don't know what's in the training set either”

Claim: The speaker argues that scientific evaluation of AI systems is difficult because their training data is not transparent.

“Poor question going back to my work in 1998 is how do they generalize beyond what they've been trained on? And we just don't have transparency on that.”

Claim: Since at least 1998, the speaker has focused on how AI systems generalize beyond their training; he says transparency about this is lacking.

“so first of all, I totally agree about the transparency thing.”

Claim: The speaker agrees that transparency about AI systems is lacking.

“Let me just put one other example on the table in a similar flavor, which is I played Grok in tic-tac-toe and I came up with some variation.”

Claim: The speaker played tic-tac-toe with Grok and devised a variant of the game.

“and then I might write this up this weekend and so I asked some variation we played it and then it offered another variation which was let's only play tic-tac-toe where you can let's play tic-tac-toe where you can only win if your three in a row is on the edges and I said fine we had some moves back and forth and then it suggested some moves for me at a point where I could in fact make a three in a row so it had failed to develop the right strategy”

Claim: In a tic-tac-toe variant where winning required three in a row on the edges, Grok suggested moves that let the speaker make three in a row, showing a failure to develop the correct strategy.

“um i hadn't played this variation either before but it was obvious what to do so he failed to identify what three in a row was which i would say goes to a conceptual weakness like you should have had enough data to understand what three in a row is and so he it suggested other moves didn't recognize it it's a pretty bad failure and then we played again after i corrected it you know how you get these sycophantic answers i'm sorry and you know and so so we played again and it lost to me again and then”

Claim: After the speaker corrected Grok's mistake, Grok lost to the speaker in another game.

“um i hadn't played this variation either before but it was obvious what to do so he failed to identify what three in a row was which i would say goes to a conceptual weakness like you should have had enough data to understand what three in a row is and so he it suggested other moves didn't recognize it it's a pretty bad failure and then we played again after i corrected it you know how you get these sycophantic answers i'm sorry and you know and so so we played again and it lost to me again and then”

Claim: The speaker considers Grok's failure to understand the three-in-a-row condition in the tic-tac-toe variant a conceptual weakness.

“So if you move these things off the typical problem, they're even worse.”

Claim: The speaker says AI systems perform worse when evaluated on problems outside the typical cases.

“If they had a robust understanding of a domain like tic-tac-toe or chess, it wouldn't be too hard.”

Claim: The speaker argues that robust understanding of domains such as tic-tac-toe or chess should make related tasks relatively easy for AI systems.

“which was like I had some variation on chess like give me a board state where there are three queens on white side but black can immediately win and performance was not great so like you you take knowledge that for an expert would be out of what they've specifically practiced but they understand the domain well enough and experts will have no trouble at all and these systems still do”

Claim: The speaker reports that AI performance was poor on a chess task involving a position with three white queens and an immediate win for black.

“which was like I had some variation on chess like give me a board state where there are three queens on white side but black can immediately win and performance was not great so like you you take knowledge that for an expert would be out of what they've specifically practiced but they understand the domain well enough and experts will have no trouble at all and these systems still do”

Claim: The speaker argues that human experts can readily solve unfamiliar tasks within their domain because of their broader understanding, whereas AI systems still struggle with them.

“So I totally agree with those limitations of our current systems,”

Claim: The speaker agrees that current AI systems have limitations in generalizing to unfamiliar tasks within a domain.

“but the thing that I would say is that it seems like the trends are, you know, going up.”

Claim: The speaker says the relevant AI capability trends appear to be improving.

“So I would predict that if you measured this sort of thing for the last five years, even though the current systems might still have weaknesses, they would be less weak than the systems of two years ago, which are less weak.”

Claim: The speaker predicts that measures of AI-system weaknesses over the preceding five years would show improvement, with current systems less weak than systems from two years earlier, despite still having weaknesses.

“So I would predict that if you measured this sort of thing for the last five years, even though the current systems might still have weaknesses, they would be less weak than the systems of two years ago, which are less weak.”

Claim: The speaker predicts that measurements of these capabilities over the past five years would show current systems to be less weak than systems from two years earlier.

“And I think that has limitations, in particular a streetlight effect where, when it gets 100% that will suggest that it has the task. or that suggests it's solved the task.”

Claim: Benchmark-based forecasting has a streetlight-effect limitation: a model reaching 100% on a benchmark may appear to have solved the task, but that benchmark result can be misleading.

“The issue is that they often have some structural defects that are only obvious later in the, or when you go fairly far out into the curve.”

Claim: Some benchmarks have structural defects that become apparent only later or farther along the capability curve.

“in video understanding, if you looked at the benchmarks from a few years ago, they're totally at the top of them, but then you can come up a year later with several other sorts of benchmarks that challenge them.”

Claim: In video understanding, systems that topped benchmarks from a few years earlier could be challenged by different benchmarks introduced a year later.

“I think that there tends to be a gravitation toward what's very tractable for AI systems and where some interesting action is happening. That's a selection pressure on the sort of benchmarks.”

Claim: Benchmark selection tends to favor tasks that are tractable for AI systems and areas with active progress, creating selection pressure in the benchmarks used.

“If one is looking at cognitive tasks such as those that you would give... kids, if you're testing their intelligence, for instance, if you randomly sample many of those, the the models don't do that well.”

Claim: Models perform poorly on many randomly sampled cognitive tasks of the kind used to test children's intelligence.

“Maybe on you could m it'd be a double-digit percentage, maybe be almost on half of them they don't do that well. For instance, count the number of faces in this photograph. O3 can't do that very well.”

Claim: The speaker estimates that models may perform poorly on a double-digit percentage, possibly nearly half, of randomly sampled cognitive tasks, and says O3 does not count faces in photographs very well.

“Uh or connect the dots or fill in fill in the colours in this in this picture, uh just an example for for visual um ability. So”

Claim: O3 has difficulty connecting dots and filling in colors in images, illustrating limitations in visual ability.

“So there's, I don't think when Gary's pointing these out, it's just that he's just running the cherry picking program and what he's going to do is he's going to keep cherry picking and doing God of the gaps thing for AI until it basically is AGI.”

Claim: The speaker rejects the characterization that Gary is merely cherry-picking AI failures and using a “God of the gaps” argument that would continue until AI becomes AGI.

“I do think that there is a non adversarial distribution from the cognitive science angle, you would see many of these. these sorts of issues.”

Claim: The speaker believes that many of the cited AI limitations would appear on a non-adversarial distribution of tasks informed by cognitive science.

“And I think that there's potentially some speaking past each other in part because he's not viewing intelligence as some unidimensional thing entirely.”

Claim: The speaker thinks the disagreement may partly reflect talking past each other, because Gary does not regard intelligence as entirely one-dimensional.

“A lot might be correlated together, but there are various other mental faculties that are important that aren't really online that much.”

Claim: Although many mental faculties may be correlated, some other important faculties are not substantially present in current AI systems.

“much. Uh um we we saw with GPT um with the GPT series that by pre-training on a lot of uh text we got some sub-components of intelligence. We got reading writing ability and we got a lot of crystallized intelligence or acquired knowledge from that. Um uh and that process took several years. Um so it's it's not the case there's often people will sort of point out once it gets traction on it then it will solve it immediately.”

Claim: Pretraining the GPT series on large amounts of text developed reading and writing abilities and substantial crystallized intelligence, or acquired knowledge, over several years.

“much. Uh um we we saw with GPT um with the GPT series that by pre-training on a lot of uh text we got some sub-components of intelligence. We got reading writing ability and we got a lot of crystallized intelligence or acquired knowledge from that. Um uh and that process took several years. Um so it's it's not the case there's often people will sort of point out once it gets traction on it then it will solve it immediately. media to layers to solve it very shortly thereafter.”

Claim: The speaker argues that gaining traction on a capability does not mean AI systems will solve it immediately.

“mathematical reasoning would be a recent example with Minerva. Google's Minerva system got 50% on the math benchmark, some benchmark I made some while ago, in 2022.”

Claim: Google's Minerva scored 50% on a mathematics benchmark that the speaker says he had created before 2022.

“And I think we've only recently crushed it in 2025. So it took a good three years and I think the reading,”

Claim: The speaker says the mathematics benchmark was only recently surpassed in 2025, roughly three years after Minerva scored 50%.

“writing ability took already took um got to an interesting state and now relatively complete uh four years later I think crystallized intelligence as well is now relatively complete however there's still a variety of others I wouldn't expect visual processing ability including with for video to be taken care of in a year or two maybe audio audio ability would be taken care of in a year or two though uh we don't see almost any progress on long-term”

Claim: The speaker says GPT-series reading and writing abilities reached an interesting state and became relatively complete about four years later, and that crystallized intelligence is now relatively complete.

“already took um got to an interesting state and now relatively complete uh four years later I think crystallized intelligence as well is now relatively complete however there's still a variety of others I wouldn't expect visual processing ability including with for video to be taken care of in a year or two maybe audio audio ability would be taken care of in a year or two though uh we don't see almost any progress on long-term”

Claim: The speaker does not expect AI visual processing, including video, to be solved within one or two years, but thinks audio ability might be solved within that timeframe.

“already took um got to an interesting state and now relatively complete uh four years later I think crystallized intelligence as well is now relatively complete however there's still a variety of others I wouldn't expect visual processing ability including with for video to be taken care of in a year or two maybe audio audio ability would be taken care of in a year or two though uh we don't see almost any progress on long-term long-term memory um basically. Uh I think that uh they can't really that meaningfully maintain um a state across long periods of times over complex interactions. Uh and so once we start to get traction on that, then we can start to forecast that out. The reason this is um”

Claim: The speaker says AI systems have made almost no progress on long-term memory and cannot meaningfully maintain state across long periods of complex interactions.

“such as what we see in the RKG-I thing and Raven's Progressive Matrices test, that still seems very deficient as well. So I think when thinking about these, it's important to split up the one notion of cognition, because if there is a severe limitation on any of these dimensions,”

Claim: The speaker says AI fluid intelligence remains very deficient on measures such as the RKG-I task and Raven's Progressive Matrices.

“such as what we see in the RKG-I thing and Raven's Progressive Matrices test, that still seems very deficient as well. So I think when thinking about these, it's important to split up the one notion of cognition, because if there is a severe limitation on any of these dimensions, then the models will be fairly defective economically, or at least for many tasks for you.”

Claim: If AI models have a severe limitation on any cognitive dimension, they will be economically defective or unsuitable for many tasks.

“For instance, if a person doesn't have good long-term memory, it will be very difficult to enculturate them and teach them how to be productive in a working environment and load all that context.”

Claim: A person with poor long-term memory would be difficult to enculturate, train for workplace productivity, and provide with the necessary context.

“If they have very slow reaction time, that's also a problem. Or if they have low fluid intelligence, that will limit their ability to generalize substantially.”

Claim: Very slow reaction time is a problem, while low fluid intelligence substantially limits generalization ability.

“So I think the numbers are going up substantially or continually on many of these axes.”

Claim: The speaker thinks AI performance is rising substantially or continually across many cognitive dimensions.

“um afford these benchmarks. Uh so but at the same time when those benchmarks are at their conclusion, there might be another peak for those, and there's many of these, and for it to be an A_G_I_ you're gonna need all of them as well. And for some of them, they're fairly early on in their uh uh capabilities such as with uh long-term memory. Uh so I I I think um if one's doing forecasting, I think”

Claim: Even as benchmarks are completed, new capability peaks may emerge; achieving AGI would require all relevant cognitive dimensions, some of which, including long-term memory, remain at an early stage.

“understanding intelligence somewhat more and having a uh um uh a more sophisticated account such as one would find in cognitive science um can help um foresee bottlenecks that will actually be fairly action relevant.”

Claim: A more sophisticated, cognitive-science-informed account of intelligence can help forecasters identify bottlenecks that are relevant to action.

“are really very serious limits right now.”

Claim: Physical and spatial intelligence are serious current limitations for AI systems.

“So if you ask O3 to label a diagram, for example, it'll be quite poor at it.”

Claim: O3 performs poorly at labeling diagrams.

“If you ask it to reason about an environment, Yeah. it's gonna be poor at that.”

Claim: O3 performs poorly at reasoning about an environment.

“so I mean I agree with like you made it sound like I think intelligence is a single dimension I don't I don't know where you got that impression. I agree that there's all these limitations,”

Claim: The speaker rejects the view that he considers intelligence a single dimension and agrees that AI systems have multiple limitations.

“so yeah the benchmarks is you take all the benchmarks that you like and you extrapolate them and you try to see when they saturate and then the gaps is thinking about all the stuff that you just mentioned and thinking about how You know just because you've knocked down these benchmarks doesn't mean that you've already reached AGI There's all this other stuff that the benchmarks might not be measuring, you know, you have to sort of try to reason about that”

Claim: A benchmarks-and-gaps approach extrapolates selected benchmarks to estimate when they saturate and separately reasons about capabilities the benchmarks may not measure.

“You know just because you've knocked down these benchmarks doesn't mean that you've already reached AGI There's all this other stuff that the benchmarks might not be measuring,”

Claim: Passing benchmarks does not establish that AGI has been reached, because benchmarks may omit other relevant capabilities.

“Another way of putting my argument is I identified a set of gaps in 2001, rightly or wrongly, but identified them, I think that they were real in 2001.”

Claim: The speaker says he identified a set of real gaps in 2001.

“I mean, those were applying to multi-layer perceptions, not to transformers, which hadn't been invented yet.”

Claim: The speaker says the gaps he identified in 2001 concerned multilayer perceptrons rather than transformers, which had not yet been invented.

“But I still see exactly the same gap. gaps. And I see some quantitative improvement, but no principled solution to any of the gaps.”

Claim: The speaker says the same gaps remain despite quantitative improvement, with no principled solution to any of them.

“So there were three core chapters in that book. One of them was about operations over variables, one was about structured representations, and one was about types and tokens or kinds of individuals.”

Claim: The speaker says his book had three core chapters covering operations over variables, structured representations, and types and tokens (or kinds of individuals).

“And I just don't see that the things that I described then have been solved.”

Claim: The speaker believes the problems he described in his book remain unsolved.

“I see certain cases where they can be solved, but all of the problems that I see seem to be reflexes of those same core problems.”

Claim: The speaker says some instances of the problems can be solved, but the problems he observes appear to reflect the same underlying core problems.

“And so if I seem like a grumpy old man, it's because for twenty some years, and really I pointed these things out in nineteen ninety eight, twenty seven years I have not really seen them. And so the notion that we're gonna solve those gaps all in three years seems weird. Like I thought your estimates of solving a few of the things you just mentioned were generous. You say, you know, we solve re reading”

Claim: The speaker says he has not really seen the gaps he has raised since 1998 solved over roughly 27 years, and finds it strange to expect them all to be solved within three years.

“And so if I seem like a grumpy old man, it's because for twenty some years, and really I pointed these things out in nineteen ninety eight, twenty seven years I have not really seen them. And so the notion that we're gonna solve those gaps all in three years seems weird. Like I thought your estimates of solving a few of the things you just mentioned were generous. You say, you know, we solve re reading”

Claim: The speaker considers the estimates for solving some of the discussed problems overly optimistic.

“the kinds of numbers that you gave were like three or four for this one three or four for that one and i think like really you know my estimate is really 10 years is most of the distribution is past 10 years it's because i see several problems that like we'd be really lucky if we told let's say the visual intelligence problems in three or four years we'd be really lucky if we solved the um the video kind of visual intelligence over time let's call it problems in three or four years. I just see too many problems, too many gaps to use your phrase to think we'd get all of those at once like it in the movie everywhere every what is it everything everywhere all at once.”

Claim: The speaker believes solving visual-intelligence problems, including video-related problems, within three or four years would be very lucky.

“the kinds of numbers that you gave were like three or four for this one three or four for that one and i think like really you know my estimate is really 10 years is most of the distribution is past 10 years it's because i see several problems that like we'd be really lucky if we told let's say the visual intelligence problems in three or four years we'd be really lucky if we solved the um the video kind of”

Claim: The speaker estimates that most of the timeline probability distribution lies beyond 10 years.

“visual intelligence over time let's call it problems in three or four years. I just see too many problems, too many gaps to use your phrase to think we'd get all of those at once like it in the movie everywhere every what is it everything everywhere all at once.”

Claim: The speaker believes there are too many problems and gaps for all of them to be solved simultaneously.

“To me to get to 2027 would be everything everywhere all at once for a set of things that I've been worried about for 25 years.”

Claim: The speaker considers it an exceptionally rapid convergence if the issues he has worried about for 25 years were all resolved by 2027.

“It's the word automatically that I bristle at.”

Claim: The speaker objects to the claim that continued scaling will automatically solve AI problems.

“for the years i like like we had this potential dispute about whether it's just llms or whether it's neurosymbolic i don't want to fight you about that like we can say it's neurosymbolic if you like we can say it's not automatic but rather involves some new ideas but the point is i think it's going to happen in like three or four years”

Claim: The speaker predicts that the relevant AI development will occur in about three or four years, regardless of whether it is called neurosymbolic or characterized as involving new ideas rather than happening automatically.

“So I just think new ideas are hard to project.”

Claim: The speaker believes it is difficult to forecast when new ideas will emerge.

“everybody thought genes were made of proteins. Somebody even won a Nobel Prize on that premise. They were all wrong.”

Claim: The speaker says many people in the early 20th century incorrectly believed genes were made of proteins.

“And it took a while for people to figure out that this is wrong.”

Claim: The speaker says people took time to recognize that genes were not made of proteins.

“And Oswald Avery did these experiments in the 1940s which really ruled it out and discovered by process of elimination that it was DNA.”

Claim: The speaker says Oswald Avery's 1940s experiments ruled out proteins as the genetic material and identified DNA by process of elimination.

“And it didn't take that long to move very fast in molecular biology once people got rid of the bad assumption.”

Claim: The speaker says molecular biology progressed rapidly once researchers discarded the mistaken assumption about genes being made of proteins.

“I think we're making some bad assumptions. But it's hard to predict when people will move past and need a...”

Claim: The speaker suspects the field is making mistaken assumptions and says it is difficult to predict when researchers will move beyond them.

“dA fix, which is where I think we are now. I mean, proving that is very hard, I can give a lot of, you know, qualitative evidence to point in that direction. But there's at least a possibility that I'm right about that. If we don't have the right set of ideas, it depends on when we come up with new ideas. It could be that we automate the discovery of new ideas, but most of what LLMs have done is not discovery of new ideas, it's really exploiting existing ideas.”

Claim: The speaker allows that discovery of new ideas might be automated.

“dA fix, which is where I think we are now. I mean, proving that is very hard, I can give a lot of, you know, qualitative evidence to point in that direction. But there's at least a possibility that I'm right about that. If we don't have the right set of ideas, it depends on when we come up with new ideas. It could be that we automate the discovery of new ideas, but most of what LLMs have done is not discovery of new ideas, it's really exploiting existing ideas.”

Claim: The speaker says that if the necessary ideas are missing, progress depends on when new ideas are developed.

“dA fix, which is where I think we are now. I mean, proving that is very hard, I can give a lot of, you know, qualitative evidence to point in that direction. But there's at least a possibility that I'm right about that. If we don't have the right set of ideas, it depends on when we come up with new ideas. It could be that we automate the discovery of new ideas, but most of what LLMs have done is not discovery of new ideas, it's really exploiting existing ideas.”

Claim: The speaker characterizes most LLM accomplishments as exploiting existing ideas rather than discovering new ones.

“dA fix, which is where I think we are now. I mean, proving that is very hard, I can give a lot of, you know, qualitative evidence to point in that direction. But there's at least a possibility that I'm right about that. If we don't have the right set of ideas, it depends on when we come up with new ideas. It could be that we automate the discovery of new ideas, but most of what LLMs have done is not discovery of new ideas, it's really exploiting existing ideas.”

Claim: The speaker says it is possible that his view that AI needs a fix is correct, although proving it is very difficult and his supporting evidence is qualitative.

“The timelines, there are many paths to it being shorter before 2030, for instance,”

Claim: The speaker believes there are multiple possible paths to achieving the outcome before 2030.

“I think that the absolute most bullish, if we want to use that word, case is 2030. Like in the very fastest case, it'd be 2030.”

Claim: The speaker considers 2030 the earliest plausible date in the most bullish scenario.

“I think that the absolute most bullish, if we want to use that word, case is 2030. Like in the very fastest case, it'd be 2030.”

Claim: The speaker places the most optimistic timeline at 2030, including the very fastest case.

“I just can't get my mind around 2027. It just does not seem plausible because of the number of problems.”

Claim: The speaker considers an AI milestone by 2027 implausible because many problems would need to be solved.

“And I think also what you said is right. We want to integrate these two approaches to make the forecast. And nobody's really quite done that. Maybe that's an adversarial slash collaborative collaboration that we could do.”

Claim: The speaker says that no one has yet integrated the two approaches into a forecast.

“Those are always remedied. I was told, I don't know if it's true, you know, people patch them up. The specific examples in that sense, like literally this prompt gets this weird answer, a lot of them have been solved.”

Claim: Many reported prompt-specific odd answers from LLMs have been fixed.

“Those are always remedied. I was told, I don't know if it's true, that people in open AI actually look on social media for examples, including mine.”

Claim: The speaker has heard, but cannot verify, that OpenAI staff search social media—including his posts—for examples of model failures.

“think LLMs will solve it. I mean, I think someone could come up with a scheme to get LLMs to play chess without making illegal moves by calling a tool. I think that's possible.”

Claim: The speaker thinks it is possible to make LLMs play chess without illegal moves by using a tool call.

“Without calling a tool, I don't think it would be possible in the next six months. I'd be very surprised, and you can ridicule me in six months if it happens, and that's fine.”

Claim: The speaker considers it unlikely that LLMs will play chess without illegal moves and without tool use within the next six months.

“But also let me add a twist to it, which is what we talked about in rebooting AI was like, for example, if you train a particular system on, I think we use Go as an example, but if you train it on a 15 by 15, If you train it on an ordinary 21 by 21 board, we'll be able to play. on a rectangle instead of a square in different variations.”

Claim: The speaker suggests that a Go system trained on a standard 21-by-21 board could play on rectangular boards and handle other variations.

“But even then, it was sterile knowledge in the sense that it was purpose-built for this, but it wasn't general knowledge of chess.”

Claim: The speaker characterizes the chess system's knowledge as task-specific rather than general chess knowledge.

“If you asked it to do my put three queens on a board and but still three white queens but make sure the black wins on the next move, it wouldn't be able to do that at all.”

Claim: The speaker says the chess system could not solve a contrived position requiring three white queens and a Black win on the next move.

“Right? The the there's a flexibility to human expert knowledge that we emphasized in rebooting AI that I see no evidence of of progress or maybe a little evidence of progress but you know relatively little progress on that.”

Claim: The speaker sees little, though possibly some, progress in AI systems' ability to match the flexibility of human expert knowledge.

“I think distribu... I'll put it this way. There'll be many, many examples I can come up with distribution shift. All of what I was just doing will still find failures on next year.”

Claim: The speaker expects many distribution-shift cases to continue exposing failures in AI systems next year.

“am sure that I will be able to come up with variations on chess that are not orthodox variations, but maybe there's some already known.”

Claim: The speaker is sure he can devise non-orthodox chess variations, while acknowledging that some may already be known.

“Chess actually has lots of variations or chess problems and things like that that these systems won't be able to do. I'll probably be able to come up with variations on tic-tac-toe. So five by five board, but you can only win on the edges.”

Claim: The speaker expects current systems to fail on many chess variants and problems, and says he can devise tic-tac-toe variants such as a five-by-five board where wins are allowed only on the edges.

“There will be tons of problems like that. They're all kind of outlier-ish. Here-ish, but I think that a year from now, these systems will still be very vulnerable to those outliers.”

Claim: The speaker predicts that AI systems will remain highly vulnerable to outlier problems one year later.

“I mean, here's another way to put it is the AGI that I think we're afraid might be unconstrained or whatever, there really shouldn't be a lot of edge cases like that,”

Claim: The speaker believes the unconstrained AGI they fear should not exhibit many of the edge-case failures being discussed.

“So I don't know what the human data on it, but I know that the machine data on it was zero percent. And what was interesting about the live code pro...”

Claim: The speaker says machine performance on the LiveCodeBench Pro benchmark was 0%, and that he does not know the human performance.

“excuse me, was that they were all brand new problems. And so they ruled out data contamination. And I've seen two careful studies ruling out data contamination.”

Claim: The speaker says the LifeCode BenchPro used entirely new problems and that two careful studies ruled out data contamination.

“The other was with the US AMO, US Math Olympiad problems six hours after testing. Performance was terrible on them.”

Claim: The speaker says AI performance on US Math Olympiad problems presented six hours after testing was terrible.

“I think if you rule out data contamination, the performance on”

Claim: The speaker believes that, if data contamination is ruled out, AI performance on new problems is poor.

“problems that are new is not good and for programming especially programming AGI that's super relevant you know if you're using these things to code up a website that's not new there's so many examples but if you're using it to do something that's new you get out of distribution there's still a lot of problems”

Claim: The speaker argues that novel programming tasks, unlike familiar website coding with many examples, create out-of-distribution challenges that are especially relevant to programming AGI.

“there's some contamination issues there you know open AI had access to the problems”

Claim: The speaker says frontier-math evaluations may have contamination issues because OpenAI had access to the problems.

“ai is just not i mean you can agree with me on this it's not an entirely forthcoming place about what they did i”

Claim: The speaker believes OpenAI is not fully forthcoming about what it did.

“think they have okay but everybody is is teaching to the test right now and they're all doing augmentation we don't know what the augmentation is the point is”

Claim: The speaker thinks Claude and Gemini may have acceptable frontier-math scores, while current systems are being optimized for the test through augmentation whose nature is unknown.

“Because if you have AGI, it's not just about teaching to the test anymore and you need to be able to solve things that are original.”

Claim: The speaker says AGI must be able to solve original problems, not merely perform well through teaching to the test.

“I mean, we're lucky if AI never can do that because then it gives an Achilles heel.”

Claim: The speaker says that if AI never becomes able to solve original problems, this limitation would provide an Achilles heel.

“I mean I also just do, for the record, I disagree with you guys. Like I think that progress is being made in this sort of um hard to measure a dimension that you're pointing to. Uh uh it's hard to measure. The the best ways we have to measure it are for you to come up with examples uh and”

Claim: The speaker believes AI progress is occurring in a dimension that is difficult to measure.

“we know there's been progress made every year, absolutely.”

Claim: The speaker says driverless-car technology has made progress every year.

“But the distribution shift problem still remains.”

Claim: The speaker says distribution shift remains a problem for driverless cars.

“So, you know, even Waymo, who I think is maybe ahead in this, like they just announced New York City, but when they go to New York City, they're going to have a safety driver there. They're going to have geofencing and so forth.”

Claim: The speaker thinks Waymo may be ahead in driverless cars, but says its New York City deployment will use a safety driver and geofencing.

“The general form of driving solution would be level five, there'd be no geofencing, you wouldn't need specialized maps, etc.”

Claim: The speaker describes a general driving solution as level-five autonomy that requires neither geofencing nor specialized maps.

“So there's been progress.”

Claim: The speaker says driverless cars have made progress.

“for 40 years every year in driverless cars. But the distribution shift problem is really what is still hobbling that from being a thing everywhere as opposed to a thing in San Francisco.”

Claim: The speaker says distribution shift continues to hinder driverless cars from operating everywhere, rather than being limited to places such as San Francisco.

“And unfortunately for other abilities that probably the ability has most, or well, its two main abilities would be crystallized intelligence, acquired knowledge, and reading writing ability.”

Claim: The speaker identifies AI's two main abilities as crystallized intelligence (acquired knowledge) and reading and writing.

“its two main abilities would be crystallized intelligence, acquired knowledge, and reading writing ability.”

Claim: The speaker characterizes the models’ two main abilities as crystallized intelligence (acquired knowledge) and reading and writing.

“the models when you ask them to write an essay, if you score them on like the GRE score out of six, they'll get like 4.5 or so. They're not particularly great writers and we can't, you know, you can't automate writers that well with.”

Claim: The speaker says models score around 4.5 out of 6 on a GRE-style writing assessment and are not particularly strong writers.

“if you score them on like the GRE score out of six, they'll get like 4.5 or so. They're not particularly great writers and we can't, you know, you can't automate writers that well with. with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing”

Claim: The speaker says that models do not yet automate somewhat complicated writing well, so people still need to perform that work.

“if you score them on like the GRE score out of six, they'll get like 4.5 or so. They're not particularly great writers and we can't, you know, you can't automate writers that well with.”

Claim: The speaker says that models score about 4.5 out of 6 on the GRE writing scale and are not particularly strong writers.

“with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing more automation or full automation that could actually require resolving some of these other bottlenecks that are other other cognitive ability bottlenecks that that Gary is alluding to”

Claim: The speaker suggests that extrapolating current models may yield much of the low-hanging fruit, while greater or full automation could require overcoming additional cognitive-ability bottlenecks.

“with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing”

Claim: Despite extensive training data—and GitHub data for coding—models still have difficulty automating somewhat complicated writing.

“with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing more automation or full automation that could actually require resolving some of these other bottlenecks that are other other cognitive ability bottlenecks that that Gary is alluding to”

Claim: The speaker suggests that extrapolating current models may yield many easy gains, but achieving greater or full automation could require overcoming other cognitive-ability bottlenecks.

“with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing”

Claim: The speaker says somewhat complicated writing still requires human work and is not well automated by models.

“with um if if they're doing somewhat complicated writing you still people still need to be doing that and that's somewhat surprising because they've had so much data uh they've had all the data in the world basically and likewise for coding they have had github so that at least b it is it's potentially um the case that as you extrapolate them out they'll get a lot of the low-hanging fruit, but it crossing some sort of threshold for doing”

Claim: The speaker says models have had access to extensive data, including GitHub for coding.

“I would say that a lot of it is sort of interpolation rather than extrapolation. We haven't solved the extrapolation problem, but interpolation, we made huge progress.”

Claim: The speaker says AI has made substantial progress in interpolation but has not solved extrapolation, and characterizes much of the progress as interpolation rather than extrapolation.

“It's mostly just in virtue of having more data and more compute.”

Claim: The speaker attributes most recent AI interpolation progress to increased data and compute.

“But there's no question that new systems are much better at interpolating than previous systems.”

Claim: The speaker says newer AI systems interpolate substantially better than earlier systems.

“And that has lots of practical consequences for labor, for example. There's a whole bunch of things that you can use these systems to do that you couldn't use them before that's already affecting labor markets.”

Claim: The speaker says that newer AI systems enable tasks that were previously unavailable and that these capabilities are already affecting labor markets.

“Whereas my intuitive sense is on alignment, all we have is maybe a human reinforcement learning.”

Claim: The speaker’s tentative view is that human reinforcement learning has provided only limited help with AI alignment.

“um helps a little bit so that like if you ask these systems the most obvious question like how do i build a biological weapon they'll decline that's a little bit of progress but like we all know that those are things are easily jailbroken like my view and you can agree or disagree or whatever is like an alignment system still don't really do what we want them to do there's still kind of sorcerer's apprentice style problems they're kind of you”

Claim: The speaker believes current alignment systems still fail to reliably do what users intend and exhibit sorcerer’s-apprentice-style failures.

“um helps a little bit so that like if you ask these systems the most obvious question like how do i build a biological weapon they'll decline that's a little bit of progress but like we all know that those are things are easily jailbroken like my view and you can agree or disagree or whatever is like an alignment system still don't really do what we want them to do there's still kind of sorcerer's apprentice style problems they're kind of you”

Claim: The speaker says current AI systems may refuse an obvious biological-weapons request, but such refusals are easily bypassed through jailbreaking.

“problems of just like they don't quite fit they don't do what we want we have system prompts that say don't produce copyrighted material and they still do don't hallucinate they still do like they you know they can approximate what we ask for them but like they don't really do what we ask them i take i take even like don't make illegal moves in chess to be a form of the alignment problem like a very simple microcosm of the alignment problem and there's stills”

Claim: The speaker says that system prompts against producing copyrighted material or hallucinating do not reliably prevent those behaviors.

“problems of just like they don't quite fit they don't do what we want we have system prompts that say don't produce copyrighted material and they still do don't hallucinate they still do like they you know they can approximate what we ask for them but like they don't really do what we ask them i take i take even like don't make illegal moves in chess to be a form of the alignment problem like a very simple microcosm of the alignment problem and there's stills struggling with that. That's my view. How about you guys?”

Claim: The speaker treats obeying a simple constraint such as not making illegal chess moves as a small-scale alignment problem that AI systems still struggle to solve.

“problems of just like they don't quite fit they don't do what we want we have system prompts that say don't produce copyrighted material and they still do don't hallucinate they still do like they you know they can approximate what we ask for them but like they don't really do what we ask them i take i take even like don't make illegal moves in chess to be a form of the alignment problem like a very simple microcosm of the alignment problem and there's stills”

Claim: The speaker says AI systems can approximate user requests without reliably fulfilling them as intended.

“so there's some different notions of alignment or alignability. And I would distinguish between aligning proto super intelligences and aligning a sort of recursion that gives rise to super intelligence.”

Claim: The speaker distinguishes aligning proto-superintelligent models from aligning the recursive process that could produce superintelligence.

“Those are very qualitatively different. One is more model level and one is more. process level.”

Claim: The speaker says that model-level alignment and process-level alignment are qualitatively different problems.

“So I don't think you're going to solve that process level one of doing a recursion, fully de-risking that, anticipating all the unknowns and unknowns and solving a wicked problem in a nice clean way beforehand,”

Claim: The speaker doubts that the risks and unknowns of a recursive process leading to superintelligence can be fully anticipated and resolved in advance.

“which points to the necessity for resolving geopolitical competitive pressures and giving them an out to proceed with a recursion more slowly or substantially forestall that.”

Claim: The speaker argues that addressing geopolitical competition and enabling slower progress or substantial postponement of recursive AI development are necessary responses to the difficulty of fully de-risking it in advance.

“I think that they follow the instructions fairly reasonably. There are some other parts of it.”

Claim: The speaker says proto-ASI systems follow instructions fairly reasonably.

“But like, if you have them guiding weapons or something like that, there are many circumstances where fairly reasonably is not good enough.”

Claim: The speaker says that merely fairly reasonable instruction-following is inadequate in some high-stakes situations, including systems guiding weapons.

“There are definitely safety safety critical domains for and I think for for instance in by I think for refusal for instance in in some high stakes context or given some high stakes queries it depends there are some cases where I think you can actually get multiple nines of reliability such as with bio weapons refusal however for other types of refusal like don't cause any criminal or don't”

Claim: The speaker thinks some high-stakes refusal tasks, including refusing biological-weapons requests, may reach reliability of multiple nines.

“safety critical domains for and I think for for instance in by I think for refusal for instance in in some high stakes context or given some high stakes queries it depends there are some cases where I think you can actually get multiple nines of reliability such as with bio weapons refusal however for other types of refusal like don't cause any criminal or don't don't take a criminal or tortious action. That's a lot fuzzier, and I it's”

Claim: The speaker says that reliably refusing requests involving criminal or tortious conduct is a much less clear-cut problem than refusing biological-weapons requests.

“So in production they're not using, they're not using the techniques that are as adversarially robust because they come at a cost of maybe a percent or two in M_M_L_U_ and so they're”

Claim: The speaker says production systems omit more adversarially robust techniques because they may reduce MMLU performance by about one or two percentage points.

“but actually so I think there are some types of there are some types of solutions for some types of malicious use where you can have some lines of reliability but for the general problem of refusal including for everyday criminal and tortious actions those would be extremely relevant for for agents and making them uh feasible I don't see substantial progress.”

Claim: The speaker says some malicious-use refusal problems have solutions with some reliability, but substantial progress is lacking for general refusals involving everyday criminal or tortious conduct.

“those would be extremely relevant for for agents and making them uh feasible I don't see substantial progress.”

Claim: The speaker says reliable refusal of everyday criminal and tortious actions is highly relevant to the feasibility of AI agents.

“And and the the cleaned up version law is actually the the law kinda cleans it up and saying don't cause foreseeable harm, 'cause harm is way too strict. But foreseeable harm is what we ex demand of humans. And”

Claim: The speaker says the legal standard is to avoid foreseeable harm rather than all harm, because avoiding all harm is too strict, and that humans are held to the foreseeable-harm standard.

“And so I think for most of these reliability issues and safety issues, we'll keep seeing new symptoms crop up and we'll have specific solutions that partly target them if there's willingness”

Claim: The speaker expects new reliability and safety failure symptoms to keep emerging, with issue-specific solutions partly addressing them when there is willingness to act.

“And I expect that this game will keep continuing and we won't get to a state where that. That is basically mostly managed in time because the risk surface will keep evolving with agents that will present new things and we'll have to deal with those current cases that will create a substantial backlog and we just won't have the adaptive capacity.”

Claim: The speaker expects the ongoing response to cybercrime and evolving AI-agent risks to continue without reaching a state where the risks are mostly managed in time.

“That is basically mostly managed in time because the risk surface will keep evolving with agents that will present new things and we'll have to deal with those current cases that will create a substantial backlog and we just won't have the adaptive capacity.”

Claim: The speaker predicts that evolving risks from AI agents will create a substantial backlog that society will lack the adaptive capacity to address.

“And so consequently, on both fronts for aligning recursion and aligning proto ASIs, the geopolitical competitive pressures make it such that we're probably not. not going to solve either problem.”

Claim: The speaker believes geopolitical competitive pressures probably prevent solving either alignment recursion or proto-ASI alignment.

“if we can't do alignment well on the refusals, et cetera, and the cause to harm causing harm to humans, foreseeable harm, even to humans, that's a reason to say, we got to wait until we have a better solution here. here if that takes 500 years if it takes five years like in the safety critical stuff isn't that a reason to you know slow things down a bit sure”

Claim: The speaker argues that safety-critical AI uses should wait for a better alignment solution if systems cannot reliably refuse harmful requests and prevent foreseeable human harm, even if the solution takes five or 500 years.

“I think that that's a direction or an interpretation of those facts that is reasonable, but there's a question of incentives and what, you know, there might be a somewhat different aspect”

Claim: The speaker considers slowing AI development a reasonable response to the alignment risks, while noting that incentives are another relevant consideration.

“Yeah, yeah, so that's why in superintelligence strategy we'll... will suggest some different things like making the threat of preemption if you're crossing these sorts of lines, as opposed to stopping today because that is a lot harder to get I'll institutions”

Claim: The speaker says Superintelligence Strategy will propose threatening preemption when AI developers cross specified lines, rather than seeking an immediate halt that is harder to secure institutionally.

“I mean I'm not actually advocating holding the train per se, but on these safety critical things, I really could see an argument for some kind of intervention. As you note, it could be incentives rather than than more time. But my view is that we are not going to solve the technical alignment, the sort of nearer term technical problem with let me finish the sentence with LLMs, that they're just not up to the task. Maybe with neurosymbolic. we might have a chance. So neurosymbolic gives you a chance to state explicit constraints that is very difficult in pure LLMs. And so I think there's reason to explore that avenue as a way, I don't know that it helps with superintelligence, but at least with the near-term use of these systems for safety-critical measures,”

Claim: The speaker believes LLMs alone are inadequate for the nearer-term technical alignment problem, while neurosymbolic methods might offer a chance.

“I mean I'm not actually advocating holding the train per se, but on these safety critical things, I really could see an argument for some kind of intervention. As you note, it could be incentives rather than than more time. But my view is that we are not going to solve the technical alignment, the sort of nearer term technical”

Claim: The speaker sees a possible case for intervention on safety-critical AI issues, potentially through incentives rather than delaying development.

“we might have a chance. So neurosymbolic gives you a chance to state explicit constraints that is very difficult in pure LLMs. And so I think there's reason to explore that avenue as a way, I don't know that it helps with superintelligence, but at least with the near-term use of these systems for safety-critical measures,”

Claim: The speaker says neurosymbolic systems make it possible to state explicit constraints, something very difficult with pure LLMs.

“we might have a chance. So neurosymbolic gives you a chance to state explicit constraints that is very difficult in pure LLMs. And so I think there's reason to explore that avenue as a way, I don't know that it helps with superintelligence, but at least with the near-term use of these systems for safety-critical measures, there might be some mileage to be gotten there.”

Claim: The speaker thinks neurosymbolic methods are worth exploring for near-term safety-critical AI applications, while remaining unsure whether they help with superintelligence and saying they might offer some benefit.

“with both of it it's a resource thing and I think it's less of a technical thing. The technical things can r increase the capacity to deal with these problems or uh p have more efficient solutions for for some of these particular symptoms um or these these new failure modes that crop up, but I I'm not expecting a total um monolithic uh solution um uh that applies would necessarily give. I think you have”

Claim: The speaker views alignment and recursive-development challenges primarily as resource-capacity problems rather than purely technical ones; technical advances may improve capacity or address specific failure modes more efficiently, but the speaker does not expect one comprehensive solution.

“You have to have the background context in both cases be that you're able to proceed with development under some risk tolerance that's much lower than what there is today.”

Claim: The speaker says AI development should proceed under a risk tolerance substantially lower than the current level.

“We're totally not on track to have figured out the alignment stuff in time.”

Claim: The speaker believes current progress is not on track to solve the AI alignment problem in time.

“forecasting even there we're not probably as far apart as maybe people thought that we were right so you know i i push all of my probability mass to five years out and have basically none before and you've got some at three years out or two years out that i don't we both have some out at 2045 i have some even past then and you may or may not um”

Claim: The speaker says the participants' forecasts are closer than people may assume: one concentrates probability at five years out and later, while the other assigns some probability to two or three years out, and both assign some probability to 2045.

“We have slightly different methodologies that we have talked about.”

Claim: The speaker says the participants use somewhat different forecasting methodologies.

“And I think we've acknowledged the value in some of the forecasting techniques that the other has used, even if we don't. So we're not hugely apart there.”

Claim: The speaker says the participants acknowledge value in some forecasting techniques used by the other, despite their differences.

“I think we are completely agreed that we're not doing a great job on the alignment problem and that. we need to do much better that there's a temporal dimension to that as you were just saying which is like you know it's not great for humanity if we solve that problem in 200 years and we we have agi or asi you know in the next decade or two um and i think”

Claim: The speaker says the participants agree that current efforts on AI alignment are inadequate and need substantial improvement.

“we need to do much better that there's a temporal dimension to that as you were just saying which is like you know it's not great for humanity if we solve that problem in 200 years and we we have agi or asi you know in the next decade or two um and i think”

Claim: The speaker says it would be bad for humanity if AI alignment were solved only 200 years from now while AGI or ASI arrived within the next 10 to 20 years.

“We agree also that the current companies are not entirely trustworthy.”

Claim: The speaker says current AI companies are not entirely trustworthy.

“I think by the end of the year, we will already be less far ahead on agents as you hypothesize there.”

Claim: The speaker predicts that by the end of the year, progress on AI agents will be less advanced than the AI 2027 scenario hypothesizes.

“Like, I think the first couple months were actually kind of in agreement, but, you know, there's a divergence in speed for sure.”

Claim: The speaker says the scenario and reality were roughly aligned during the first couple of months, but have diverged in their pace.

“think that the level of intelligence that you attribute to the machines towards the end of the essay, I remember some of that actually happens after the year 2027. I think. You know, I don't really think is very likely in the next decade. And I don't think it's super likely in the decade after that, but it's certainly possible.”

Claim: The speaker considers the machine-intelligence level depicted toward the end of AI 2027 unlikely within the next decade and not very likely in the following decade, while allowing that it is possible.

“So I think apprentice engineer, we're actually close already.”

Claim: The speaker believes current AI systems are already close to performing at the level of an apprentice software engineer.

“But top software engineer, I don't think we're close. I think that that requires an understanding of the problem, what humans want solved by the problem. It requires a deep understanding of various domains. means. I just don't think that we're that close to that.”

Claim: The speaker believes AI systems are not close to matching a top software engineer, because that requires understanding the problem, the human goals behind it, and multiple domains.

“So I do think these things will continue to improve regularly. We will get more and more value out of them.”

Claim: The speaker predicts that AI systems will continue improving regularly and provide increasing value.

“They will improve programmer productivity with some asterisks around how secure is the code, how maintainable is the code, etc.”

Claim: The speaker predicts that AI systems will improve programmer productivity, while noting caveats concerning the security and maintainability of generated code.

“But I don't think that they're going to replace the best coders. You're not going to get a machine that is Jeff Dean anytime in the next decade.”

Claim: The speaker predicts that AI systems will not replace the best coders or reach Jeff Dean's coding level within the next decade.

“I think that the human mind sucks in light of scenarios, that it takes them very seriously. The vivid details, overwhelm people's ability to see things.”

Claim: The speaker says people tend to take scenarios seriously, and vivid details can overwhelm their ability to assess them.

“What I would like to see actually would be a distribution of scenarios. Everybody goes home and thinks that that scenario is real. But you and I know that that was one scenario of many. There's reasons to consider that scenario. And it's sort of, you know, the darker one is is a very vivid version of the dark scenario. we want to understand the distribution of scenario. But what I would like to see is really a distribution of scenarios.”

Claim: The speaker argues that scenario analysis should present a distribution of possible futures, since audiences may mistake one vivid scenario for reality despite it being only one of many.

“I think that that would be helpful.”

Claim: The speaker believes that presenting a distribution of possible scenarios would be helpful.

“My own personal scenario is like. In three or four years, neurosymbolic AI starts to take off.”

Claim: The speaker predicts that neurosymbolic AI will begin to take off in three or four years.

“In three or four years, neurosymbolic AI starts to take off.”

Claim: The speaker predicts that neurosymbolic AI will begin to take off in three or four years.

“Like it's, I already see signs of this. AlphaFold just won a Nobel Prize. That's a nice thing for neurosymbolic AI. The conferences for it are getting bigger and so forth.”

Claim: The speaker says there are already signs of neurosymbolic AI beginning to take off.

“Like it's, I already see signs of this. AlphaFold just won a Nobel Prize. That's a nice thing for neurosymbolic AI. The conferences for it are getting bigger and so forth.”

Claim: The speaker says there are already signs that neurosymbolic AI is gaining momentum.

“And I think eventually there will be a state change. I find it very hard to know when there will be a state change.”

Claim: The speaker expects neurosymbolic AI eventually to undergo a state change but says its timing is very hard to predict.

“And I think eventually there will be a state change. I find it very hard to know when there will be a state change.”

Claim: The speaker expects a future state change in AI but says its timing is very difficult to predict.

“But I think in 2035, we will look at LLMs and be like, nice try. We still use them for some things, but that wasn't really. That wasn't really the answer.”

Claim: The speaker predicts that by 2035, people will still use LLMs for some tasks but will regard them as not having been the answer to achieving advanced AI.

“But I think in 2035, we will look at LLMs and be like, nice try. We still use them for some things, but that wasn't really. That wasn't really the answer.”

Claim: The speaker predicts that by 2035 people will regard LLMs as an incomplete answer, although they will continue using them for some tasks.

“I think Jan LeCun would say the same thing. Again, despite our differences, I think we both think LLMs are not really the route to AGI and that when we get there, it's going to look pretty different.”

Claim: The speaker believes that LLMs are not really the route to AGI and that achieved AGI will look quite different; he thinks Jan LeCun shares this view.

“I think Jan LeCun would say the same thing. Again, despite our differences, I think we both think LLMs are not really the route to AGI and that when we get there, it's going to look pretty different.”

Claim: The speaker believes, and thinks Yann LeCun also believes, that LLMs are not the primary route to AGI and that achieved AGI will look substantially different.

“It might use LLMs. They're great at kind of distributional learning.”

Claim: The speaker says future AGI might use LLMs because they are good at distributional learning.

“It might use LLMs. They're great at kind of distributional learning.”

Claim: The speaker says future AGI might use LLMs, which he considers good at distributional learning.

“It might replace them because they're very inefficient in terms of energy and data.”

Claim: The speaker says LLMs might be replaced because they are very inefficient in their use of energy and data.

“It might replace them because they're very inefficient in terms of energy and data.”

Claim: The speaker says LLMs might be replaced because they are highly inefficient in their use of energy and data.

“So somebody might find a better way to do the same kind of thing of learning the models of distributions of things, which is...”

Claim: The speaker says someone might discover a better method for learning models of distributions than current LLMs provide.

“So somebody might find a better way to do the same kind of thing of learning the models of distributions of things, which is... which is a super helpful cognitive skill. It's not the only one, but it's super helpful.”

Claim: The speaker says someone might develop a better method for learning distributions; he describes distribution learning as a very helpful, but not exclusive, cognitive skill.

“which is a super helpful cognitive skill. It's not the only one, but it's super helpful.”

Claim: The speaker considers learning models of distributions a very useful cognitive skill, but not the only important cognitive skill.

“But we'll have much better ways of doing reasoning and planning.”

Claim: The speaker predicts that future AI systems will have much better reasoning and planning methods.

“But we'll have much better ways of doing reasoning and planning. We'll have much more stable world models.”

Claim: The speaker predicts that future AI systems will have substantially better reasoning and planning methods and more stable world models.

“But we'll have much better ways of doing reasoning and planning. We'll have much more stable world models. I think it will take five or ten years to develop that.”

Claim: The speaker estimates that developing much better reasoning and planning and more stable world models will take five or ten years.

“We'll have much more stable world models.”

Claim: The speaker predicts that future AI systems will have much more stable world models.

“I think it will take five or ten years to develop that.”

Claim: The speaker estimates that developing substantially better reasoning, planning, and stable world models will take five to ten years.

“I think the semantics that the current models have is very superficial. It's really about distributions of words.”

Claim: The speaker considers the semantics represented by current models very superficial, describing them as primarily distributions of words.

“I think the semantics that the current models have is very superficial. It's really about distributions of words.”

Claim: The speaker considers the semantics represented by current AI models superficial and primarily based on word distributions.

“And we need a deeper one.”

Claim: The speaker argues that AI systems need a deeper form of semantics than current models provide.

“And we need a deeper one.”

Claim: The speaker says AI systems need a deeper kind of semantics than current models provide.