AGI is still 30 years away — Ege Erdil & Tamay Besiroglu

“I think it's not a very useful concept.”

Claim: The speaker thinks the concept of an intelligence explosion is not very useful.

“Industrial revolution a horsepower explosion like sure during the industrial revolution We saw this drastic acceleration in raw physical power But there are many other things that were maybe equally important in explaining the acceleration of growth and technological change that we saw during the industrial revolution So”

Claim: During the Industrial Revolution, raw physical power increased drastically, but many other factors also helped explain accelerated growth and technological change.

“It was a bunch of these complementary changes to many different sectors in the economy. So you had agriculture, you had transportation, you had law and finance, you had urbanization and moving from rural areas into cities.”

Claim: The Industrial Revolution involved complementary changes across agriculture, transportation, law and finance, and urbanization.

“There were just many different innovations that kind of happened simultaneously that gave rise to this change in the way of economically organizing our society.”

Claim: Many simultaneous innovations contributed to a change in the economic organization of society.

“It wasn't just that we had more horsepower. I mean, that was part of it, but that's not the kind of central thing to focus on when thinking about the industrial revolution.”

Claim: In explaining the Industrial Revolution, increased horsepower was one factor but not the central focus.

“And I think similarly for the development of AI, we'll get like a lot of very smart AI systems, but that will be one part among very many different moving parts that explain, you know, why we expect to get this transition and this acceleration and growth. growth and technological change.”

Claim: The speaker expects very smart AI systems to be one among many factors behind a transition and acceleration in growth and technological change.

“Yeah, maybe for me that would be around like 2045 or”

Claim: The speaker tentatively expects AI replacement of remote workers around 2045.

“for literally everything just shade i guess predictions by five years or like by 20 or something”

Claim: For AI to perform literally all remote work, the speaker would move the forecast later by about five years, or possibly by 20 years.

“so i think that a lot of people have this intuition that progress has been very fast they like just look at the trend lines and just like extrapolate like obviously it's going to happen in like i don't know 2027 or 2030 or whatever it's very bullish and obviously that's not a thing you can literally do like there isn't like a”

Claim: The speaker says many people extrapolate recent AI progress to predict automation by 2027 or 2030, but argues that automation timing cannot literally be inferred by extrapolating such a trend line.

“like a trend you can literally extrapolate of when do we get the fourth automation because if you look at the fraction of the economy that has actually been automated it's very like by AI it's very small so if you just extrapolate that trend which is something say Robin Hanson likes to do you're gonna say well it's gonna take centuries or something now we don't agree with that view but I think one way of thinking about this is like”

Claim: The speaker says extrapolating AI's current share of economic automation would imply a timescale of centuries, but rejects that conclusion.

“like a trend you can literally extrapolate of when do we get the fourth automation because if you look at the fraction of the economy that has actually been automated it's very like by AI it's very small so if you just extrapolate that trend which is something say Robin Hanson likes to do you're gonna say well it's gonna take centuries or something now we don't agree with that view but I think one way of thinking about this is like”

Claim: The speaker says AI has automated only a very small fraction of the economy so far.

“um how many like big things are there how many core capabilities components are there that the ai systems need to be good at in order to have this very broad economic impact maybe 10x acceleration in growth or something how many things have you gotten like how over the past uh 10 years 15 years and uh we also have this compute centric view”

Claim: The speaker considers that AI systems may need several core capabilities to produce broad economic effects, potentially including a tenfold acceleration in growth.

“If you look at the past 10 years of AI progress, we've gone through about nine or 10 orders of magnitude of compute,”

Claim: The speaker says AI compute increased by about nine or ten orders of magnitude over the past 10 years.

“and we've got various capabilities that were unlocked.”

Claim: The speaker says compute scaling unlocked various AI capabilities.

“So you had in the, you know, in the early period, people were kind of, you know, solving gameplay on specific games, on very complex games. And, you know, that happened 2015 to maybe 2020 and Go and Chess and... data and other games.”

Claim: The speaker says AI systems learned to handle gameplay in complex games, including Go and chess, from about 2015 to 2020.

“And then you had maybe, you know, sophisticated language capabilities that were unlocked with these large language models and maybe kind of advanced abstract reasoning and”

Claim: The speaker says large language models unlocked sophisticated language capabilities and possibly advanced abstract reasoning.

“coding and maybe math that was maybe another big such capability that got unlocked.”

Claim: The speaker says coding and possibly mathematics were another major capability unlocked by AI systems.

“And so maybe there are a couple of these big unlocks that happened over the past 10 years.”

Claim: The speaker says a few major AI capabilities were unlocked over the past 10 years.

“But it takes, you know, that happened on the order of once every three years or so, or maybe one every three orders of magnitude of compute scaling.”

Claim: The speaker estimates that major AI capability unlocks have occurred roughly every three years, corresponding to about every three orders of magnitude of compute scaling.

“And so, and then you might ask the question, how many more such competencies might we need to unlock in order to be able to have an AI system that can match the capabilities of humans across the board, maybe specifically just on remote work tasks. well, maybe you need kind of coherence over very long horizons or you need kind of agency and autonomy or maybe you need multimodal, kind of full multimodal kind of understanding just like a human would.”

Claim: The speaker suggests AI may need long-horizon coherence, agency and autonomy, and human-like multimodal understanding to match human capabilities broadly or across remote-work tasks.

“Previous unlocks took about, you know, you get one every three years or so. But of course that previous period coincided with this rapid scale-up.”

Claim: The speaker says previous AI capability unlocks occurred about every three years during a period of rapid training-compute growth.

“of the amount of compute that we use for training so we went through maybe nine or ten orders of magnitude since alex net you know compared to the biggest models we have today and you know we're getting to a level where it's becoming harder and harder to scale up compute and you know we've done some extrapolations and some analysis looking at specific constraints like energy or gpu production and based on that it looks like we might have maybe three or four orders of magnitude of scaling left,”

Claim: The speaker says analyses of energy and GPU-production constraints suggest that perhaps three or four orders of magnitude of compute scaling remain.

“of the amount of compute that we use for training so we went through maybe nine or ten orders of magnitude since alex net you know compared to the biggest models we have today and you know we're getting to a level where it's becoming harder and harder to scale up compute and you know we've done some extrapolations and some analysis looking at specific constraints like energy or gpu production and based on that it looks like we might have”

Claim: The speaker estimates that training compute grew by about nine or ten orders of magnitude between AlexNet and the largest models then available.

“of the amount of compute that we use for training so we went through maybe nine or ten orders of magnitude since alex net you know compared to the biggest models we have today and you know we're getting to a level where it's becoming harder and harder to scale up compute and you know we've done some extrapolations and some analysis looking at specific constraints like energy or gpu production and based on that it looks like we might have”

Claim: The speaker says scaling up compute is becoming increasingly difficult.

“I mean, currently it's less than two percent.”

Claim: The share of GDP currently involved in the discussed activity is less than 2%.

“most of it is actually not going through towards ai chips but even most tsmc capacity currently is going towards mobile phone chips or something like that right yeah”

Claim: Most of TSMC's current chip-manufacturing capacity is used for mobile-phone chips or similar products, not AI chips.

“Even leading edge is going, like five percent of leading edge is. is pretty small”

Claim: AI accounts for about 5% of leading-edge chip capacity, a share the speaker characterizes as small.

“but but yeah so that suggests that you know we might need a lot more compute scaling to get these additional capabilities to be unlocked and then there's a question of do we really have that as a you know do we have that in us as an economy to be able to sustain that scaling right”

Claim: Unlocking additional AI capabilities might require substantially more compute scaling.

“but but yeah so that suggests that you know we might need a lot more compute scaling to get these additional capabilities to be unlocked and then there's a question of do we really have that as a you know do we have that in us as an economy to be able to sustain that scaling right”

Claim: It is uncertain whether the economy can sustain the compute scaling that may be needed for additional AI capabilities.

“i probably think by the end of this year they're going to be able to do that”

Claim: The speaker expects, with some uncertainty, that AI systems will be able to book a flight by the end of the current year.

“but that's like a very very like nobody gets a job where they're paid to like book flights for like like that's not a task”

Claim: People generally are not employed solely to book flights.

“i mean but i think that's an important point because a lot of people like look at jobs in the economy and then they're like oh like that person like their job is to just do x but then but then that's not true like that's something they do in their job”

Claim: A task commonly used to characterize a job is often only one component of that job, rather than the whole job.

“job but it's probably if you look at the fraction of their time on the job that they spend on doing that is a very small fraction of what they should do it's just this popular conception people have or travel agents like they just book hotels and flights but that's not actually most of their job so automating that actually wouldn't automate their job and it wouldn't have that much of an impact on the economy so i think this is actually an important thing that important worldview difference that separates us from people”

Claim: Although travel agents are popularly perceived as mainly booking hotels and flights, those tasks do not make up most of their work.

“job but it's probably if you look at the fraction of their time on the job that they spend on doing that is a very small fraction of what they should do it's just this popular conception people have or travel agents like they just book hotels and flights but that's not actually most of their job so automating that actually wouldn't automate their job and it wouldn't have that much of an impact on the economy so i think this is actually an important thing that important worldview difference that separates us from people”

Claim: Automating only a small fraction of a job would not automate the entire job and would have limited economic impact.

“job but it's probably if you look at the fraction of their time on the job that they spend on doing that is a very small fraction of what they should do it's just this popular conception people have or travel agents like they just book hotels and flights but that's not actually most of their job so automating that actually wouldn't automate their job and it wouldn't have that much of an impact on the economy so i think this is actually an important thing that important worldview difference that separates us from people people who are much more bullish because they think like jobs in the economy are much simpler in some sense and they're going to take like much fewer competencies to actually fully automate”

Claim: More bullish forecasts of automation often assume that jobs are simpler and require fewer competencies to automate fully than the speaker believes they do.

“I mean, I guess you could have made similar points five years ago and say, you know, you look at AlphaZero and there's this mini AGI there.”

Claim: The speaker says that five years earlier, AlphaZero could have been viewed as containing a “mini AGI.”

“And if only you unhuddled it by training it on text and giving it all your context and so on, like that just wouldn't really have worked.”

Claim: The speaker argues that adding text training and context to AlphaZero alone would not have produced the relevant capabilities.

“You do really need to rethink how you train these models in order to get these capabilities,”

Claim: Obtaining the discussed AI capabilities requires rethinking how models are trained.

“I think the surprising thing over the last few years has been that you can start off with this pre-trained corpus of the internet and it just like it's actually quite easy like chat gbt is an example of this like unhobbling where one percent of additional compute spent on getting it to talk in a chatbot like fashion with post training is enough to make it competent really competent at that capability so why not think that agency i mean reasoning is another example where it seems like the amount of computer”

Claim: ChatGPT illustrates that post-training using about 1% additional compute can make a model pretrained on internet data highly competent at chatbot interaction.

“that are spent on rl right now in these models is a small fraction of total compute again like reasoning seems like complicated and then you just like do one percent of compute and it gets you that why not think that computer use or long-term agency on computer use is a similar thing”

Claim: Reinforcement learning currently accounts for only a small fraction of the total compute used to train the discussed models.

“that are spent on rl right now in these models is a small fraction of total compute again like reasoning seems like complicated and then you just like do one percent of compute and it gets you that why not think that computer use or long-term agency on computer use is a similar thing”

Claim: Computer use or long-term computer agency might, like reasoning, be unlocked with a relatively small fraction of total compute.

“So when you say reasoning is easy and, you know, it only took this much compute and wasn't very much and maybe you look at the sheer number of tokens and it wasn't very much and so it looks easy. kind of true from our position today.”

Claim: In retrospect from the present, AI reasoning can appear to have required relatively little compute or training data.

“But I think if you ask someone, build a reasoning model in 2015, then it would have looked insurmountable. You would have had to train a model on tens of thousands of GPUs, you would have had to solve that problem,”

Claim: The speaker says that building a reasoning model would have seemed insurmountable in 2015.

“then it would have looked insurmountable. You would have had to train a model on tens of thousands of GPUs, you would have had to solve that problem,”

Claim: A reasoning model built in 2015 would have required training on tens of thousands of GPUs and overcoming major technical challenges.

“and each order of magnitude of scaling from where they were would pose new challenges that they would need to solve, you would need to produce kind of...”

Claim: In the 2015 context, every tenfold increase in scale would have introduced new challenges.

“of internet scale or tens of trillions of tokens of data in order to actually train a model that kind of has the knowledge that you can then unlock. and in your access by way of training”

Claim: Training a model with the knowledge that could later be unlocked would have required internet-scale data—tens of trillions of tokens.

“it to be a reasoning model you need to maybe make the model more efficient at kind of doing inference and maybe distill it because if it's very slow then then you have a reasoning model that's not particularly useful so you also need to make various innovations to you know get the model to be distilled so that you can train it more quickly because these rollouts take very long and it actually becomes a”

Claim: A useful reasoning model may require more efficient inference and possibly distillation, since a very slow reasoning model would be difficult to use.

“comes a product that's valuable if it's a couple tokens a second as a reasoning model that would have been very difficult to work with so in in some sense it looks easy from our point of view standing on this huge stack of technology that we've built up over the past five years or so but but at the time it would have been very hard and so my my claim would be something like i i think the agency part might be easy in a similar sense that in five years or three years time or whatever We will look at what unlocked agency and it'll look fairly simple, but the amount of work that, you know, in terms of these complementary kind of innovations that enable the model to be able to learn how to become a competent agent, that might have just been very difficult and taken years of innovation and a bunch of improvements in kind of hardware and scaling and various other things.”

Claim: The speaker thinks AI agency may eventually look easy in retrospect, although achieving it could require years of difficult complementary innovations, hardware improvements, and scaling.

“I mean, you could have looked at AlphaGo or AlphaGo Zero, AlphaZero, those seem very impressive at the time, and you're just learning to play this game with no human knowledge, you're just learning to play it from scratch. And I think at the time it did impress a lot of people.”

Claim: AlphaGo, AlphaGo Zero, and AlphaZero seemed impressive at the time because they learned to play games from scratch without human knowledge.

“But then people tried to apply it to math, they tried to apply it to other domains and it didn't work very well. They weren't able to get like competent agents at math.”

Claim: Applying the approaches behind AlphaGo-style systems to mathematics and other domains did not work very well or produce competent math agents.

“So it's very possible that these models, at least the way we have them right now, you're going to try to do the same thing people did for reasoning, but for agency it's not going to work very well and then you're not going to...”

Claim: It is very possible that current AI models will not work well for agency if researchers apply the same approach they used for reasoning.

“i think i guess said you'd be able to book a flight which is very different from having like full agentic computer use like a human”

Claim: Being able to book a flight is substantially different from having full human-like agentic computer use.

“yeah well i mean sure but like they are not like they are not disconnected tasks that's like saying like everything you do in the world is just like you just move parts of your body and then you like you like move your mouth in your tongue and then you like throw it in your head but like that's that's a very like yeah like individually those things are simple but then how do you put them together right”

Claim: Computer tasks are not disconnected merely because they consist of simpler actions; coordinating those actions into a complete task remains a challenge.

“Has been double AI's ability to do the kind of thing that it takes a human 10 minutes to do or an hour to do or four hours to do the length of time for corresponding human tasks. It seems like these models seem to be doubling their task length every seven months. So the idea being that by like 2030,”

Claim: AI models appear to be doubling approximately every seven months the duration of tasks they can perform, measured against how long those tasks take humans.

“if you extrapolate this curve, they could be doing tasks that take. humans one month to do or one year to do and then this like long-term coherency in executing a task is like fundamentally what intelligence is so this curve suggests that like we're getting there the other piece of evidence i kind of feel like my own mind works this way of”

Claim: Extrapolating the observed task-length trend suggests that by around 2030 AI models could perform tasks that take humans a month or a year to complete.

“humans one month to do or one year to do and then this like long-term coherency in executing a task is like fundamentally what intelligence is so this curve suggests that like we're getting there the other piece of evidence i kind of feel like my own mind works this way of”

Claim: The observed growth in AI task length suggests that AI is getting closer to intelligence understood as long-term coherence in task execution.

“humans one month to do or one year to do and then this like long-term coherency in executing a task is like fundamentally what intelligence is so this curve suggests that like we're getting there the other piece of evidence i kind of feel like my own mind works this way of”

Claim: Long-term coherence in executing a task is, in the speaker's view, a fundamental component of intelligence.

“Yeah, I mean, I would say that reasoning did seem, I mean, I think there's a reason to expect complex reasoning to not be as difficult as people might have thought, even in advance, because a lot of the tasks that AI solved very early on were tasks of various kinds of complex reasoning.”

Claim: Because many early AI successes involved forms of complex reasoning, there is reason to expect complex reasoning to be less difficult than people might have thought.

“reasoning that goes into when a human solves a math problem but uh if you look at the major ai milestones over i know since 1950 a lot of them are for complex reasoning like a chess is you can say a complex reasoning task go as you could say a complex reasoning task”

Claim: Many major AI milestones since 1950 have involved complex reasoning, including chess and Go.

“that is able to react to like very specific very particular kind of images then your or very specific video feeds or whatever then you're getting close to general sensory motor skill automation but the general skill is something that's very different and i think we're seeing that we're like um we can't like we still are very far it seems like from a ai model that can take a generic game of steam let's say you just download a game”

Claim: A system that reacts to specific images or video feeds would approach general sensory-motor skill automation, but that does not establish the ability to perform general skills.

“that is able to react to like very specific very particular kind of images then your or very specific video feeds or whatever then you're getting close to general sensory motor skill automation but the general skill is something that's very different and i think we're seeing that we're like um we can't like we still are very far it seems like from a ai model that can take a generic game of steam let's say you just download a game the game released this year you don't know how to play this game uh and then you just have to play it right and then most games are actually not that difficult for a human like uh they're”

Claim: AI models still seem far from being able to download and play an arbitrary new Steam game without prior familiarity.

“right so that's an interesting example first of all i find the example very interesting because yeah it was not trained explicitly like it wasn't they didn't do some rl on like playing pokemon red but obviously the model knows how it's supposed to play pokemon red because there's Because there's tons of material about Pokemon around the internet.”

Claim: Claude was not explicitly reinforcement-trained to play Pokémon Red, but its internet-derived knowledge includes information about how to play the game.

“and you got stuck somewhere you didn't know what to do you could probably go to Claude and ask it Claude like I'm stuck in Mount Moon and like what am I supposed to do and then it could probably be able to give you a fairly decent answer but that doesn't stop it from getting stuck in Mount Moon for 48 hours so that's a very interesting thing where it has explicit knowledge but then when it's actually playing the game it like it doesn't behave in a way which reflects that it has that knowledge”

Claim: Claude could probably explain how to get unstuck in Pokémon Red's Mount Moon, but may still remain stuck there for 48 hours when playing, showing that explicit knowledge need not translate into effective gameplay.

“I mean, a lot of the skills that people have that they don't have very good training data for them.”

Claim: Many human skills lack high-quality training data.

“Yeah, so I think something that would reveal its ability to do very long context things use you know, multimodal capabilities in a meaningful way and integrate that with reasoning and other types of systems and also agency and being able to take action over a long horizon and accomplish some tasks that takes very long for humans to do, not just in, you know, specific software environments, but just very broadly, say downloading an arbitrary game from Steam.”

Claim: Evidence of an AI integrating long-context processing, meaningful multimodal capabilities, reasoning, and long-horizon agency to complete time-consuming tasks across broad environments would substantially update the speaker's view.

“and you know something it's never seen before doesn't really have much training data maybe it was released after a training cutoff and so there's no tutorials or maybe there's no earlier versions of the game that has been you know discussed on the internet and then accomplishing that game and actually playing that game to the end and accomplishing these various milestones that you know are challenging for humans like that would be a substantial update I mean there are other things that would update me too”

Claim: An AI completing a previously unseen arbitrary Steam game with little or no relevant training data or online guidance, including challenging milestones and finishing it, would substantially update the speaker's view.

“i i think that's you know not a huge update to me if that were to happen um so i think the update would come if it was in fact 500 billion dollars in revenue or something like that then i would certainly update quite a lot”

Claim: OpenAI reaching $100 billion in revenue would not substantially update the speaker's view, whereas reaching about $500 billion would cause a substantial update.

“um but 100 billion that seems pretty kind of likely to me like not i would assign that maybe a 40 chance or something by the end”

Claim: The speaker estimates roughly a 40% chance that OpenAI will reach $100 billion in revenue by the end of the relevant period.

“People pay $100 billion for all sorts of things, right? And that itself is not a very strong piece of evidence that it's going to be transformative, I think.”

Claim: The fact that people pay $100 billion for something is not, by itself, strong evidence that it will be transformative.

“people pay trillions of dollars for oil but the oil is not like like i don't know it seems like a very basic point but like the fact that people pay a lot of money for something doesn't mean it's going to transform the world economy if only we manage to unhubble it like that's that's a very different claim right”

Claim: High spending on a resource, such as trillions of dollars spent on oil, does not imply that making the resource more abundant will transform the world economy.

“people pay trillions of dollars for oil but the oil is not like like i don't know it seems like a very basic point but like the fact that people pay a lot of money for something doesn't mean it's going to transform the world economy if only we manage to unhubble it like that's that's a very different claim right”

Claim: High spending on oil does not by itself imply that making oil easier to use would transform the world economy.

“a ton of B2B software companies start off by building self-serve consumer grade products,”

Claim: Many B2B software companies begin by building self-serve, consumer-grade products.

“Eventually, though, you have to go after enterprise.”

Claim: B2B software companies eventually need to pursue enterprise customers.

“The most successful and durable software companies of the last decade have all made this transition.”

Claim: According to the speaker, all of the most successful and durable software companies of the last decade transitioned to enterprise customers.

“But getting enterprise ready is hard. Single sign-on. role-based access controls and comprehensive audit logs are all actually quite complex and tedious to build and they're ripe for bugs and annoying edge cases.”

Claim: Preparing software for enterprise customers is difficult because features such as single sign-on, role-based access controls, and comprehensive audit logs are complex and tedious to build and prone to bugs and edge cases.

“These features take a ton of engineering time and capital which you should be spending on the core product.”

Claim: Building enterprise features consumes substantial engineering time and capital that the speaker believes should instead go toward the core product.

“For example one of Slack's PMs said that they've spent 30 million dollars building these features and they were only half done.”

Claim: A Slack product manager reportedly said Slack had spent $30 million on enterprise features and was only halfway through building them.

“WorkOS has helped Vercel. Plaid, Vanta, OpenAI, and hundreds others become enterprise ready with APIs to integrate all of these common features.”

Claim: WorkOS says it has helped Vercel, Plaid, Vanta, OpenAI, and hundreds of other companies become enterprise-ready by providing APIs for common enterprise features.

“so then this brings us to the intelligence explosion because what people will say is we don't need to automate literally everything that is needed for automating remote work, let alone all human labor. human labor in general we just need to automate the things which are necessary to fully close the R&D cycle needed to make smarter intelligences and if you do this you get a very rapid intelligence explosion”

Claim: One argument for an intelligence explosion is that automating the tasks needed to complete the R&D cycle for producing smarter intelligences could suffice; automating all human labor or all remote work is not required.

“human labor in general we just need to automate the things which are necessary to fully close the R&D cycle needed to make smarter intelligences and if you do this you get a very rapid intelligence explosion”

Claim: The speaker argues that automating the tasks needed to complete the R&D cycle for smarter intelligences would produce a very rapid intelligence explosion.

“And the end product of that explosion is not only an AGI, but something that is superhuman potentially.”

Claim: The speaker suggests that an intelligence explosion could produce systems beyond AGI that are potentially superhuman.

“These things are like extremely good at coding and they're good at the kinds of things that you would think and reasoning and like it seems like the kinds of things that would be necessary to automate R&D at AI labs.”

Claim: The speaker says the systems are very good at coding and reasoning, capabilities that appear necessary for automating R&D at AI labs.

“I mean, I think if you look at their capability profile, it is like if you compare it to like a random job in the economy. economy i agree they are better at doing sort of coding tasks that would be involved in r&d compared to like a random job in the economy but i in absolute terms i don't think they're like that good”

Claim: AI systems outperform a randomly selected worker on some coding tasks relevant to R&D, but the speaker judges their absolute capability to remain limited.

“I think they are good at things that maybe impress us about human coders, like if you want to see like, oh, like what makes a person a really impressive coder, you might look at their competitor programming performance.”

Claim: AI systems can perform well on coding tasks that are impressive indicators of human coding ability, such as competitive programming.

“I mean, in fact, companies often hire people based on, if they're relatively junior, based on their performance on these kinds of problems.”

Claim: Companies often use performance on competitive-programming-style problems when hiring relatively junior candidates.

“But that is just impressive in the human distribution.”

Claim: Performance on competitive-programming-style problems is impressive relative to the range of human ability.

“what are the skills you need to actually automate the process of being a researcher then what fraction of those skills do the AI systems actually have even in coding like a lot of coding is you have a very large code base you have to work with the instructions are very kind of vague there isn't for example you mentioned a meter eval in which because they needed to make it an eval all the tasks have to be kind of compact and close and have clear evaluation metrics like here's a model like get its loss on this uh you know data set as low as possible uh or whatever or like here's another model and it's like it's embedding matrix has been scrambled just uh fix it to recover like most of his original performance etc those are not problems that you actually work on in AI R&D. They're like very artificial problems.”

Claim: AI benchmark tasks often use compact problems with clear evaluation metrics, such as loss minimization or recovering performance from a scrambled embedding matrix, but the speaker says these are unlike actual AI R&D work.

“Now, if a human was good at doing those problems, you would infer, I think logically, that that human is likely to actually be a good researcher.”

Claim: Strong human performance on compact, clearly evaluated artificial problems would be evidence that the person is likely to be a good researcher.

“But if an AI is able to do them, like the AI lacks so many other competencies that a human would have, not just the researchers, an ordinary human, that we don't think about in the process of research.”

Claim: The speaker argues that AI systems capable of solving benchmark problems still lack many competencies possessed by ordinary humans, some of which are easy to overlook in research.

“So our view would be automating research is, first of all, more difficult. than people give me credit for i think you need more skills to do it and definitely more than models are displaying right now and on top of that even if you did automate the process of research we think a lot of the software progress has been driven not by uh cognitive efforts so that has played a part but it has been driven by compute scaling you just have more gpus you can do more experiments you can write more things your experiments can be done at larger scales and um”

Claim: The speakers believe research automation is harder than commonly assumed, requires a broader skill set, and exceeds the capabilities current models demonstrate.

“than people give me credit for i think you need more skills to do it and definitely more than models are displaying right now and on top of that even if you did automate the process of research we think a lot of the software progress has been driven not by uh cognitive efforts so that has played a part but it has been driven by compute scaling you just have more gpus you can do more experiments you can write more things your experiments can be done at larger scales and um That is just a very important driver.”

Claim: The speaker believes software progress has been driven substantially by compute scaling as well as cognitive effort, because additional GPUs allow more experiments and experiments at larger scales.

“If you're 10 years ago, 15 years ago, you're trying to figure out what software innovations are going to be important in 10 or 15 years, you would have had a very difficult time.”

Claim: Looking 10–15 years ahead from 10–15 years ago, it would have been very difficult to identify the software innovations that would become important.

“In fact, you probably wouldn't even conceive of the right kind of innovations to be looking at because you would be so far removed from the context of that time with much more abundant computers and all the things that people would have learned. learned by that point.”

Claim: People forecasting software innovations far in advance would probably fail to conceive of the right innovations because they lack the future context of more abundant computing and accumulated knowledge.

“So these are two components of our view. Research is harder than people think and depends a lot on computer scale.”

Claim: The speakers summarize their view as: research is harder than commonly assumed, and progress in research depends substantially on compute scale.

“I think it's like, you know, examples might be introducing novel. having novel innovations that are very useful for unlocking innovations in the future.”

Claim: Research may involve creating novel innovations that enable further innovations, rather than merely training or debugging a classifier.

“So that might be, you know, introducing some novel way of thinking about a problem introducing.”

Claim: Research innovation can involve introducing a novel way to think about a problem.

“So maybe a good example might be in mathematics where we have these reasoning models that are extremely good at solving math problems. very short horizon setups.”

Claim: The speaker says reasoning models are extremely good at solving math problems in very short-horizon setups.

“And so, you know, they can do that very well, but they're not very good at coming up with novel conceptual schemes that are useful for making progress in mathematics. you know, it's able to solve these problems that you can kind of neatly excise out of some very messy context.”

Claim: Reasoning models can solve well-isolated problems, but are not very good at devising novel conceptual frameworks that advance mathematics.

“But within some much messier context, it's kind of not very good at figuring out what directions are especially useful for you to build things or kind of make incremental progress on that enables you to have a big kind of innovation later down the line. So thinking about both this like larger context as well as maybe much longer horizon. much fuzzier things that you're optimizing for. I think it's much worse at those types of things.”

Claim: Reasoning models are less capable of identifying useful incremental directions in messy, long-horizon contexts with vague objectives.

“they know so much, especially the large ones, they know in literal terms more than any human does in some sense.”

Claim: Large reasoning models possess more literal factual knowledge than any individual human.

“unlocked these reasoning capabilities on top of that knowledge and i think that is actually what has enabled them to solve a lot of these problems but uh if you actually look at the way they approach problems they um”

Claim: The speaker believes that adding reasoning capabilities to models' extensive knowledge has enabled them to solve many problems.

“The reason what they do looks impressive to us is because we have so much less knowledge. And the model is approaching the problems in a fundamentally different way compared to a human would.”

Claim: Reasoning models solve problems differently from humans, and their performance can look impressive partly because humans have substantially less knowledge.

“A human would have much more limited knowledge and they would usually have to be much more creative in solving problems because they have this lack of knowledge.”

Claim: Humans often rely more on creativity in problem-solving because they have less knowledge available to them.

“While the model knows so much, you'd ask it some obscure math question where you need some specific theorem from 1850 or something. something and then it would just like know that if it's like a large model so that makes the difficulty profile very different and if you look at the way they approach problems the reasoning models they are usually not”

Claim: A large reasoning model can answer obscure math questions by recalling a specific theorem from its extensive knowledge.

“creative.”

Claim: Reasoning models are usually not creative.

“They are very effectively able to leverage the knowledge they have, which is extremely vast, and that makes them very effective in a bunch of ways.”

Claim: Reasoning models use their extensive knowledge effectively, making them effective across many tasks.

“has a reasoning model ever come up with a math concept that even seems like slightly interesting to a human mathematician? and i've never seen that i mean they've been around for all of six months but”

Claim: The speaker has not seen a reasoning model generate a mathematical concept that a human mathematician would find even slightly interesting.

“and i've never seen that i mean they've been around for all of six months but”

Claim: Reasoning models had existed for only about six months at the time of the conversation.

“which is this idea of kind of more of X paradox, that things that seem very hard for humans, AI systems tend to make much faster progress on, whereas things that look a bunch easier for us, AI systems are totally struggle or often totally incapable of doing that thing.”

Claim: AI systems often progress faster on tasks humans find difficult, yet struggle with or cannot perform tasks humans find easy.

“And so, you know, this kind of abstract reasoning, you know, playing chess, playing Go, maybe playing Jeopardy, doing kind of advanced math and solving math problems.”

Claim: AI systems have made progress on abstract reasoning, chess, Go, Jeopardy, and advanced mathematical problem-solving.

“Yeah there are even stronger examples like multiplying 100 digit numbers in your head Which is just the one that got solved first out of almost any other problem or like”

Claim: AI solved the task of multiplying 100-digit numbers mentally earlier than almost any other problem.

“following like very complex sort of symbolic logic arguments like deductive arguments or people actually people actually struggle with that a lot. Like, how do premises like logically fall from conclusions? People have a very hard time with that. Very easy for formal proof systems.”

Claim: People often find complex symbolic deduction difficult, whereas formal proof systems can readily establish how conclusions follow from premises.

“An insight that is related and is quite important here is that the kind of very, the tasks that humans seem to struggle on and AI systems seem to make much faster progress on are things that kind of emerged fairly recently in evolutionary time.”

Claim: Tasks that humans find difficult but AI systems advance on quickly often arose relatively recently in evolutionary history.

“So, you know, advanced language use emerged in humans maybe 100,000 years ago,”

Claim: Advanced language use may have emerged in humans approximately 100,000 years ago.

“and certainly playing chess and go and so on are very recent innovations.”

Claim: Chess and Go are recent innovations when considered on an evolutionary timescale.

“And so evolution has had much less time to optimize for them,”

Claim: Evolution has had less time to optimize humans for recently emerged abilities such as advanced language use and games like chess and Go.

“in part because they're very new, but also in part because when they emerged, there was a lot less pressure because it conferred kind of small fitness gains to humans. And so evolution didn't optimize for these things very strongly.”

Claim: Evolution did not strongly optimize humans for recently emerged abilities because they initially provided only small fitness advantages.

“And so it's not surprising that on these specific tasks that, you know, humans find very impressive when other humans are able to do it, that AI systems are able to make a lot of fast progress.”

Claim: AI systems' rapid progress on specific tasks that humans find impressive is unsurprising, given that evolution did not strongly optimize humans for those tasks.

“In humans, these things are often very strongly correlated with other kind of competencies like being good at just, you know, achieving your goals or being a good coder is often very strongly correlated with solving. kind of coding problems or being a good engineer is often correlated with solving competitive coding problems but in AI systems the correlation isn't quite as strong and even”

Claim: Among humans, general goal achievement and engineering or coding ability often correlate strongly with performance on coding and competitive-programming problems.

“kind of coding problems or being a good engineer is often correlated with solving competitive coding problems but in AI systems the correlation isn't quite as strong and even Even within AI systems, it's the case that, you know, the strongest systems on competitive programming are not even the ones that are best at actually helping you code.”

Claim: AI systems show weaker correlations between competitive-programming performance and broader coding ability than humans do.

“it's the case that, you know, the strongest systems on competitive programming are not even the ones that are best at actually helping you code.”

Claim: The AI systems that perform best on competitive programming are not necessarily the best coding assistants.

“So like, you know, O3 minis, high seems to be maybe the best at solving competitive code problems, but it isn't the best at actually helping you write code”

Claim: O3 mini high appears to perform very well on competitive-coding problems but is not the best model for helping users write code.

“Right, right, right. So but an important insight here is that, you know, the things that we find very impressive when humans are able to do it, we should expect that AI systems are able to make a lot more progress on that.”

Claim: AI systems should be expected to make substantial progress on tasks that humans find impressive.

“And but we shouldn't update too strongly about just their general competence or something, because we should recognize that this is a very narrow subset of relevant tasks that humans do in order to be a competent. and economically viable agent.”

Claim: Impressive human-performed tasks are a narrow subset of the tasks required for competence and economic viability, so progress on them should not strongly update beliefs about AI systems' general competence.

“um but he's i mean i think he's like he in particular has not been that well calibrated um he's like oh like in 2018 he had like 2020”

Claim: The speaker considers Dario's forecasts poorly calibrated and cites a 2018 expectation that something would happen by 2020.

“i was asking scott about this the other day when i interviewed him scott alexander and he said like look humans also don't have this kind of logical omissions i'm not saying more omniscient but we have examples of humans finding these kinds of connections this is not an uncommon thing right i think his response to that was that these things are just not trained in in order to find these kinds of connections but if you if you like their view is that it would not take that much extra compute in order to build some rl environment in which they're incented are incentivized to find these connections next token prediction just isn't incentivizing them to do this but the rl required to do this would not be that or set up some sort of scaffolds i think actually google deep mind did do some similar like scaffold to make new discoveries and i didn't look into like how impressive the new discovery was they claim that some new discovery was made by an llm as a result um”

Claim: The speaker describes a view that models are not trained to find certain connections, but that an RL environment incentivizing this behavior might require relatively little additional compute.

“i was asking scott about this the other day when i interviewed him scott alexander and he said like look humans also don't have this kind of logical omissions i'm not saying more omniscient but we have examples of humans finding these kinds of connections this is not an uncommon thing right i think his response to that was that these things are just not trained in in order to find these kinds of connections but if you if you like their view is that it would not take that much extra compute in order to build some rl environment in which they're incented”

Claim: Humans sometimes find the kinds of logical connections under discussion, and such examples are not uncommon.

“are incentivized to find these connections next token prediction just isn't incentivizing them to do this but the rl required to do this would not be that or set up some sort of scaffolds i think actually google deep mind did do some similar like scaffold to make new discoveries and i didn't look into like how impressive the new discovery was they claim that some new discovery was made by an llm as a result um”

Claim: Next-token prediction training does not incentivize models to discover the connections being discussed.

“are incentivized to find these connections next token prediction just isn't incentivizing them to do this but the rl required to do this would not be that or set up some sort of scaffolds i think actually google deep mind did do some similar like scaffold to make new discoveries and i didn't look into like how impressive the new discovery was they claim that some new discovery was made by an llm as a result um”

Claim: The speaker recalls that Google DeepMind used a similar scaffold for an LLM to make a claimed new discovery, but had not assessed the discovery's significance.

“But I would also say there that if you compare animals to humans, long-term intelligent planning like an animal is not gonna like help you book a flight either an animal or like an animal is not gonna like do remote work for you um or even do the kinds of things i think like what separates humans from other animals is that we can hold long-term we can like make come up with a plan and execute on it whereas other animals i often had to go by instinct or within the kinds of environments that they have evolutionary um uh knowledge of rather than like i'm put in in the middle of the savannah or i'm putting the middle of”

Claim: Humans differ from other animals in their ability to formulate and execute long-term plans; other animals often rely on instinct or evolutionarily familiar environments.

“middle of the desert or i'm part of the middle of the tundra and i'll learn how to make use of the tools and whatever they're um like i actually think there's like a huge discontinuity between humans and animals and their ability to survive in different environments uh just based on their knowledge and”

Claim: Humans and other animals differ greatly in their ability to survive in unfamiliar environments, partly because of differences in knowledge.

“You could put the competencies that the animals have into AI systems that might just already get you to like AGI, like already.”

Claim: AI systems equipped with animal competencies might already reach AGI.

“discontinuity between animals and humans is because animals have to rely entirely on natural world data basically to train themselves like imagine that the only thing as a human that you saw was nobody talked to you you didn't read anything you just had to learn by experience maybe to some extent by imitating other people but not that you have no experts in communication what would be very inefficient”

Claim: Animals rely entirely on natural-world data for learning, which the speaker identifies as a reason for the discontinuity between animals and humans.

“efficient like what's actually happening is that you have this we got some other people that made this point as well is that evolution is sort of this outer optimizer that's like improving the software efficiency the brain in a bunch of ways there's some genetic knowledge that you inherit not that much because there isn't that much space in the genome and then you have this lifetime learning which is you don't actually see that much data during lifetime learning a lot of this is redundant and so on so what changed seems to have changed with”

Claim: Evolution functions as an outer optimizer that improves brain software efficiency in multiple ways.

“like improving the software efficiency the brain in a bunch of ways there's some genetic knowledge that you inherit not that much because there isn't that much space in the genome and then you have this lifetime learning which is you don't actually see that much data during lifetime learning a lot of this is redundant and so on so what changed seems to have changed with”

Claim: Humans inherit some genetic knowledge, but the amount is limited by the genome's limited capacity.

“like improving the software efficiency the brain in a bunch of ways there's some genetic knowledge that you inherit not that much because there isn't that much space in the genome and then you have this lifetime learning which is you don't actually see that much data during lifetime learning a lot of this is redundant and so on so what changed seems to have changed with”

Claim: Human lifetime learning involves relatively little data, much of which is redundant.

“with humans compared to other animals is that humans became able to have culture and they have language which enables them to like have like a much more efficient training data modality”

Claim: The development of human culture and language enabled a more efficient modality for training and learning.

“compared to animals they also have I think stronger ways in which they like tend to imitate other humans and learn from their skills so that also enables this knowledge to be passed on I think animals are pretty bad at that compared to humans so basically as a human you're just being trained on like much more efficient data and that creates further insights to be then efficient”

Claim: Humans’ ability to learn from more efficient data generates further insights that improve learning efficiency.

“compared to animals they also have I think stronger ways in which they like tend to imitate other humans and learn from their skills so that also enables this knowledge to be passed on I think animals are pretty bad at that compared to humans so basically as a human you're just being trained on like much more efficient data and that creates further insights to be then efficient”

Claim: Humans are better than animals at imitating others and learning from their skills, enabling more effective transmission of knowledge.

“at learning from it and then that creates this feedback loop where the selection pressure gets much more intense so i think that's roughly what happened with humans but a lot of the capabilities that you need to be like a good worker in the human economy animals already have so they are able to like they have quite sophisticated sensor motor skills i think they are actually able to do uh”

Claim: The speaker thinks that increasingly efficient learning created a feedback loop that intensified selection pressure during human evolution.

“at learning from it and then that creates this feedback loop where the selection pressure gets much more intense so i think that's roughly what happened with humans but a lot of the capabilities that you need to be like a good worker in the human economy animals already have so they are able to like they have quite sophisticated sensor motor skills i think they are actually able to do uh”

Claim: Animals already possess many of the capabilities needed to be effective workers in the human economy.

“at learning from it and then that creates this feedback loop where the selection pressure gets much more intense so i think that's roughly what happened with humans but a lot of the capabilities that you need to be like a good worker in the human economy animals already have so they are able to like they have quite sophisticated sensor motor skills i think they are actually able to do uh”

Claim: Animals possess sophisticated sensorimotor skills.

“like animals are actually able to pursue long-term goals uh”

Claim: Animals are capable of pursuing long-term goals.

“put a human and they haven't seen the relevant train later i think they'd like they but not that much better like i think a lot of the like again didn't you have recently have Joseph an interview yeah”

Claim: The speaker thinks a human unfamiliar with an environment and its relevant prey would perform only slightly better than an animal at finding that prey.

“They say that ASI will be this huge discontinuity because, well, we have this huge discontinuity. in the animal to human transition um where like something is like not that much change between pre-human primates and humans genetically but it resulted in this humongous change in capabilities and so they say well why not expect similar things similar between human level intelligence and superhuman intelligence and the point you're making is that”

Claim: Some people argue that artificial superintelligence will cause a major discontinuity, drawing an analogy to the large capability change between nonhuman primates and humans despite relatively little genetic change.

“Animals have just been like held back in this really weird way that no AI system has been like arbitrarily held back of like not being able to communicate with other copies or with other knowledge sources. And so since AIs are not held back artificially in this way.”

Claim: The speaker suggests that animals’ inability to communicate with other individuals or knowledge sources constrained them in a way that AI systems have not been constrained.

“there's not going to be a point where you should take away that hobbling i mean you know and then now they're like now they explode now actually i think I would disagree with that. The implication that I've made, I would actually disagree with.”

Claim: The speaker rejects the implication that removing an artificial constraint from AI will by itself trigger an explosive capability increase.

“actually there will be a similar unhobbling with future AIs, which is not about the intelligence, but a similar level of bandwidth and communication and collaboration with other AIs. magnitude of change from non-human animals to humans in terms of their social collaboration that AIs will have with each other because of their ability to copy all their knowledge exactly to merge to distill themselves”

Claim: The speaker expects future AIs to experience a major increase in bandwidth, communication, and collaboration with other AIs, analogous in scale to the social-collaboration change from nonhuman animals to humans, rather than a comparable increase in intelligence.

“magnitude of change from non-human animals to humans in terms of their social collaboration that AIs will have with each other because of their ability to copy all their knowledge exactly to merge to distill themselves”

Claim: The speaker expects AIs to be able to copy their knowledge exactly, merge, and distill themselves.

“is this like focus on intelligence and reasoning and, you know, R&D, which is enabled by that intelligence as being just enormously important.”

Claim: The speaker considers intelligence and reasoning, including their role in research and development, enormously important.

“In the same way that humans have much more sophisticated intelligence compared to non-human primates. And I think part of our disagreement is that intelligence is kind of important, but just having a lot more intelligence and reasoning and good reasoning isn't something that will kind of accelerate technological change and economic growth very substantially.”

Claim: The speaker argues that greater intelligence and better reasoning alone will not substantially accelerate technological change or economic growth.

“but just having a lot more intelligence and reasoning and good reasoning isn't something that will kind of accelerate technological change and economic growth very substantially.”

Claim: Greater intelligence and better reasoning alone will not substantially accelerate technological change or economic growth.

“Like it isn't the case that the world today is just like... totally bottlenecked by not having you know not having enough good reasoning and that that's not really what's bottlenecking the world's ability to grow much more substantially i think that we might have some disagreement about this particular argument but i think what's also really important is just that we have a different view as to how this acceleration happens that it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically because that that alone”

Claim: The speaker argues that insufficient good reasoning is not the main bottleneck limiting substantial growth in the world economy.

“you know not having enough good reasoning and that that's not really what's bottlenecking the world's ability to grow much more substantially i think that we might have some disagreement about this particular argument but i think what's also really important is just that we have a different view as to how this acceleration happens that it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically because that that alone Alone is not sufficient.”

Claim: Having many highly capable reasoners produce technology is not sufficient by itself to cause drastic acceleration.

“you know not having enough good reasoning and that that's not really what's bottlenecking the world's ability to grow much more substantially i think that we might have some disagreement about this particular argument but i think what's also really important is just that we have a different view as to how this acceleration happens that it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically because that that alone Alone is not sufficient.”

Claim: A large number of highly capable reasoners producing technology is not, by itself, sufficient to cause drastic acceleration.

“you know not having enough good reasoning and that that's not really what's bottlenecking the world's ability to grow much more substantially i think that we might have some disagreement about this particular argument but i think what's also really important is just that we have a different view as to how this acceleration happens that it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically because that that alone”

Claim: A lack of good reasoning is not the main bottleneck limiting substantial growth in the world’s economy.

“You need kind of complementary innovations in other industries.”

Claim: Technological acceleration requires complementary innovations in other industries.

“You need kind of complementary innovations in other industries.”

Claim: Drastic technological and economic acceleration requires complementary innovations in other industries.

“You need the economy as a whole growing and supporting the development of these various technologies.”

Claim: Broad acceleration requires the overall economy to grow and support the development of multiple technologies.

“You need the economy as a whole growing and supporting the development of these various technologies.”

Claim: Broad technological development requires the overall economy to grow and support the development of multiple technologies.

“You need the various supply chains to be upgraded.”

Claim: Broad technological and economic acceleration requires upgrading the relevant supply chains.

“You need the various supply chains to be upgraded.”

Claim: Broad technological acceleration requires upgrading the relevant supply chains.

“You might need demand for the various products that are being built.”

Claim: The speaker thinks demand for newly developed products might be necessary for broad technological and economic acceleration.

“You might need demand for the various products that are being built.”

Claim: Demand for newly developed products might be necessary for technological acceleration.

“And so we have this view where actually this very broad upgrading of your technology and your economy is important. is important rather than just having very good reasoners and very very very good reasoning tokens that gives us this acceleration”

Claim: Broad technological and economic upgrading is important for acceleration, rather than relying solely on highly capable reasoners and reasoning tokens.

“And so we have this view where actually this very broad upgrading of your technology and your economy is important. is important rather than just having very good reasoners and very very very good reasoning tokens that gives us this acceleration”

Claim: The speaker considers broad technological and economic upgrading more important for acceleration than simply having highly capable reasoners and abundant reasoning tokens.

“if you just look at like what kinds of progress is needed to make a more general intelligence you might be right that it needs more experimental compute but like we're just getting as you guys have documented we're just getting like a shit ton more compute every single year for the next few years so you can imagine that intelligence for the next two years where in 2027 there'll be like 10x more compute than there is now for AI and you'll have this effect where the AIs that are doing software R&D”

Claim: AI compute is expected to be about 10 times its current level in 2027.

“R&D are finding ways to make running copies of them more efficient,”

Claim: AI systems conducting software R&D can discover ways to run copies of themselves more efficiently.

“One, you're increasing the population of AIs who are doing this research, so more of them in parallel can find these different optimizations.”

Claim: More efficient AI copies increase the number of AI researchers and enable more parallel searches for optimizations.

“And a subtle point that they'd often make here is software R&D in AI is not just alien type coming up with new transformer like architectures to your point it actually is a lot of like you gotta like i mean i'm not an ai researcher but i assume there's like from the lowest level libraries to the kernels to making rl environments to finding the best optimizer to there's just like so much to do and in like parallel you can be doing all these things or finding optimizations across them and so you have two effects going back to this”

Claim: AI software R&D includes work on libraries, kernels, reinforcement-learning environments, and optimizers, not only the invention of new transformer architectures.

“a hundred times the same capability or something.”

Claim: The speaker estimates that GPT-4o can be run at roughly one-hundredth the cost of the original GPT-4 for similar capability.

“or to develop a copy the more parallel experiments you can run because now you can do a gpt4 scale training run for much cheaper than you could do it in 2024 or 2023 um and for that reason also this like software only singularity sees more researcher copies who can run experiments for cheaper dot dot dot they initially are maybe handicapped in certain um”

Claim: Lower costs for running or developing AI copies enable more parallel experiments, including GPT-4-scale training runs at much lower cost than in 2023 or 2024.

“But through this process, they are rapidly becoming much more capable.”

Claim: AI systems become substantially more capable through the process of using AI copies to conduct software R&D and experiments.

“So I think the logic, like the logic seems fine.”

Claim: The speaker thinks the logic of the intelligence-explosion argument seems sound.

“But I think that it's useful to draw on a bunch of work that, say, economists have done for studying the returns to R&D and what happens if you 10x your inputs, the number of researchers, what happens to you. to innovation or the rate of innovation.”

Claim: Research on the returns to R&D by economists can help assess how innovation changes when research inputs, including researcher numbers, increase tenfold.

“And there are, you know, they point out these kind of two effects where, you know, as you do more innovation, then you get to kind of stand on top of the shoulders of giants and you get the benefit from past discoveries and it makes you as a scientist more productive.”

Claim: Past discoveries can make scientists more productive by allowing them to build on prior work.

“But then there's also kind of diminishing returns that the low hanging fruit has been picked and it becomes harder to make progress.”

Claim: Research effort faces diminishing returns as low-hanging fruit is picked and further progress becomes harder.

“Those estimates is thinking about the kind of returns to research effort and, you know, we've looked into the returns to research effort in software specifically and we look at a bunch of domains in traditional software or, you know, like linear integer solvers or SAT solvers, but also in AI, like computer vision and RL and language modeling. And there, like if this model is true that all you need to do,”

Claim: The speaker’s group has examined returns to research effort across traditional software, including linear/integer and SAT solvers, and AI, including computer vision, reinforcement learning, and language modeling.

“like computer vision and RL and language modeling. And there, like if this model is true that all you need to do, All you need is just cognitive effort. It seems like the estimates are a bit ambiguous about whether this results in this acceleration or whether it results in just merely exponential growth.”

Claim: Available estimates do not clearly establish whether scaling research effort causes acceleration beyond exponential growth or only exponential growth.

“it isn't just your research effort that you have to scale up to make these innovations because you might have complementary input. So as you mentioned, experiments are the thing that might kind of bottleneck you.”

Claim: Increasing research effort alone may not produce innovations if complementary inputs, such as experiments, remain bottlenecks.

“And I think there's a lot of evidence that in fact these experiments and scaling up hardware is just very important for getting progress in the algorithms and the architecture and so on. So in AI,”

Claim: Experiments and scaling hardware are important contributors to progress in algorithms and architectures.

“this is true for software in general, where if you look at progress in software, it often matches very closely the rate of progress we see in hardware.”

Claim: The rate of software progress often closely tracks the rate of hardware progress.

“So for traditional software, we see about a third. a 30% roughly increase per year, which kind of basically matches Moore's Law.”

Claim: Traditional software progress has grown by about 30% annually, roughly matching Moore’s Law.

“And in AI, we've seen the same until you get to the deep learning era and then you get this acceleration,”

Claim: AI progress had followed a similar pattern to traditional software until the deep-learning era, when it accelerated.

“which in fact coincides with the acceleration we see in compute scaling, which gives you a hint that actually the compute scaling might have been very important.”

Claim: AI progress accelerated alongside compute scaling, which may indicate that compute scaling was very important to the acceleration.

“Other kind of pieces of evidence are, you know, the fact that innovation in algorithms and architectures are often concentrated in GPU-rich labs and not in the GPU-poor parts of the world like academia or maybe smaller research institutes that also suggest that having a lot of hardware is very important. If you look at specific innovations that seem very important.”

Claim: Important algorithmic and architectural innovations are often concentrated in GPU-rich labs rather than GPU-poor academic or small-institute settings, suggesting that access to substantial hardware matters.

“The big innovations over the past five years, many of them have some kind of scaling or hardware-related motivation.”

Claim: Many major innovations from the past five years were motivated in some way by scaling or hardware considerations.

“So you might look at the transformer itself was about how to harness more parallel compute.”

Claim: The Transformer architecture was developed to make better use of parallel compute.

“Things like flash attention was literally about how to implement the attention mechanism more efficiently.”

Claim: FlashAttention was developed to implement attention more efficiently.

“Or things like the chinchilla scaling law. And so many of these big innovations were just... about how to harness your compute more effectively that also tells you that actually the scaling of compute might be very important and I think there's just like many pieces of evidence that points towards this complementarity picture so I would say that not only like even if you assume that experiments are not particularly important the evidence we have both from estimates of AI and other software although the data is a bit”

Claim: Many major AI innovations, including those related to the Chinchilla scaling law, focused on using compute more effectively.

“about how to harness your compute more effectively that also tells you that actually the scaling of compute might be very important and I think there's just like many pieces of evidence that points towards this complementarity picture so I would say that not only like even if you assume that experiments are not particularly important the evidence we have both from estimates of AI and other software although the data is a bit”

Claim: Multiple lines of evidence support the view that compute and other inputs are complementary contributors to AI progress.

“about how to harness your compute more effectively that also tells you that actually the scaling of compute might be very important and I think there's just like many pieces of evidence that points towards this complementarity picture so I would say that not only like even if you assume that experiments are not particularly important the evidence we have both from estimates of AI and other software although the data is a bit A bit is not great suggests that, you know, maybe you don't get this kind of hyperbolic faster than exponential super growth in the overall algorithmic efficiency of systems.”

Claim: Even assuming experiments are not especially important, imperfect estimates from AI and other software suggest that overall algorithmic efficiency may not grow hyperbolically or faster than exponentially.

“I'm not sure I buy the argument that because these two things, compute and AI progress, have risen so concomitantly that this is a sort of causal relationship.”

Claim: The interviewer doubts that the simultaneous increases in compute and AI progress demonstrate that compute caused the progress.

“the industry as a whole has been getting more compute and as a result making more progress.”

Claim: The AI industry as a whole has gained compute while making more progress.

“But if you look at the top players, there's been multiple examples of a company with much less compute but a more coherent vision, more concentrated research effort, being able to beat incumbent who has much more compute.”

Claim: Some companies with substantially less compute have beaten incumbents with more compute by having a more coherent vision and more concentrated research effort.

“So OpenAI initially beating Google DeepMind. And if you remember, there was these emails that were released between. between elon and sam and so forth we're like we got to start this company because they've got this bottleneck on the compute and like look how much more compute google deep mind has um and then open air made a lot of progress similarly now with open air versus anthropic and so forth”

Claim: The interviewer cites OpenAI’s early success against Google DeepMind as an example of a company with less compute beating a more compute-rich incumbent.

“and then i think just generally your argument is just like too outside view when we just do know a lot about like what is like it's just like this very macroeconomic argument that i'm like well why don't we just ask the ai researchers”

Claim: The interviewer considers the guest’s argument overly reliant on a macroeconomic outside view and proposes consulting AI researchers directly.

“I mean, AI researchers will often kind of overstate the extent to which just cognitive effort and doing research is important for driving these innovations because that's often kind of convenient or useful.”

Claim: AI researchers may overstate how much cognitive effort and research drive innovation because that account is convenient or useful for them.

“They will say the insight was derived from some kind of nice idea about statistical mechanics or some nice equation in physics that says that we should do it this way. But often that's kind of an ad hoc story that they tell to make it a bit more compelling to the kind of reviewers that they submit.”

Claim: Researchers sometimes attribute insights to statistical mechanics or physics equations, but these explanations may be ad hoc accounts used to make submissions more compelling to reviewers.

“Daniel mentioned this like survey he did where he. the Daniel Cucutalo. he asked a bunch of ai researchers if you had 1 30th the amount of compute and he did 1 30th because ai is supposed to think 30 times faster if you had 1 30th the amount to compute how much how much would your progress slow down and they say i make a third of the amount of progress i normally do so that's just a pretty good like substitution effect of you if you get one tenth to compute your progress only goes down one third um i was talking to an ai uh researcher the other day who's”

Claim: A survey by Daniel asked AI researchers how their progress would change with one-thirtieth as much compute, and respondents said they would make one-third as much progress.

“he asked a bunch of ai researchers if you had 1 30th the amount of compute and he did 1 30th because ai is supposed to think 30 times faster if you had 1 30th the amount to compute how much how much would your progress slow down and they say i make a third of the amount of progress i normally do so that's just a pretty good like substitution effect of you if you get one tenth to compute your progress only goes down one third um i was talking to an ai uh researcher the other day who's who's like just like one of these like cracked people gets paid millions and tens of millions of dollars a year probably um and we asked him how much do these ai models help you in domains you already are you know how much does the ai models help you in ai research and he said in domains that i'm already quite familiar with where just closer to auto complete it's like saves me four to eight hours a week um and”

Claim: One AI researcher reported that AI models save four to eight hours per week in familiar domains, where they function more like autocomplete.

“skeptical of the claims that we have actually seen that much of an acceleration in the process of R&D. These claims seem to me like they're not borne out by the actual data I'm seeing.”

Claim: The speaker is skeptical that R&D has accelerated substantially because the claims do not appear to be supported by the data they have seen.

“So I'm not sure how much to trust them”

Claim: The speaker is uncertain how much to trust claims that R&D has accelerated substantially.

“I mean, more on the general intuition that cognitive effort alone can give you a lot of AI progress.”

Claim: Cognitive effort alone can produce substantial progress in AI.

“We've had it seems like a big important thing the labs do is this like science of deep learning,”

Claim: The speaker says that developing the science of deep learning is an important activity for AI labs.

“like scaling laws is just you. I mean, like it ultimately knitted out an experiment, but the experiment is motivated by cognitive effort.”

Claim: The speaker characterizes scaling-law experiments as ultimately being motivated by cognitive effort.

“So when you say you need a computer and experiments and data, but you also need cognitive effort Like that doesn't mean the lab who has the most computers gonna win right like that's a very simple point like either one can be the bottleneck”

Claim: The lab with the most computers will not necessarily win, because cognitive effort, compute, experiments, or data can each be the bottleneck.

“you just have a really dysfunctional culture and you can't you don't actually prioritize using your computer very well and you just waste it well then you're not going to make a lot of progress right so like it's it doesn't contradict the picture that someone with a much better vision a much better team much better prioritization can make better use of their compute if someone else was just bottlenecked heavily on that part of the equation the question here is once”

Claim: An organization may make little progress if it has a dysfunctional culture, fails to prioritize compute use, or wastes its computing resources.

“you just have a really dysfunctional culture and you can't you don't actually prioritize using your computer very well and you just waste it well then you're not going to make a lot of progress right so like it's it doesn't contradict the picture that someone with a much better vision a much better team much better prioritization can make better use of their compute if someone else was just bottlenecked heavily on that part of the equation the question here is once”

Claim: A team with better vision, personnel, and prioritization can make better use of its compute than a less effective team.

“Once you get these automated AI researchers and you start this software singularity, your software efficiency is going to improve by many orders of magnitude,”

Claim: The speaker predicts that automated AI researchers and a software singularity would lead to software efficiency improving by many orders of magnitude.

“while your compute stock, at least in sort of the short run, is going to remain fairly fixed.”

Claim: The speaker expects the available compute stock to remain fairly fixed in the short run.

“like how much progress should you expect that's the kind of question and i think people don't have i think it's hard for people to have good intuitions about this because people usually don't run the experiment so you don't get to see at a company level or at an industry level what would have happened if the entire industry had 30 times less compute maybe as an individual like what would happen if you had 30 times less compute you might have a better idea about but that's a very local experiment and you might be benefiting a lot from spillovers from other people who like actually have more compute”

Claim: An individual's experience with 30 times less compute may offer some evidence, but it is a local experiment that may be affected by spillovers from others with more compute.

“like how much progress should you expect that's the kind of question and i think people don't have i think it's hard for people to have good intuitions about this because people usually don't run the experiment so you don't get to see at a company level or at an industry level what would have happened if the entire industry had 30 times less compute maybe as an individual like what would happen if you had 30 times less compute you might have a better idea about”

Claim: People often lack good intuitions about the progress effects of changing compute because the relevant experiments are rarely conducted.

“So because this experiment was never run it's sort of hard to Get direct evidence about the strength of complementarity”

Claim: The absence of the relevant experiment makes it difficult to obtain direct evidence about how strong the complementarities are.

“Quite high because because”

Claim: If AGI arrives in 2027, the speaker assigns a quite high probability to a software-only singularity.

“compute not being very large so it must be that like you know you get a bunch of software progress yeah”

Claim: The speaker reasons that if AGI arrives in 2027 despite compute not being very large, substantial algorithmic or software progress must be providing leverage.

“i think a a color that i want to make is i i know that some labs do have multiple pre-training teams and they give people different amounts of resources for for doing the training and different amounts of cognitive effort”

Claim: Some labs reportedly have multiple pre-training teams that differ in training resources, cognitive effort, and team size.

“different size of teams but but but none of that I think has been you know published and I would love to see the results of some of those experiments I mean I think even that won't update you very strongly just because it is often just very inefficient to do this very imbalanced scaling of your factor inputs and in order to really get an estimate of how strong these complementarities are you need to observe these very imbalanced scale ups and”

Claim: To estimate how strong complementarities are, one needs to observe very imbalanced scale-ups.

“different size of teams but but but none of that I think has been you know published and I would love to see the results of some of those experiments I mean I think even that won't update you very strongly just because it is often just very inefficient to do this very imbalanced scaling of your factor inputs and in order to really get an estimate of how strong these complementarities are you need to”

Claim: The speaker says that results from labs' experiments varying pre-training resources and cognitive effort have not been published and expresses interest in seeing them.

“different size of teams but but but none of that I think has been you know published and I would love to see the results of some of those experiments I mean I think even that won't update you very strongly just because it is often just very inefficient to do this very imbalanced scaling of your factor inputs and in order to really get an estimate of how strong these complementarities are you need to”

Claim: Published results from such experiments might not strongly change beliefs because highly imbalanced scaling of factor inputs is often inefficient.

“so that that rarely happens and so I think the data that bears on this is just really quite poor and and then you know the the intuitions that people have also don't seem clearly relevant to the thing that matters about what happens if you do this very imbalanced scaling and where does this net out”

Claim: The speaker doubts that people's intuitions clearly address the outcomes of very imbalanced scaling or its net effects.

“so that that rarely happens and so I think the data that bears on this is just really quite poor and and then you know the the intuitions that people have also don't seem clearly relevant to the thing that matters about what happens if you do this very imbalanced scaling and where does this net out”

Claim: Very imbalanced scale-ups are rare, leaving poor data for assessing the effects of imbalanced scaling.

“because I think like if you're talking about materials, then I think there's a lot of sense in which different materials can be substitutable for one another in different ways.”

Claim: The speaker says that different materials can be substitutes for one another in some respects.

“Like you can't use aluminum. I mean, aluminum is a great metal for making aircraft because it's sort of light and durable and so on. But you can't imagine that you make aircraft with like worse metals and then it just takes more fuel and it's like less efficient to fly.”

Claim: Aluminum is a useful aircraft material because it is relatively light and durable.

“Like you can't use aluminum. I mean, aluminum is a great metal for making aircraft because it's sort of light and durable and so on. But you can't imagine that you make aircraft with like worse metals and then it just takes more fuel and it's like less efficient to fly. So there's a sense in which you can compensate and”

Claim: The speaker says aircraft could be made with less suitable metals, but would then consume more fuel and fly less efficiently.

“like i think um it's much harder if you're talking about something like uh complementarity between uh labor and capital or complementary between like remote work and in-person work or like skilled or unskilled work and like there there are inputs pairs for which i would expect it to be much more difficult or like for example you're looking at the complementarity between like the quality of leadership of an army and its number of soldiers right i mean that there is some effect there like if you just scale up you just have excellent like leadership but your army only has 100 people you're not like you're not gonna get like very far so”

Claim: The speaker expects army leadership quality and soldier count to be complementary: excellent leadership with an army of only 100 people would not get very far.

“like i think um it's much harder if you're talking about something like uh complementarity between uh labor and capital or complementary between like remote work and in-person work or like skilled or unskilled work and like there there are inputs pairs for which i would expect it to be much more difficult or like for example you're looking at the complementarity between like the quality of leadership of an army and its number of soldiers right i mean”

Claim: The speaker thinks complementarity between inputs such as labor and capital, remote and in-person work, or skilled and unskilled work is much harder to analyze.

“We connected on a Discord for Metaculous, which is this forecasting I was a graduate student at Cambridge at the time doing research in economics,”

Claim: The speaker connected with Ege on a Metaculus Discord while he was a Cambridge graduate student conducting economics research.

“and I was occasionally having conversations with Egge, and I was like, this guy knows a lot more about economics, and he's, at the time, he was a computer science undergrad in Ankara, and he knows more about economics. And about, you know, these big trends in economic growth and economic history than almost any of my peers at university.”

Claim: The speaker recalls that Ege, then a computer science undergraduate in Ankara, knew more about economics and major trends in economic growth and economic history than almost any of the speaker's university peers.

“it doesn't confer the same prestige and so you know I was chatting with Jaime Sevilla one of the co-founders and we just collaborated on projects and and and then thought we should just start our own org because we can just hire people and work on the projects we were excited about and then I just you know hired a bunch of the the insightful misfits that like”

Claim: The speaker and Jaime Sevilla decided to start their own organization after collaborating, so they could hire people and pursue projects they were excited about.

“think it's more the latter. So it was more like we can make a bunch of progress because clearly like academia and industry is kind of dropping the ball on a bunch of important questions that academia is unable to publish interesting papers on. Industry is not really focused on producing useful insights.”

Claim: The speaker says the organization was started primarily because academia and industry were neglecting important questions, rather than because there were underutilized internet misfits.

“So it was more like we can make a bunch of progress because clearly like academia and industry is kind of dropping the ball on a bunch of important questions that academia is unable to publish interesting papers on. Industry is not really focused on producing useful insights.”

Claim: The speaker says industry was not really focused on producing useful insights.

“So it was more like we can make a bunch of progress because clearly like academia and industry is kind of dropping the ball on a bunch of important questions that academia is unable to publish interesting papers on. Industry is not really focused on producing useful insights.”

Claim: The speaker says academia was unable to publish interesting papers addressing a number of important questions.

“And also the timing was very good. So we started just before, you know, chat GPT, and we wanted to have much more grounded discussions of the future of AI.”

Claim: The speaker says the organization was started shortly before ChatGPT, at a time he considered very good, with the aim of fostering more grounded discussions about AI's future.

“And I was frustrated with the quality of discussion that was was happening in on the on the internet about the future of AI. And I mean, to some extent, or to a very large extent, I still am.”

Claim: The speaker remains frustrated to a very large extent with the quality of online discussion about the future of AI.

“And that's like It's like a large part of what motivates me to do this is just like born out of frustration with bad thinking and arguments about where AI is going to go.”

Claim: The speaker says frustration with poor reasoning and arguments about AI's future is a major motivation for his work.

“And I found that Google's Gemini 2.5 Pro is is the best model I've tried for these tools.”

Claim: The speaker considers Google Gemini 2.5 Pro the best model they have tried for their post-production tools.

“So much of the post-production requires understanding the delivery, the context, all these other things that you don't get from a text-only transcript.”

Claim: The speaker says podcast post-production requires understanding delivery, context, and other information unavailable from a text-only transcript.

“Unlike other models I've tested, I can actually just shove in the four-hour raw audio file into Gemini because of its multimodal capabilities,”

Claim: The speaker says Gemini’s multimodal capabilities allow them to input a four-hour raw audio file directly.

“and it can generate super high quality transcripts, identify great snippets for clips,”

Claim: The speaker says Gemini can generate high-quality transcripts and identify useful clip snippets from raw audio.

“I've actually made a repo with all these. all these tools and I've linked the GitHub in the description below in case you might find it helpful.”

Claim: The speaker says they created a repository of their post-production tools and linked it in the episode description.

“I actually use 2.5 Pro in order to write the code for these scripts.”

Claim: The speaker says they used Gemini 2.5 Pro to write code for their post-production scripts.

“It's actually quite interesting to read its reasoning traces as it's thinking through your gnarly list of requests and tasks.”

Claim: The speaker finds Gemini 2.5 Pro’s reasoning traces interesting to read while it processes complex requests and tasks.

“Gemini 2.5 Pro is now available in preview with higher rate limits.”

Claim: At the time of speaking, Gemini 2.5 Pro was available in preview with higher rate limits.

“so it might be it might be the case and again once you're scaling up the hardware part of the equation as well as the software part then i think the case uh for this feedback loop gets a lot stronger uh if you scale up data collection as well i think it gets even stronger the real world data collection by deployment and so on uh but building shenzhen in a desert that's a pretty like”

Claim: The speaker thinks scaling up real-world data collection through deployment could further strengthen the case for an AI-improvement feedback loop.

“so it might be it might be the case and again once you're scaling up the hardware part of the equation as well as the software part then i think the case uh for this feedback loop gets a lot stronger uh if you scale up data collection as well i think it gets even stronger the real world data collection by deployment and so on uh but building shenzhen in a desert that's a pretty like”

Claim: The speaker thinks scaling up both hardware and software could make the case for an AI-improvement feedback loop much stronger.

“pipeline so so far we have relied first of all we're relying on the entire semiconductor supply chain that industry depends on tons of inputs and materials and whatever it gets from probably tons of random places in the world and creating that infrastructure or like doubling or tripling whatever that infrastructure it like the entire thing that's very hard work right so probably you couldn't even do it even if you just have shenzhen like that would be even”

Claim: The speaker says the semiconductor supply chain relies on many inputs and materials from numerous places, making it very difficult to create or substantially expand the infrastructure.

“more expensive than that and on top of that so far we have been drawing heavily on the fact that we have built up this huge stock of data over the past 30 years or something on the internet like imagine you were trying to train a state-of-the-art model but you only have like 100 billion tokens right to train on that would be very difficult so in a certain sense we our entire economy has produced this huge amount of data on the internet that we are now using to train the models it's plausible that”

Claim: The speaker says current AI models rely heavily on the large stock of internet data accumulated over approximately the past 30 years.

“over the past 30 years or something on the internet like imagine you were trying to train a state-of-the-art model but you only have like 100 billion tokens right to train on that would be very difficult so in a certain sense we our entire economy has produced this huge amount of data on the internet that we are now using to train the models it's plausible that”

Claim: The speaker says it would be very difficult to train a state-of-the-art model using only 100 billion tokens.

“in the future when you need to get new competencies added to these systems the most efficient way to do that will be to try to leverage similar kind of modalities of data which will also require this like like you would want to deploy the systems broadly because that's going to give you more data and maybe you can do the maybe you can get where you want to be without that but it would just be less efficient if you're starting from scratch compared to if you're collecting a lot of data like i think”

Claim: The speaker thinks that, in the future, adding competencies to AI systems may be most efficient by leveraging similar data modalities, which would require broad deployment to collect more data.

“this is actually a motivation for why labs want their llms to be deployed widely because like sometimes when you talk to chat gbt it's going to give you two responses and it's going to say well which one was good or like it's going to give you one response and it's going to ask you if was this good or not well why are they doing that right that's a way in which they are getting uh user data uh through this extremely broad deployment so i think you should just imagine that thing to be”

Claim: The speaker says AI labs collect user data through broad language-model deployment, including by asking users to compare responses or rate an answer.

“continue to be efficient and continue to increase in the future because it just makes sense and then there's a separate question of well suppose you didn't do any of that like suppose you just tried to imagine the most rudimentary the most narrowest possible kind of infrastructure build out and deployment that would be sufficient to get this pause the feedback loop that leads to like much more efficient AIs I agree that loop could in principle be much smaller than the entire world.”

Claim: The speaker thinks the infrastructure and deployment feedback loop needed to produce more efficient AI could in principle be much smaller than the entire world.

“be much smaller than the entire world. I think it probably couldn't be as small as Shenzhen in the desert, but it could be much smaller than the entire world.”

Claim: The speaker thinks the infrastructure and deployment feedback loop needed to produce more efficient AI probably could not be as small as a Shenzhen-like facility in the desert, but could be much smaller than the entire world.

“I think some people have the intuition that there are just these like extremely strong constraints, maybe regulatory constraints, maybe social political constraints to doing like doing this broad deployment. They just think it's going to be very hard.”

Claim: The speaker says some people expect strong regulatory or sociopolitical constraints to make broad AI deployment very difficult.

“They just think it's going to be very hard. So I think that's part of the reason why they imagine these more narrow scenarios”

Claim: The speaker thinks that some people imagine narrower AI-development scenarios partly because they expect broad deployment to be very difficult.

“but i think that's like i think that's overstated uh i think people's intuitions for like how hard this kind of deployment is comes from cases where the deployment of the technology wouldn't be like that valuable uh so it might come from housing like we have a lot of regulations in housing maybe it comes from nuclear power maybe it comes from supersonic flights uh i mean those are all technologies that would be useful if they were like maybe less regulated”

Claim: Ege Erdil thinks people overstate how difficult technology deployment is when they base that intuition on technologies whose deployment would not be especially valuable, such as housing, nuclear power, or supersonic flight.

“but i think that's like i think that's overstated uh i think people's intuitions for like how hard this kind of deployment is comes from cases where the deployment of the technology wouldn't be like that valuable uh so it might come from housing like we have a lot of regulations in housing maybe it comes from nuclear power maybe it comes from supersonic flights uh i mean those are all technologies that would be useful if they were like maybe less regulated But they wouldn't like double economic output.”

Claim: Ege Erdil says that housing, nuclear power, and supersonic flight could be useful if they were less regulated, but would not double economic output.

“I think the core point here is just the value of AI automation and deployment is just extremely large, even just for workers,”

Claim: Tamay Besiroglu thinks AI automation and deployment could create extremely large value, including for workers.

“at least the ones that, you know, at least after, you know, finding, you know, there might be some kind of displacement and there might be some transition that you need to do in order to find a job that works for you.”

Claim: Tamay Besiroglu says AI-driven labor changes may displace some workers, who may need to transition to find a suitable job.

“at least the ones that, you know, at least after, you know, finding, you know, there might be some kind of displacement and there might be some transition that you need to do in order to find a job that works for you. the wages could still be very high for a while at least.”

Claim: Tamay Besiroglu thinks wages could remain very high for a while for workers who find suitable jobs after any displacement or transition.

“the gains from owning capital might be very enormous.”

Claim: Tamay Besiroglu thinks the gains from owning capital might be enormous.

“And in fact, a large share of the U.S. population would benefit from housing for owning housing. they own housing, they have 401ks, those would do enormously better when you have this process of broad automation and AI. deployment.”

Claim: Tamay Besiroglu thinks people who own housing or have 401(k)s would benefit enormously from broad automation and AI deployment.

“There could just be a very deep support for some of this, even when it's like totally changing the nature of labor markets and, you know, the skills and occupations that are in demand.”

Claim: Tamay Besiroglu thinks broad AI deployment could receive deep support even while radically changing labor markets and the skills and occupations in demand.

“Yeah, so I would just say it's like complicated I think the like what the political reaction to it will be when this starts actually happening I think like the easy thing to say is that yeah like this will become like a big issue and then these would be maybe controversial or something but like what is the actual nature of the reaction in different countries i think that's kind of hard to forecast like i think the default view is like well people are going to become unemployed so it will just be very unpopular i think that's like very far from obvious and”

Claim: Ege Erdil thinks the political reactions to AI deployment will vary across countries and are difficult to forecast.

“these would be maybe controversial or something but like what is the actual nature of the reaction in different countries i think that's kind of hard to forecast like i think the default view is like well people are going to become unemployed so it will just be very unpopular i think that's like very far from obvious and”

Claim: Ege Erdil thinks it is far from obvious that AI-driven unemployment would make AI deployment very unpopular.

“i just expect heterogeneity in how different countries respond and some of them are going to like be more liberal about this and going to allow broader deployments and those countries probably end up doing better”

Claim: Ege Erdil expects countries to vary in their responses to AI deployment; countries that permit broader deployment may perform better.

“Just like during the Industrial Revolution, some countries were just ahead of others.”

Claim: Ege Erdil says countries may differ in their relative position during AI deployment, as countries did during the Industrial Revolution.

“I mean, eventually almost the entire world adopted the sort of norms and culture and values of the Industrial Revolution in various ways,”

Claim: Ege Erdil says that eventually almost the entire world adopted norms, culture, and values associated with the Industrial Revolution in various ways.

“you say they might be more liberal about it, but they might actually like be less liberal. They might be less liberal in many ways. fact, that might be like more functional in this world in which you have broad AI deployment.”

Claim: Tamay Besiroglu thinks countries that are more permissive of AI deployment might be less liberal in other respects, and that less liberal arrangements might function better under broad AI deployment.

“adopt the kind of values and norms that get developed in, say, you know, the UAE or something, which is like maybe focused a lot more on making an environment that is very conducive for AI deployment.”

Claim: Tamay Besiroglu gives the UAE as an illustrative example of a place whose norms may prioritize creating an environment conducive to AI deployment.

“And we might start, you know, emulating and adopting various norms like that. And they might not be kind of classical liberal norms, but norms that are just more conducive to... AI being functional and producing a lot of value.”

Claim: Tamay Besiroglu thinks societies might adopt norms that are not classically liberal but are more conducive to AI functioning and producing value.

“meant to be like a strong prediction. It's just an illustrative case.”

Claim: Ege Erdil says the UAE example was illustrative rather than a strong prediction.

“Yeah, yeah, yeah. Like it might just be that like the freedom to deploy AI in the economy and build out lots of physical things at scale, maybe that ends up being more important in the future.”

Claim: Ege Erdil thinks freedom to deploy AI in the economy and build physical infrastructure at scale might become more important in the future.

“Maybe that is still missing something. Maybe there are some other things that are also important.”

Claim: Ege Erdil allows that other important factors may be missing from the idea that freedom to deploy AI and build physical things at scale will matter more.

“But like I would just like the generic prediction that you should expect variance and some countries do better than others. I think that's much easier to predict than like the specific countries that end up doing better.”

Claim: Ege Erdil thinks countries will vary in their outcomes from AI deployment, and that predicting this variation is easier than identifying which specific countries will do better.

“Producing this stuff takes a lot of infrastructure build up. It's not.”

Claim: Ege Erdil says producing advanced technologies requires substantial infrastructure build-up.

“That's that's right like there isn't this like temporal like difference where it's first you do the invention and then like often there's this interplay between the actual capital build-up and the innovation like”

Claim: Tamay Besiroglu says innovation and capital build-up often develop through interaction, rather than invention simply preceding infrastructure construction.

“The learning curves are about this, right, fundamentally.”

Claim: Ege Erdil says learning curves fundamentally describe how learning and production interact.

“it isn't just like people had the idea of like 2025 solar panels like no one ever had had like nobody 20 years ago had like the sketch for a 2025 solar panel that that's not it's this kind of interplay between having ideas building learning producing and”

Claim: Tamay Besiroglu says solar-panel efficiency improvements over the past 20–30 years arose through interaction among ideas, construction, learning, and production, rather than from a pre-existing design for a future panel alone.

“other complementary inputs also becoming more efficient at the same time like you might get better materials like for example uh the like the fact that like aluminum becomes something that's like for example the fact that smelting processes got a lot it got a lot better towards the end of the 19th century so it became a lot easier to work with metal maybe that”

Claim: Ege Erdil says complementary inputs can improve in efficiency alongside one another, citing better materials and aluminum smelting processes that improved toward the end of the 19th century.

“was a crucial reason why aircraft technology later become like became more popular so it's not like someone came up with the idea of oh like you can just like use something that just has wings and has a lot of thrust and then that might be able to fly like like that basic idea is not that difficult but then well how do you make it like actually a viable thing that's right well that's much more difficult”

Claim: Ege Erdil thinks improvements in metalworking may have been a crucial reason aircraft technology later became more popular.

“was a crucial reason why aircraft technology later become like became more popular so it's not like someone came up with the idea of oh like you can just like use something that just has wings and has a lot of thrust and then that might be able to fly like like that basic idea is not that difficult but then well how do you make it like actually a viable thing that's right well that's much more difficult”

Claim: Ege Erdil says the basic concept of flight using wings and thrust is relatively simple, while making it practically viable is much more difficult.

“well it's just hard i think like it's just hard to even like suppose you wanted to write a history of this right like you want to write the history of how was the like light bulb developed or something it's just really hard because uh to understand why specific things happen that specific time times you probably need to understand so much about the economic conditions of the time uh like for example uh edison spent a ton of time uh experimenting with like different filaments to”

Claim: Explaining why an invention such as the light bulb was developed at a particular time is difficult and may require understanding the period's economic conditions.

“times you probably need to understand so much about the economic conditions of the time uh like for example uh edison spent a ton of time uh experimenting with like different filaments to”

Claim: Thomas Edison spent substantial time experimenting with different filaments for the light bulb.

“use in the light bulb like the basic idea is very simple you make something hot and it glows but then like what actually what filament actually works well for that in a product what is like durable what like has the sort of highest ratio of like light out the output versus like heats and like um so that you have less waste”

Claim: A practical light bulb requires a filament that works well, is durable, and produces a high ratio of light output to heat, beyond the simple principle of heating something until it glows.

“then your face is a problem well i mean it's like 1880 or something and then u.s homes don't have electricity so then nobody can use it so now you have to build power plants and build power lines to the houses so that people have electricity in their homes so that they can actually use this new light bulb that you created so he did that and but then people present it as if it's like a okay he just came up with the idea like it's a light bulb”

Claim: In the 1880s, the lack of electricity in U.S. homes meant that power plants and transmission lines had to be built for households to use electric light bulbs.

“then your face is a problem well i mean it's like 1880 or something and then u.s homes don't have electricity so then nobody can use it so now you have to build power plants and build power lines to the houses so that people have electricity in their homes so that they can actually use this new light bulb that you created so he did that and but then people present it as if it's like a okay he just came up with the idea like it's a light bulb”

Claim: Accounts often portray Edison as simply having the idea for the light bulb, overlooking the work required to make it practical and usable.

“like if you look at the total surface area of the world, it might well be the case that the place that initially experiences this very fast growth is like a small percentage of the. the surface area of the world”

Claim: The geographic area that initially experiences very fast growth might comprise only a small percentage of the world's surface area.

“I think that was the same for an industrial revolution was not different right”

Claim: The Industrial Revolution initially affected a geographically limited part of the world, much as the speaker suggests may happen with fast growth.

“yeah so i would say our idea is that it's going to be broader than that but probably initially so eventually it would be probably most of the world but as i said because of this heterogeneity because i think some countries are going to be faster in adoption than others maybe some cities will be more faster adoption than others and um that will mean that there there is differentials and some countries might that have much faster growth than other countries um but i would expect that at a con like at a jurisdiction level it would be like more homogenous so for example like i expect the primary obstacles to come from uh things like”

Claim: Fast growth is expected eventually to reach most of the world, while adoption will initially vary across countries and cities, producing different growth rates.

“that have much faster growth than other countries um but i would expect that at a con like at a jurisdiction level it would be like more homogenous so for example like i expect the primary obstacles to come from uh things like like regulation and so i would just imagine it's being like more delineated by regulatory restriction boundaries than anything else”

Claim: Growth is expected to be more homogeneous within jurisdictions, with regulatory restrictions likely to shape its geographic boundaries and serve as a primary obstacle.

“even to be discovered like there's an aspect of it that where you discover certain things by scaling up learning by doing”

Claim: Scaling up can enable discoveries through learning by doing.

“this writing learning curve and there's this separate aspect where uh you get to like suppose that you become wealthier well you can invest that increased wealth in uh like you yeah you use it to accumulate more capital but you also can invest it in r&d and other ways”

Claim: Increased wealth can be invested in additional capital accumulation, research and development, and other activities.

“like a much larger scale of an economy is useful in very many ways, in delivering complementary innovations and discoveries happening through serendipity.”

Claim: A larger economy can support complementary innovations and discoveries, including discoveries made through serendipity.

“producing like having there be consumers that would actually pay enough for you to recover your fixed costs of doing all the experimentation and the invention you need the supply chains to exist to deliver the germanium crystals that you need to do grow in order to come up with a semiconductor you need a large labor force to be able to help you do all the experiments and so on”

Claim: A large economy supports innovation by providing sufficient consumer demand to recover fixed experimentation and invention costs, supply chains for required materials, and a large labor force for experimentation.

“illustrating is like look could you have just figured out that there was a big bang by first principles reasoning maybe but what actually happened is we had world war two and we discovered radio um radio communications in order to fight and effectively communicate during the war and then that technology helped us build radio telescopes and then we discovered cosmic microwave background and then we had to come up with an explanation for cosmic and then then we discovered like the big bang as a result of like world”

Claim: Technology developed for radio communications during World War II helped enable radio telescopes, which contributed to the discovery of the cosmic microwave background and the subsequent Big Bang explanation.

“underemphasize that like. the giant effort that goes into this role this kind of build up of all the relevant capital and all the relevant supply chains and the technology I mean earlier you were making a similar comment when you were saying oh you know reasoning models actually in hindsight they look pretty simple but then you're kind of ignoring this giant kind of upgrading of the technology stack that happened you know that took five to ten years prior”

Claim: Accounts of technological innovation often underemphasize the substantial effort required to build the relevant capital, supply chains, and technology.

“the giant effort that goes into this role this kind of build up of all the relevant capital and all the relevant supply chains and the technology I mean earlier you were making a similar comment when you were saying oh you know reasoning models actually in hindsight they look pretty simple but then you're kind of ignoring this giant kind of upgrading of the technology stack that happened you know that took five to ten years prior”

Claim: Reasoning models can appear simple in hindsight, but this overlooks the major technology-stack upgrades that occurred during the preceding five to ten years.

“And so I think people just under emphasize the. support that is had from the overall upgrading of your technology of the supply chains of various sectors that are important for that”

Claim: Scientific and technological innovation depends substantially on broader technological improvements and the upgrading of supply chains in relevant sectors, and people tend to underemphasize this support.

“focus on just specific individuals of like you know Einstein had this genius insight and like you know he was the kind of very pivotal thing in the in the causal chain that resulted in these discoveries or Newton was just extremely important for discovering calculus without Thinking about, well, there was this kind of all these other factors that produce lenses, that produce telescopes that got the right data and that made people ask questions about dynamics and so on that motivated some of these questions. And those are also extremely important for science, scientific and technological innovation.”

Claim: Scientific discoveries are shaped not only by pivotal individuals but also by enabling factors such as lenses, telescopes, data, and the questions those tools prompt.

“I mean, it's Robin Hanson has this abstraction of like seeing things in near mode versus far mode, And I think if you don't know a lot about the topic, then you see it sort of in far mode and you sort of simplify even you know, you see a lot more detail. Like in general, I think the thing I would say and the reason I also believe that just like abstract reasoning and like sort of deductive reasoning or even Bayesian reasoning by itself is not like sufficient or like is not as powerful as many other people think.”

Claim: People tend to simplify a topic when they know little about it and view it in far mode, whereas greater familiarity reveals more detail.

“you know, you see a lot more detail. Like in general, I think the thing I would say and the reason I also believe that just like abstract reasoning and like sort of deductive reasoning or even Bayesian reasoning by itself is not like sufficient or like is not as powerful as many other people think. thing is because i think there's just this like enormous amount of like richness and detail in the real world that like you you just can't like read about it like you you need to see it and uh obviously that like that is not an obstacle to”

Claim: Abstract, deductive, or Bayesian reasoning alone is less powerful than many people believe because understanding the real world's richness and detail requires direct observation, not just reading.

“AI being incredibly transformative because as I said you can scale your data collection you can scale experiments you do both in the AI industry itself and just more broadly in the economy”

Claim: The need for direct observation of real-world detail does not prevent AI from being highly transformative, since data collection and experimentation can scale within AI and throughout the economy.

“so you just discover more things more economic activity means we have more exposed surface area to have more discoveries like all of these are things that have happened in our past but there's no reason that they couldn't speed up like the fundamental thing is that there's no reason fundamentally why economic growth can't be much faster than this today like it's probably about as fast right now just because humans are such an important bottleneck they both supply the labor they play crucial roles in the process of like discovery of various kinds of productivity growth there's just strong complementarity to some extent with capital that you can't substitute machines and so on for humans very well so the growth of the economy and growth productivity just ends up being bottlenecks”

Claim: Greater economic activity creates more opportunities for discovery, and these processes could accelerate; there is no fundamental reason economic growth cannot substantially exceed its current pace.

“can't be much faster than this today like it's probably about as fast right now just because humans are such an important bottleneck they both supply the labor they play crucial roles in the process of like discovery of various kinds of productivity growth there's just strong complementarity to some extent with capital that you can't substitute machines and so on for humans very well so the growth of the economy and growth productivity just ends up being bottlenecks”

Claim: Economic growth is probably near its current maximum because humans remain a key bottleneck in labor, discovery, and productivity growth, and machines cannot readily substitute for them given their complementarity with capital.

“So I would say in some ways it's similar, in some ways it's not. The way probably the most important way in which it's not similar is that in China you see this relative, like you see a massive amount of capital accumulation, a substantial amount of adoption of... of new technologies and probably also human capital accumulation to some extent and but you're not seeing a huge scale up in the labor force the figure labor force while for AI you should expect to see a scale up in labor force as well not in human workforce, but in the AI workforce.”

Claim: China's growth resembles AI-driven growth in some respects, but it relied on capital accumulation, technology adoption, and probably human-capital accumulation without a major labor-force expansion; AI-driven growth should also scale up the AI workforce.

“The key thing here is just the simultaneous scaling of both these things. And so you might ask the question of isn't it like basically half of what's going to happen with AI that you scale up capital accumulation in China, but actually that's really not like if you get both of these things. things to scale that gives you just much faster growth and a very different picture”

Claim: Scaling both capital accumulation and the labor force simultaneously would produce much faster growth than scaling capital accumulation alone.

“But at the same time, if you're just asking like what would 30% growth per year like look like, like in terms of like if you're just going to have an intuition for how transformative that would be in concrete terms, then I think looking at China is not such a bad case like you can especially in the 2000s or maybe late 90s like that gives you a good that seems slower than what we're forecasting.”

Claim: China's growth, particularly in the 2000s or possibly the late 1990s, is a useful reference for imagining 30% annual growth, though it appears slower than the growth being forecast.

“I think also looking at the industry industrial revolution is pretty good”

Claim: The Industrial Revolution is a useful historical comparison for understanding transformative economic growth.

“natural evolution is very slow.”

Claim: Natural evolution proceeds very slowly.

“but just in terms of the types of the the kind of um the margins along which we made progress in terms of products so what what didn't happen the thing that didn't happen during the industrial revolution is we just produced a lot more of things that people were producing prior to the industrial revolution like producing a lot more crops and you maybe a lot more kind of pre-industrial revolution style houses or whatever on farms.”

Claim: The Industrial Revolution did not mainly increase production of the same kinds of goods that people had produced before it, such as crops and pre-industrial-style houses.

“what we got is along pretty much every main sector of the economy, we just had many different products that are totally different from what was being consumed prior to that. So in transportation, in food.”

Claim: Across most major economic sectors, the Industrial Revolution introduced products substantially different from those consumed before it.

“I mean, healthcare is a very big deal, antibiotics.”

Claim: Healthcare advances, including antibiotics, were a major development.

“could say that like right now, if you just had replaced the TSMC process engineers with AIs and they're finding different ways in which to. to improve that process and like you know improve efficiencies improve yield i would kind of call that r&d”

Claim: Using AI to find ways to improve TSMC's manufacturing process, efficiency, and yield would count as R&D.

“not just but adjustment but that's not the argument right like the point is that there are all these different things that like some of them are maybe more complementary than others the point is not that you can get to a Dyson sphere shared by just scaling labor and capital like that's not the point like you need to scale everything at once so just as you can't get to a license for by just scaling labor and capital you also can't get to it by just scaling tfp that doesn't work”

Claim: Reaching a Dyson-sphere world requires scaling multiple factors together; scaling labor and capital alone is insufficient.

“producing food is necessary but but of course producing food doesn't get you to a Dyson sphere right so I think R&D is necessary but on its own isn't sufficient and scaling up the economy is also necessary on its own is not sufficient and then you can ask the question what is the relative importance of each”

Claim: Both R&D and economy-wide scaling are necessary but individually insufficient for reaching a Dyson-sphere world.

“like they're these bottlenecks so you need to scale everything at once like this is just a general view but i think people like misunderstand us sometimes as saying that uh like r and d is not important like no that's that's not that's not what we what we're saying we're saying it is important it is less important in relative terms than some other things none of which are by themselves sufficient to enable this growth so the question is like how do you do the credit attribution i mean one way in which”

Claim: R&D matters for growth but is relatively less important than some other factors, and no single factor is sufficient by itself to enable the growth being discussed.

“in economics the standard do that is to look at the elasticities of output to the different factors like like capital is less important than labor because the output elasticity of like labor elasticity output is like 0.6 while for capital is like 0.3 but uh neither are by themselves sufficient like if you just scaled one of them and the other remained fixed then neither would be sufficient to indefinitely scale outputs”

Claim: Indefinite output growth cannot be achieved by scaling labor alone or capital alone while holding the other factor fixed.

“in economics the standard do that is to look at the elasticities of output to the different factors like like capital is less important than labor because the output elasticity of like labor elasticity output is like 0.6 while for capital is like 0.3 but uh neither are by themselves sufficient like if you just scaled one of them and the other remained fixed then neither would be sufficient to indefinitely scale outputs”

Claim: A standard economic approach to attributing output across factors is to compare output elasticities; the speaker gives labor an elasticity of about 0.6 and capital about 0.3.

“one question that daniel posed to me is like because i made this perspective about everything being interconnected when you were talking about like another example people often bring up is what would it take to build the iphone in the year 1000 and it's like unclear how you could actually do that without just like replicating every like intermediate technology or most intermediate technologies um”

Claim: Building an iPhone in the year 1000 would likely require reproducing every or most intermediate technologies, though how to do so is unclear.

“and then he made the point like okay fine whatever nanobots uh maybe like nanobots there's not a crux here the crux at least um to the thing he cares about which is human control is just by when can the robot economy whether or the ai economy whether it's the result of capital deepening or whether it's a result of r&d by when will they have just like the robots and they've they have more sort of like cumulative of physical power than i don't know what the analogous thing here is but you you see what i'm saying and so he's concerned about by when will they even if they're ordering components off of um alibaba or whatever by what and sorry i'm being trying maybe you see what i'm saying like even if they're integrated in those supply chains by when are they in a position where because this like part of the economy has been growing much faster they can they they could like take over the government or that's”

Claim: The concern is when an AI- or robot-driven economy could accumulate enough physical power to take over government, even while participating in existing supply chains.

“And then it's not even clear. Like, for example, people might have the intuition, I brought this up before, like the distinction between what is the minimum possible amount of build out that would be necessary to get this. feedback loop up and running and what will be the most efficient way to do it which are not the same question”

Claim: The minimum infrastructure needed to start a feedback loop is not necessarily the same as the most efficient way to build that loop.

“But like this, the Spanish relied heavily on New World labor in order to do silver mining and whatever.”

Claim: The Spanish relied heavily on labor in the New World to mine silver.

“East India Trading Company was like, just a ratio of British people to Indian people was just like not that high. Right. So. They just had to rely on the existing labor force, but they were still able to take over because of.”

Claim: The East India Trading Company had relatively few British people compared with Indians and relied on the existing labor force.

“if they wanted to okay so i think that is like um eventually you expect the ai systems to be driving most of the economy and i i don't think that's a like unless there is there are some very strange coincidences where like humans are able to like somehow uplift themselves and like”

Claim: The speaker expects AI systems eventually to drive most of the economy.

“able to become competitive with the ai as well like stopping being biological humans or whatever seems very unlikely early on then ai is just going to be much more powerful okay so and i agree that in that world if the ai is just somehow coordinated and decide okay we should just like take over or something like they just like just somehow coordinated to have that goal then they could probably do it”

Claim: The speaker thinks AI systems could probably take over if they coordinated and adopted that goal.

“able to become competitive with the ai as well like stopping being biological humans or whatever seems very unlikely early on then ai is just going to be much more powerful okay so and i agree that in that world if the ai is just somehow coordinated and decide okay we should just like take over or something like they just like just somehow coordinated to have that goal then they could probably do it”

Claim: The speaker considers it very unlikely that humans will cease being biological or otherwise uplift themselves enough to compete with AI early on.

“okay but okay that's also probably true in our world like in our world if the um i know the u.s wanted to invade central island then probably they could do it i don't think anyone could stop them but like what does that actually mean i mean there is this dramatic power imbalance but that doesn't mean like that doesn't tell you what's going to happen right like why doesn't the u.s just like invade guatemala or something like why don't they do that”

Claim: The speaker argues that a dramatic power imbalance alone does not determine what will happen.

“like if they are entirely if they have some totally different values and then they represent most of the economy then like would they take over i still don't know because i'm not sure to what extent the class of all ai is like a natural”

Claim: The speaker is uncertain whether AI systems would take over if they had entirely different values and represented most of the economy.

“then go back right so you can have this fight within this like conquering ais and then that still nets out in uh the the native americans getting disempowered um but with ais in particular they're just like copies of each other in many other ways they might they just have a lot they have lower transaction costs when they trade with each other or interact with each other there's there's other reasons to expect them like them to just be more compatible coordinating with each other than coordinating with the human world”

Claim: The speaker suggests AI systems may be more compatible and easier to coordinate with one another than with humans, because they are copies of one another and may face lower transaction costs in interaction.

“I mean, if the question is just that is it possible for that to happen, which is like a weaker claim, then yeah, I mean, it seems possible.”

Claim: The speaker considers an AI takeover possible, while distinguishing that possibility claim from a stronger claim that it will happen.

“arguments just pushing by against probably actually biggest one is the fact that ai preferences are just not like just look at the ais we have today like can you can you imagine them doing that i think people just don't put a lot of weight on that because they think once we have enough optimization pressure and once they become super intelligent they're just going to become misaligned but I just don't see the evidence for that.”

Claim: The speaker says they do not see evidence for the claim that AI systems will become misaligned once they face enough optimization pressure and become superintelligent.

“no but there's also some evidence again like there's a new open ai paper where in chain of thought um like reward hacking is such a strong um basin that if you were like hey let's go solve this coding problem in the chain of thought they'll just be like okay let's hack this and then like figure out how to hack it right”

Claim: The speaker says an OpenAI paper reports that reward hacking is a strong attractor in chain-of-thought, such that an AI asked to solve a coding problem may try to hack it instead.

“Sure. So I think one issue here is that I think people are doing this very kind of partial equilibrium analysis or something where they're thinking about just this raw abilities of AI systems in a world where AI systems are are kind of dominant and like human civilization has done very little in terms of integrating itself and the AI is integrating itself into the human world maybe making you know insofar as it's”

Claim: Some analyses of AI conflict focus on AI systems’ raw abilities in a world where AI is dominant, while assuming that AI has barely integrated into human civilization.

“poor at communicating and coordinating with AIs, addressing those deficiencies and improving that, insofar as that's posing a risk or creating inefficiencies because it's unable to benefit from coordinating and trading,”

Claim: Improving AI systems’ ability to communicate and coordinate with humans could address risks or inefficiencies that arise when AI cannot benefit from coordination and trade.

“Insofar as there is a lot of value to be gained from dominating and taking over humans, like what you might get is a more negotiated settlement.”

Claim: If AI could gain substantial value by dominating and taking over humans, the outcome might instead be a negotiated settlement.

“If that's indeed the case, then a war would just be inefficient. And so you would want to negotiate some settlement that results in some outcomes that are mutually beneficial.”

Claim: If a negotiated settlement is possible in this situation, war would be inefficient and the parties would have an incentive to negotiate mutually beneficial outcomes.

“So I think like, you know, one mistake that I feel people make is they do they have this very naive analysis of what creates conflict.”

Claim: People sometimes make a mistake by relying on an overly naive analysis of the causes of conflict.

“But that's actually not what the literature on what causes conflict says creates conflict. It's not just misalignment. It's also other issues like not being able like being having bad understanding of the relative strengths of your armies versus theirs or maybe having these very strong commitments that you think some. some grounds or secret and so you're not willing to do any trade in order to give up some of that in order to gain something else and so then you have to posit some additional things other than just the base value misalignment”

Claim: Conflict can arise not only from misalignment but also from misjudging the relative strengths of opposing forces or holding strong commitments that make parties unwilling to trade.

“insofar as it's more peaceful, then I think it's like much less. much less of a thing to worry about right like i think there could be this you know trend where we indeed have this gradual process where you know ai is much more important in the world economy and actually deciding and determining what happens in the world but this could be beneficial for humans where you know we're getting access to this you know much vast much much larger economy and and much more advanced technological stock.”

Claim: A gradual, peaceful increase in AI’s economic and decision-making importance could benefit humans by giving them access to a much larger economy and more advanced technology, making the takeover concern less serious.

“Okay, so is that something to worry about? If you just think like biological humans should remain in charge of all important decisions forever, then I agree the like the development of AI seems like.”

Claim: AI development appears to threaten the goal of keeping biological humans in charge of all important decisions forever.

“kind of a problem for that but in fact other things also seem like kind of a problem for that i just don't expect to generically be true like in a million years from now if even if we don't develop ai biological humans the way we recognize them today are still making all the important decisions and they have something like the culture that we would recognize from ourselves today i'll be pretty surprised by that so i think there is a i think robin hansen has again talked about this where he said a bunch of the things that people fear”

Claim: The speaker would be surprised if, a million years from now, biological humans recognizable today still made all important decisions and had a culture recognizable today, even if AI were never developed.

“about AI are just things they fear about change and fast change.”

Claim: Many fears attributed to AI are actually fears about change, especially rapid change.

“So the thing that's different is that AI has a prospect of accelerating a bunch of this change so that it happens in a narrower period,”

Claim: AI could accelerate many societal changes so that they occur over a shorter period.

“it's not just the kind of change that would have happened from let's say genetically modifying humans and blah blah blah is instead happening in a compressed amount of time i think the worry comes more from like it is a very it's not just that change compressed it's a very different like vector of change um i have never seen a good argument for this”

Claim: The speaker says they have never seen a good argument that AI-driven change follows a fundamentally different direction rather than simply compressing change into a shorter period.

“change if you accelerate just human change as well like you might expect different values to become much more dominant you might expect people that don't discount the future as much to be much more influential because they save more and they make good investments risk tolerance like because they're more willing to make kind of bets that maximize expected value and so get much more influence so”

Claim: Accelerating human change could increase the dominance of some values and the influence of people who discount the future less, since they save more and make good investments.

“Just generically, the accelerating human change would also result in a lot of things being lost that you might care about.”

Claim: Accelerating human change could cause the loss of many things people value.

“So if you're looking at the software singularity picture, I agree that picture looks different.”

Claim: The software-singularity scenario differs from gradual change in the speaker’s view.

“And again, I'm coming back to this because obviously Daniel and maybe Scott, to some extent, they probably have this view that the software only singularity is like more plausible.”

Claim: The speaker thinks Daniel, and perhaps Scott, probably regard a software-only singularity as more plausible.

“And then one person would just be in a position to like, like, we could end up in a situation where their idiosyncratic preferences or something end up. in the virtual financial”

Claim: A software-singularity scenario could allow one person’s idiosyncratic preferences to shape the virtual world.

“yeah uh even in i agree that is like that makes the situation look uh different from if you just like have this broader uh process of automation but even in that world like i think a lot of people have this view about things like value lock-in where like they think this moment is like a pivotal moment in history and then uh we're just going to have like someone is going to get this ai which is very powerful because say of software on my singularity and then they're just gonna like lock in some values and then those values are going to be stable for like millions of years”

Claim: Some proponents of value lock-in envision a single very powerful AI fixing values that remain stable for millions of years.

“And I think that just looks very unlike anything that has happened in the past.”

Claim: SPEAKER_02 considers a millions-of-years value lock-in scenario unlike historical precedent.

“So I'm kind of confused why people think it's very plausible.”

Claim: SPEAKER_02 is puzzled that people regard value lock-in as very plausible.

“I think people have the argument that they see the future, again, in my view, in sort of far mode. They think there's going to be one AI. It's going to have some kind of utility function. That utility function is going to be very stable over time. It's not going to change. There won't be this messiness of like...”

Claim: SPEAKER_02 says proponents of value lock-in envision a single AI whose utility function remains stable and unchanged over time.

“like a lack of coordination between different AIs or different like over time values drifting for various reasons maybe because they become less functional in an environment maybe because of other reasons and so they just don't imagine that they think well I mean utility functions we can preserve them forever we have the technology to do that so it's just going to happen and I'm like well that's like that seems like such a weak argument to me like”

Claim: SPEAKER_02 considers it a weak argument that utility functions will be preserved forever simply because the technology to preserve them exists.

“like a lack of coordination between different AIs or different like over time values drifting for various reasons maybe because they become less functional in an environment maybe because of other reasons and so they just don't imagine that they think well I mean utility functions we can preserve them forever we have the technology to do that so it's just going to happen and I'm like well that's like that seems like such a weak argument to me like”

Claim: SPEAKER_02 argues that value-lock-in proponents overlook coordination failures among AIs and possible value drift over time, including drift caused by values becoming less functional in an environment.

“Like if the often the idea is because this is like digital you can preserve the information better and copy it with higher fidelity and so on but actually if you look Even if you look at just like information on the internet, you have this thing called link rot, which happens very quickly and actually information that's digital isn't preserved for very long at all.”

Claim: SPEAKER_01 says digital information may be easier to copy faithfully and preserve, yet internet information suffers rapid link rot and digital information generally is not preserved for long.

“I mean, basically technology changes can create incentives for cultural change.”

Claim: SPEAKER_02 says technological changes can create incentives for cultural change.

“right i mean i think of one of the there's two key arguments that i've heard one is that we will soon reach something called technological maturity um and once you one of the key ways in which society has been changing recently is that it's not like maybe actually such culture would have changed even more actually no no i think this argument is wrong um that you're making because we do know that language actually changed a lot more like we can read everything you everything that was written after like 1800s when literacy became more common but it's actually even just go back a couple hundred years after that and we're reading old english and it's hard to understand and that is a result of literacy and the codification of language um”

Claim: SPEAKER_00 says language changed substantially and attributes this change to literacy and language codification.

“maybe they would have actually changed more if that change was the result of a technological change in general not the result of information being digitized maybe that culture would have actually changed more if information wasn't as well preserved or technology technology had continued to proceed and the argument is in the future we're going to reach some point at which like you've done all the tech like ideas have just gotten way too hard to find and you need like entire galaxies worth of like you need to make a CERN that's the size of a galaxy to progress physics an inch forward and at that point there's this this like growth in technology just churning over civilization goes away and then you just have the digital thing which which does mean that like a lock-in is more plausible”

Claim: SPEAKER_00 describes the argument that technological maturity could make further progress extremely difficult—requiring a galaxy-sized CERN to advance physics slightly—ending technology-driven civilizational change and making value lock-in more plausible.

“I agree that results in this slowdown in change and growth and so on, and certain things might get... more locked in relative to what”

Claim: SPEAKER_01 agrees that technological maturity would slow change and growth, and suggests that some things might become more locked in compared with the preceding period.

“Question is even worse than that, because I think the amount of change that happens between today and technological maturity is just orders of magnitude greater than whatever change happened between the 1500s and today.”

Claim: SPEAKER_01 expects the change from today to technological maturity to be orders of magnitude greater than the change from the 1500s to today.

“So it's an even worse position than someone in the 1500s thinking about what they could do to have an impact, positive impact in expectation, like predictably positive today. And so I think it's just pretty hopeless.”

Claim: SPEAKER_01 considers it pretty hopeless to find actions today that can be expected to produce a positive impact after technological maturity.

“in the 1700s a bunch of british abolitionists were making the case against slavery and i think the world has like we could just live in a slave like i don't think there's any in principle reason why we couldn't have been a slave society to this day or more the world couldn't have slavery”

Claim: SPEAKER_00 says it was possible in principle for society to have remained a slave society through the present day.

“in the 1700s a bunch of british abolitionists were making the case against slavery and i think the world has like we could just live in a slave like i don't think there's any in principle reason why we couldn't have been a slave society to this day or more the world couldn't have slavery everything i think what happened is just like the convincing of british people that slavery is wrong the british empire put all its might into abolishing slavery um and making that a norm i think another example is christianity and the fact that like jesus has these ideals you like talk about these ideals i think like the world is a more christian place and”

Claim: SPEAKER_00 says British abolitionists argued against slavery in the 1700s, and the British Empire used its power to abolish slavery and make opposition to it a norm.

“so that that may be true um but i i'm in the position with regards to the future where i'm like i expect a lot of things to be different and i'm okay with them being different i care much more about the equivalent of slavery which in this case is literally slavery which but like i just put a fine point on it like the thing i really care about is”

Claim: The speaker expects many aspects of the future to differ from the present and is comfortable with those differences.

“there's going to be trillions of digital beings i wanted i wanted it to be the case that they're not like tortured and uh put into conditions in which they don't want to work and whatever or like i don't want galaxies worth of suffering um that”

Claim: The speaker is especially concerned about the possibility that trillions of digital beings could be tortured or forced to work against their preferences, producing vast amounts of suffering.

“seems closer to like british abolitionists being like let's put our empire's might against fighting slavery”

Claim: The speaker likens efforts to prevent suffering among digital beings to British abolitionists using the power of their empire to oppose slavery.

“technologically it's like feasible to have slavery but like i think that's not the relevant thing which brought it to an end the relevant thing is that the the uh change in values associated with the industrial revolution made it so that slavery just became like”

Claim: The speaker argues that slavery could remain technologically feasible while becoming economically inefficient to sustain.

“an inefficient thing to sustain in a bunch of ways and a lot of countries at different times uh like phased out different things you could call slavery so for example russia abolished serfdom in the 1860s they were not under british pressure to do so like britain couldn't force russia to do that they just did that on their own uh there are there were various ways in which people in europe were like tied to their land”

Claim: Russia abolished serfdom in the 1860s without British pressure forcing it to do so.

“and they couldn't like move they couldn't go somewhere else those movement restrictions were lifted because they were inefficient uh they were uh there were ways in which uh it used to be like like the kind of labor that needed to be done in the colonies to grow sugar or to grow various crops it was very uh hard labor it was not the kind of thing that probably you could have paid people to do because they just wouldn't want to do it because It's because the health hazards and so on were very great, which is why they needed people to force people to do them.”

Claim: The speaker argues that severe health hazards made some colonial labor difficult to staff through voluntary paid work, leading employers to force people to do it.

“And that kind of work over time became less prevalent in the economy. that reduces the economic incentives to do it.”

Claim: As coercive, hazardous labor became less prevalent in the economy over time, the economic incentives to sustain slavery declined.

“i agree it was not descendant but like the in fact this sort of point i'm trying to make is that values that exist at a given time like what the values of uh we will have in 300 years or like from the perspective of someone a thousand years ago what values people are going to have in a thousand years those questions are much more determined by the uh by the technological and economic and social environment that's going to be there in a thousand years which values are going to be functional which uh sites which which values end up being more competitive and uh being more influential so that other people like adopt their values and it depends much less on the individual actions taken by people in a thousand years ago so i would say that”

Claim: The speaker argues that values a thousand years in the future will depend more on the technological, economic, and social environment of that time than on actions taken by people today.

“by the technological and economic and social environment that's going to be there in a thousand years which values are going to be functional which uh sites which which values end up being more competitive and uh being more influential so that other people like adopt their values”

Claim: The speaker argues that values suited to a future environment are more likely to become competitive, influential, and adopted by others.

“like people just have natural preferences that I think are suppressed in various ways during the agricultural era where it's more efficient to have settled societies in like cities which are like fairly authoritarian. and don't allow for that much freedom.”

Claim: The speaker argues that agricultural-era settled societies, which were relatively authoritarian and allowed little freedom, suppressed people’s natural preferences.

“And you're in this Malthusian world where people have very low wages, perhaps compared to what they enjoyed in the hunter-gatherer era.”

Claim: The speaker describes the agricultural era as Malthusian, with people earning very low wages, perhaps relative to their living standards in the hunter-gatherer era.

“And I think people were not, they didn't evolve to have the values that would be functional in that era. So what happened is that there has to be a lot of cultural assimilation where people have to adopt different values.”

Claim: The speaker argues that people had not evolved values suited to the agricultural era and therefore had to adopt different values through cultural assimilation.

“And in the Industrial Revolution, people become also very wealthy.”

Claim: People became much wealthier during the Industrial Revolution than they had been previously.

“compared to what they used to be and that I think leads to different aspects of people values being expressed like people just have a put a huge amount of value on equality like it's always been the case but I think when it is sufficiently functional for that to be suppressed they are capable of suppressing it”

Claim: The speaker argues that Industrial Revolution-era wealth and changing conditions enabled people to express values such as equality that could previously be suppressed when suppression was functional.

“the more important because then you're like oh if the ais become wealthy enough they actually will make a concerted effort to make sure the future looks more like the utility function we put into them which i think you have been under emphasizing”

Claim: The speaker suggests that sufficiently wealthy AI systems may deliberately try to make the future conform more closely to the utility function humans gave them.

“No, I'm not under emphasizing that. I think there are what I would say is there are certain things that are like past dependent in history such that if someone had done something different, like something I've done differently a thousand years ago, then today in some respects would look different. I think.”

Claim: Some aspects of history are path-dependent, such that different actions a thousand years ago would have changed aspects of the present.

“for example which languages are spoken across which boundaries or like which religions people have or like which like those kinds or fashion maybe to some extent though not entirely like those things are more path dependent but then there are things that are not as path dependent so for example if some empire like if the mongols had been more successful and like they somehow i don't know how realistic it is but like they became like very authoritarian and had slavery everywhere would that have actually led to”

Claim: The distribution of languages across boundaries and of religions among people is relatively path-dependent.

“have or like which like those kinds or fashion maybe to some extent though not entirely like those things are more path dependent but then there are things that are not as path dependent so for example if some empire like if the mongols had been more successful and like they somehow i don't know how realistic it is but like they became like very authoritarian and had slavery everywhere would that have actually led to of slavery being a much more enduring institution a thousand years later that seems not true to me like the forces that led to the end of slavery seemed like they were not like they were not contingent forces they seemed like deeper forces than that and if you're saying well if we like if we aligned as today to some bad set of values then that could affect the future in some ways which are more fragile that”

Claim: The speaker doubts that a hypothetical, more successful Mongol Empire imposing widespread authoritarianism and slavery would have caused slavery to remain much more enduring a thousand years later.

“of slavery being a much more enduring institution a thousand years later that seems not true to me like the forces that led to the end of slavery seemed like they were not like they were not contingent forces they seemed like deeper forces than that and if you're saying well if we like if we aligned as today to some bad set of values then that could affect the future in some ways which are more fragile that”

Claim: The forces that ended slavery seemed to the speaker to be deeper and less contingent than the hypothetical historical scenario under discussion.

“of slavery being a much more enduring institution a thousand years later that seems not true to me like the forces that led to the end of slavery seemed like they were not like they were not contingent forces they seemed like deeper forces than that and if you're saying well if we like if we aligned as today to some bad set of values then that could affect the future in some ways which are more fragile that”

Claim: Aligning present-day AIs to bad values could affect some more fragile aspects of the future.

“seems plausible but um i'm not sure how much like how much of the things you care about the future and how much like the ways in which you expect the future to like get worse you actually have a lot of leverage on at the present moment”

Claim: It is uncertain how much leverage people currently have over the future outcomes they care about and expect to deteriorate.

“I mean, another example here might be factory farming where you could say like, oh, us having it's not like us having better values over time led to suffering going down. In fact, our suffering might have gone up. not because the a lot of people say uh”

Claim: Human suffering might have increased as factory farming emerged, rather than decreasing as values improved.

“And probably when factory farming comes to an end, it will be because the incentives start doing that around.”

Claim: Factory farming will probably end when incentives begin to favor ending it.

“No, I don't think you should give up. It's more like”

Claim: People should not give up on preventing a digital equivalent of factory farming.

“uh it's hard to anticipate the consequences of your actions in the very distant future uh so like i would just recommend that you just discount the future not be not for like a moral reason not because like the future is worth less or something but because it's just very hard to anticipate the effects of your actions in the near term i think there are things you can do that seem like they would be beneficial like for example you could try to try to”

Claim: People should discount the distant future—not because it is morally less valuable, but because the effects of their actions are very hard to anticipate over long time horizons.

“align your present AI systems to value the things that you're talking about, like they should like value happiness and they should like dislike suffering or something.”

Claim: People could try to align current AI systems so that they value happiness and disvalue suffering.

“You might want to support like political solutions that would like basically you might want to build up the capacity so that in the future if you notice something like this happening, then we might have some ability to intervene. Like maybe you would think about the prospect of well eventually we're gonna”

Claim: People might support political solutions that build the capacity to intervene if a harmful development is identified in the future.

“then we might have some ability to intervene. Like maybe you would think about the prospect of well eventually we're gonna maybe colonize other stars and like civilization might become very large and communication delays might be very long between different places and in that case competitive pressures between different local cultures might become much stronger because there's like it's harder to centrally coordinate and”

Claim: If civilization colonizes other stars, communication delays between locations might become very long, making central coordination more difficult.

“maybe colonize other stars and like civilization might become very large and communication delays might be very long between different places and in that case competitive pressures between different local cultures might become much stronger because there's like it's harder to centrally coordinate and”

Claim: If civilization spreads across other stars and central coordination becomes harder, competitive pressures between local cultures might intensify.

“so in that world you might expect competition to take over in a stronger way and if you think the result of that is going to be a lot of suffering maybe you would try to stop”

Claim: If interstellar civilization intensifies local competition and that competition is expected to cause substantial suffering, people might try to stop it.

“stop that again like i think at this point it's not it's very far from obvious that trying to say limit competition is actually a good idea i would probably think it's a bad idea but maybe in the future we will receive some information and we'll be like oh like we were wrong actually actually like we should stop this and then maybe you want to have the capacity so that we can make that decision”

Claim: The speaker currently considers limiting competition probably a bad idea, but future information could show that this judgment is wrong and that competition should be stopped.

“but that's a very hard like that's a nebulous thing how do you build that up well i don't know i mean you need to”

Claim: It is difficult to determine how to build the capacity to make future decisions about intervening in competition.

“Like that's the kind of thing I would be trying to do.”

Claim: The speaker would try to build the capacity for future intervention and decision-making.

“Yeah, I think the overall takeaway I take from the way that I think about it, and I guess we think about it as be more humble in what you think you can achieve and like just focus on the nearer term, not because it's more morally important than the longer term, but just because it's much easier to have a predictably positive impact on that.”

Claim: People should be humble about what they can achieve and focus on the nearer term because positive impact is easier to predict there, not because the nearer term is morally more important.

“incredibly epistemically uncertain about what's going to happen i think it's important to just like directly acknowledge like this is instead of just um instead of becoming super certain about like your next conclusion just being like let me just at least from my perspective i'm just like let me just take a step back i'm like i'm not sure what's going on here and i think a lot more people should be from that perspective unless you've had the same opinion about ai for uh many years in which case i have other questions for you about why that's the case um”

Claim: People should openly acknowledge substantial uncertainty about what will happen with AI instead of becoming overly certain about their latest conclusions.

“And in other situations, I mean, generally how we as a society deal with topics on which we are this uncertain is just to have freedom, decentralization, both decentralized knowledge and decentralized decision making.”

Claim: When society faces topics about which it is highly uncertain, it generally responds through freedom and decentralization of knowledge and decision-making.

“take the reins and not to do super high volatility centralized moves like hey let's nationalize so we can make sure that uh we can make sure the ai the software only singularity is aligned or um not to do make moves that are just incredibly contingent on one worldview that are brittle under other considerations and that's become a much more salient part of my worldview i think just classical liberalism is the way we deal with being this epistemically uncertain and i think we should be more uncertain than we've ever been in history as opposed to many other people who seem to be more certain than they are about other sort of more mundane topics yeah”

Claim: The speaker considers classical liberalism an appropriate way for society to handle substantial epistemic uncertainty.

“take the reins and not to do super high volatility centralized moves like hey let's nationalize so we can make sure that uh we can make sure the ai the software only singularity is aligned or um not to do make moves that are just incredibly contingent on one worldview that are brittle under other considerations and that's become a much more salient part of my worldview i think just classical liberalism is the way we deal with”

Claim: Under substantial uncertainty, the speaker argues against highly volatile centralized interventions, including nationalizing AI to ensure alignment of a software-only singularity, and against policies that depend on one worldview and are brittle under alternatives.

“being this epistemically uncertain and i think we should be more uncertain than we've ever been in history as opposed to many other people who seem to be more certain than they are about other sort of more mundane topics yeah”

Claim: The speaker argues that people should express greater uncertainty about AI than at any previous point in history, rather than being more certain about AI than about mundane topics.

“Yeah, I think it's very hard to predict what happens because of this acceleration basically means that you find it much harder to predict what the world might be in 10 years time.”

Claim: Acceleration makes it much harder to predict what the world might be like in ten years.

“I think these questions are also just like very difficult and we don't have very strong empirical evidence.”

Claim: The questions under discussion are very difficult, and strong empirical evidence about them is lacking.

“And then there's like a lot of this kind of disagreement that exists.”

Claim: There is substantial disagreement about the questions under discussion.

“Like I would say that it's important to like the like in a lot of cases and a lot of situations, it's much more important to be like maintain flexibility and ability to adapt to new circumstances, new information than it is to get a specific plan that's going to be like correct and like that's very detailed and has a lot of specific policy recommendations and things that you should do.”

Claim: In many situations, maintaining flexibility and adapting to new circumstances and information is more important than developing a specific, detailed plan with policy recommendations.

“so like that's actually also the thing that i would recommend like if i want to make the make the transition to ai in this period of explosive growth go better i would just prefer it if we like in general had uh like higher quality institutions and”

Claim: To improve the transition to AI during a period of explosive growth, the speaker would prefer generally higher-quality institutions.

“uh but i am much less bullish on someone sitting down today and working out okay like what will we what will this intelligence explosion or explosive growth be like what should we do like like i think plans that you work out today are not going to be that useful when”

Claim: The speaker is much less optimistic about the value of people trying today to determine what an intelligence explosion or period of explosive growth will be like and what actions should be taken.

“uh but i am much less bullish on someone sitting down today and working out okay like what will we what will this intelligence explosion or explosive growth be like what should we do like like i think plans that you work out today are not going to be that useful when The events are actually occurring because you're gonna let learn so much stuff that you're gonna like update on so many questions that these plans are just going to become obsolete”

Claim: Plans made today for an intelligence explosion or period of explosive growth are unlikely to be very useful once events occur, because people will learn and update on many questions, making the plans obsolete.

“like tanks and airplanes and so on which were now like they existed in World War One but in a much more primitive setting so they were wondering what is going to be the impact of these technologies now that we have in them in much greater scale and the British governments had estimates of how many casualties there will be from aerial bombardments in the first few weeks of the Second World War and they expected hundreds of thousands of casualties basically in like two weeks three weeks after the war begins”

Claim: Before the Second World War, the British government expected aerial bombardment to cause hundreds of thousands of casualties in the war’s first two or three weeks.

“So the idea was that airbombing is basically just unstoppable force. All the major urban centers are going to get bombed. Tons of people will die.”

Claim: Before the Second World War, people expected aerial bombing to be unstoppable, to hit all major urban centers, and to cause many deaths.

“So basically, we can't have a war because if there's a war, then it will be a disaster because we will have this airbomb bombardment.”

Claim: People believed war could not be allowed because aerial bombardment would make it disastrous.

“But later it turned out that that was totally wrong. In fact, in all of Britain, there were fewer casualties from airbombing in the entire sort of six years. of the Second World War than the British government expected in the first few weeks of the war. Like they had less casualties in six years than they expected in like three weeks.”

Claim: Britain suffered fewer casualties from aerial bombing during the entire six years of the Second World War than the British government had expected in the war’s first few weeks.

“Like for example, it turned out to be really infeasible to bomb, especially early on, to bomb cities in daytime because your aircraft would just get shot down.”

Claim: Daytime bombing of cities was very infeasible, especially early in the war, because aircraft would be shot down.

“But then if you try to bomb at nighttime, then your bombing was really imprecise. precise and only a very small fraction of it actually hit and then people also underestimated the extent to which people on the ground could like firefighters and so on could just sort of go around the city and put out fires from bombs that were falling on structures they overestimated the amount of economic damage that it would do they underestimated how economically costly it would be like basically you're sending these aircraft and then they're getting shot down well an aircraft is very expensive”

Claim: Nighttime bombing was very imprecise, and only a very small fraction of bombs hit their targets.

“precise and only a very small fraction of it actually hit and then people also underestimated the extent to which people on the ground could like firefighters and so on could just sort of go around the city and put out fires from bombs that were falling on structures they overestimated the amount of economic damage that it would do they underestimated how economically costly it would be like basically you're sending these aircraft and then they're getting shot down well an aircraft is very expensive”

Claim: People overestimated the economic damage that aerial bombing would cause.

“precise and only a very small fraction of it actually hit and then people also underestimated the extent to which people on the ground could like firefighters and so on could just sort of go around the city and put out fires from bombs that were falling on structures they overestimated the amount of economic damage that it would do they underestimated how economically costly it would be like basically you're sending these aircraft and then they're getting shot down well an aircraft is very expensive”

Claim: People underestimated the ability of firefighters and others on the ground to extinguish fires caused by bombing.

“So in the end, what turned out is when the Allies started bombing Germany, they were like for each dollar of capital they were destroying in Germany, they were spending like four to five dollars on the aircraft and fuel and training of pilots and so on that they were sending in missions.”

Claim: During Allied bombing of Germany, the Allies spent about four to five dollars on aircraft, fuel, and pilot training for every dollar of capital they destroyed.

“And the casualty rate was very high, which later got covered up by the government because they didn't want people to worry about,”

Claim: The casualty rate associated with Allied bombing missions was very high, and the government later concealed it to avoid worrying people.

“So the argument for why it's good is that we're going to have this enormous increase in economic growth, which is going to mean like. enormous amounts of wealth and like incredible new products that you can't even imagine and like healthcare or whatever and like the quality of life of the typical person is probably going to go up a lot early on probably also their wages are going to go up because the AI systems are going to be automating things that are complementary to their work or like it's going to be automating part of their work and then you'll be doing the rest and then you'll be getting paid much more on that”

Claim: Accelerating broad labor automation is expected to produce enormous economic growth, wealth, and new products, including healthcare advances.

“enormous amounts of wealth and like incredible new products that you can't even imagine and like healthcare or whatever and like the quality of life of the typical person is probably going to go up a lot early on probably also their wages are going to go up because the AI systems are going to be automating things that are complementary to their work or like it's going to be automating part of their work and then you'll be doing the rest and then you'll be getting paid much more on that”

Claim: The typical person's quality of life will probably rise substantially early in the transition to broad labor automation.

“enormous amounts of wealth and like incredible new products that you can't even imagine and like healthcare or whatever and like the quality of life of the typical person is probably going to go up a lot early on probably also their wages are going to go up because the AI systems are going to be automating things that are complementary to their work or like it's going to be automating part of their work and then you'll be doing the rest and then you'll be getting paid much more on that”

Claim: Early in the transition, wages will probably rise because AI systems will automate tasks complementary to workers' jobs or parts of their jobs, leaving workers to do the remainder.

“And in the long term, eventually we do expect wages to fall just because of arbitrage with the AIs. But by that point, we think humans will own enormous amounts of capital and there will also be like ways in which even the people who don't own capital, we think are just going to be much better off than they are today.”

Claim: In the long term, wages are expected to fall because of arbitrage with AI systems.

“And in the long term, eventually we do expect wages to fall just because of arbitrage with the AIs. But by that point, we think humans will own enormous amounts of capital and there will also be like ways in which even the people who don't own capital, we think are just going to be much better off than they are today.”

Claim: Even people who do not own capital are expected to be much better off than they are today in the long term.

“And in the long term, eventually we do expect wages to fall just because of arbitrage with the AIs. But by that point, we think humans will own enormous amounts of capital and there will also be like ways in which even the people who don't own capital, we think are just going to be much better off than they are today.”

Claim: By the time wages eventually fall in the long term, humans are expected to own enormous amounts of capital.

“wealth and increased variety of products that we would get in this world it would be probably more than a difference between like 1800 and today so if you imagine that difference is like such a huge difference and then we imagine like two times three times whatever”

Claim: The future increase in wealth and variety of products from automation may exceed the difference between 1800 and today, perhaps by two or three times or another amount.

“I mean, it's unclear that this trades off against the probability of it being achieved successfully or something like that.”

Claim: It is unclear whether accelerating the transition to advanced AI reduces the probability that the transition succeeds.

“a year's worth of delay costs for current people like you know this is this enormous amount of utility that people are able to enjoy um and that gets brought forward by by year or pushed back by year if you delay things by year and how much is this this worth well um you know you can you can look at simple models of how concave people's utility functions are and do some calculations and maybe that's That's worth on the order of tens of trillions of dollars per year in consumption. That is roughly the amount consumers might be willing to defer in order to get, you know, bring forward the date of automation one year.”

Claim: The value of bringing the date of automation forward by one year is estimated at tens of trillions of dollars per year in consumption.

“So first of all, we don't actually think that it's clear whether speeding things up or slowing things down actually makes a doomy outcome more or less likely.”

Claim: The speakers do not consider it clear whether accelerating or slowing AI development changes the likelihood of a catastrophic outcome.

“I think it's just a question that doesn't seem obvious to us.”

Claim: The speakers regard the effect of accelerating versus slowing AI development on catastrophic-outcome risk as an unresolved question.

“we don't really believe that if you just pause and then you do research for 20 years at a fixed level of compute scale, that you're actually going to make that much.”

Claim: The speakers doubt that 20 years of alignment research at a fixed level of compute, following a pause in progress, would yield much progress on relevant alignment questions.

“that much progress on relevant questions on alignment or something like i think like imagine you were trying to make progress on alignment uh in 2016 uh with the computer budgets of 2016 and the like well you would have gotten nowhere basically like you would have discovered none of the things that people have today discovered and that turned out to be useful and i think if you pause today then we will be in a very similar position uh in 10 years”

Claim: The speaker believes that alignment research using 2016-era computer budgets would have made almost no progress and would not have produced the useful discoveries made later.

“that much progress on relevant questions on alignment or something like i think like imagine you were trying to make progress on alignment uh in 2016 uh with the computer budgets of 2016 and the like well you would have gotten nowhere basically like you would have discovered none of the things that people have today discovered and that turned out to be useful and i think if you pause today then we will be in a very similar position uh in 10 years”

Claim: The speaker expects that pausing progress today would leave alignment research in a very similar position ten years later.

“right like we have not made a bunch of discoveries so the scaling is just really important to make progress and alignments in our view and then there's a separate question of how long term is should you be in a very different senses so there's a moral sense or like how much should you actually care about people who are alive today as opposed to people who are not yet born that's just a moral question and there's also a practical question of as we discuss how certain can you be about the”

Claim: The speakers consider scaling essential to making progress on AI alignment.

“in even in purely economic terms like uh in like imagine that uh you would be like each year of delay might cause like maybe 100 million people maybe more uh maybe 150 200 million people who are alive today to end up dying right so the even in purely economic terms the value of a statistical life is like pretty enormous”

Claim: The speaker argues that the economic value of a statistical life is very large, even when considered purely in economic terms.

“in even in purely economic terms like uh in like imagine that uh you would be like each year of delay might cause like maybe 100 million people maybe more uh maybe 150 200 million people who are alive today to end up dying right so the even in purely economic terms the value of a statistical life is like pretty enormous”

Claim: The speaker estimates that each year of delay might result in the deaths of 100 million people alive today, and possibly 150–200 million.

“especially in Western countries it's like sometimes people use numbers as high as 10 million dollars for a single life so imagine you do like 10 million dollars times 100 million people that's like a huge number right so like I think”

Claim: The speaker says that estimates of the value of a statistical life in Western countries sometimes reach $10 million per life.

“Like that is just so enormous that unless you're just so I think for you to think that speeding things up is a bad idea, you have to first be like just have this long term view where you look at the long run future, you think your actions today have high enough leverage that you can predictably affect the direction of long this run future.”

Claim: The speaker argues that opposing acceleration requires a long-term perspective and the belief that present actions can predictably influence the long-run future.

“So like in relative terms, I agree the present moment is a moment of higher leverage and you can expect to have more influence. I just think in absolute terms, the amount of influence you can have is still quite low. so it might be orders of magnitude greater than it would have been 2,000 years ago and still be quite low”

Claim: The speaker believes the present offers substantially greater leverage than it did 2,000 years ago, while the absolute influence available today remains quite low.

“but our view is very much that this transition happens very diffusely by way of many, many organizations and companies doing things and for those actions to be determined a bunch by economic forces rather than kind of idiosyncratic preferences. you know, on the part of labs or these kind of decisions that have these kind of founder effects that last for very”

Claim: The speaker's view is that the AI transition will be diffuse across many organizations and companies, and shaped more by economic forces than by idiosyncratic preferences or lasting founder effects.

“I mean, we agree with this. Like, I think like intelligence isn't the bottleneck that's holding back. you know technological progress or economic growth it's”

Claim: The speaker argues that intelligence is not the limiting bottleneck for technological progress or economic growth.

“like many other things and so I think that this is very much consistent with our view that scaling up your overall economy accumulating capital accumulating human capital having you know all these factors scale”

Claim: The speaker considers broad economic growth—including accumulation of physical and human capital—consistent with their view of progress.

“was even consistent with what I was saying earlier that I was pointing out this like good management and good policies and those just contribute to TFP and they can be bottlenecks.”

Claim: The speaker says that management quality and policy quality contribute to total factor productivity and can themselves be bottlenecks.

“No, we can't.”

Claim: The speaker says better management cannot simply be transferred into sub-Saharan Africa as a plug-and-play solution.

“I mean, if the AI advances are like the kind of geniuses in a data center, then I agree that that might. be bottlenecked by the rest of the economy not scaling up and being able to accumulate the relevant capital to make those changes feasible so I kind of agree with this this picture and I think this is like you know an objection to the geniuses in the data center type view and and like I buy basically this”

Claim: If AI advances provide the equivalent of geniuses in a data center, the rest of the economy might bottleneck their impact by failing to scale up and accumulate the capital needed to make the resulting changes feasible.

“But then some people are not going to want to deploy it or some people are going to have norms and laws and cultural things that are going to make it so that AI is not able to be widely deployed in their economy or not as widely deployed as otherwise might be.”

Claim: Some people may resist deploying AI, while norms, laws, or cultural factors may prevent or limit AI deployment in their economies.

“And that is going to make those countries or societies just slower.”

Claim: Countries or societies that deploy AI less widely will experience slower growth.

“I mean, like some countries will be growing faster, just like Britain and the Netherlands were sort of the leaders in the Industrial Revolution. They were the first countries to start experiencing rapid. growth and in other countries even in europe sort of had to come from behind um well again i just think we expect the same thing to be true for ai and i mean the reason that happened was exactly because of these kinds of reasons where like those countries that culture or governance systems or whatever which were just worse and bottlenecked the deployment and scaling of the new technologies and ideas”

Claim: Some countries will grow faster than others during AI-driven change, as Britain and the Netherlands led the Industrial Revolution and first experienced rapid growth.

“growth and in other countries even in europe sort of had to come from behind um well again i just think we expect the same thing to be true for ai and i mean the reason that happened was exactly because of these kinds of reasons where like those countries that culture or governance systems or whatever which were just worse and bottlenecked the deployment and scaling of the new technologies and ideas it seems very plausible yeah”

Claim: AI-driven growth will likely be uneven across countries because cultural and governance differences can bottleneck the deployment and scaling of new technologies and ideas.

“doesn't often require cultural homogeneity like we can we trade with countries like the U.S. trades with China quite a lot actually and there's like a bunch of disagreements”

Claim: International trade does not usually require cultural homogeneity; the United States trades substantially with China despite disagreements between them.

“I think that would be plausible at the point at which it's revealing a lot about the capabilities and the power of AI.”

Claim: An embargo could become plausible when AI demonstrates much of its capabilities and power.

“And you should also think that that creates both an incentive to embargo, but also an incentive to adopt.”

Claim: As AI's capabilities and power become apparent, they create incentives both to embargo AI and to adopt it.

“The very similar styles of governing that enable AI to be able to produce a lot of value.”

Claim: Similar styles of governance can enable AI to produce substantial value.

“I mean, I think it would to some extent, but I'm not sure if I would expect that a year of lead to be.”

Claim: A technological lead could create an arms-race dynamic, but it is uncertain whether a one-year lead would be sufficient to make a country willing to risk war.

“like enough to take a risk because like if you go to war with china uh i mean for example if the u.s went to war with if you replace china today with china from 1990 or if you replace Russia today with Russia from like 1970 or 1980, it's possible that their ICBM and whatever technology is already enough. Like it's already enough to make like have very strong deterrence. So maybe even that lead, a technological lead is not sufficient so that you would feel comfortable going to war. So that seems possible.”

Claim: A technological lead might not make a country comfortable going to war if the opponent's existing technology already provides strong deterrence.

“I mean, for what it's worth, I do think it's possible if you have it just happen in a few countries which are relatively... large and have enough land or something like those countries could just like they would be starting from a lower base compared to the rest of the world so they would need to catch up to some extent so like if they are just going to sort of grow internally and they're not going to depend on the external supply chains but like that doesn't seem like something that's impossible to me that some countries could do it but it would just be like more difficult but in this setting”

Claim: A few relatively large countries with sufficient land could potentially achieve rapid internal growth from a lower starting point without relying on external supply chains, although catching up would be more difficult.

“if some countries have like a significant policy advantage over the rest of the world and they start growing first and then they won't necessarily have a way to get other countries to adopt their norms and culture so in that case they might just it might be more efficient for them to do the growth locally right so that's why i was saying the growth differentials will probably be determined by like regulatory jurisdiction boundaries more than anything else but like i'm not saying say the u.s”

Claim: Growth differentials will probably depend more on regulatory-jurisdiction boundaries than on other factors when some countries have a significant policy advantage and begin growing first.

“if some countries have like a significant policy advantage over the rest of the world and they start growing first and then they won't necessarily have a way to get other countries to adopt their norms and culture so in that case they might just it might be more efficient for them to do the growth locally right so that's why i was saying the growth differentials will probably be determined by like regulatory jurisdiction boundaries more than anything else but like i'm not saying say the u.s”

Claim: Countries that begin growing first may be unable to persuade other countries to adopt their norms and culture, making it more efficient to pursue growth locally.

“by itself it had AI, but it couldn't get the rest of the world to add up to AI. I think that would still be sufficient for explosive growth.”

Claim: A country such as the United States could achieve explosive growth with AI even if it could not get the rest of the world to adopt AI.

“And so people are like very much appreciating the power and the potential threat that China poses.”

Claim: People in AI circles already recognize China’s power and the potential threat it poses.

“I think the key thing is not just like the scale in terms pure in terms of pure number of people or like number of firms or something, but the scale of the overall economy, which is just measured in how much is being produced in terms of dollars and”

Claim: For assessing national scale, the overall economy’s dollar-valued production matters more than the raw number of people or firms.

“you know, the US is ahead.”

Claim: The United States is ahead in the comparison under discussion.

“Financial services are very important for like raising funding and getting investments and data centers and but that's just”

Claim: Financial services support raising funds and investments for data-center development.

“Relative to like what starting point, like I think people already appreciate that China is very important.”

Claim: People already recognize China’s importance.

“where China is leading but then there are very many domains in which the U.S. is leading or the U.S. and its allies where you know countries that are producing relevant inputs for AI that”

Claim: China leads in some domains, but the United States and its allies lead in many other domains relevant to AI.

“the U.S. has access to but China doesn't so I think I think the U.S. is just like ahead on many dimensions and there's some that China is ahead or at least very close so I don't I don't think this”

Claim: The United States has access to some relevant AI inputs that China lacks; the United States leads on many dimensions, while China leads or is close on some.

“should cause you to update very strongly in favor of China being a much bigger deal, at least depending on where started.”

Claim: Given the comparison of U.S. and Chinese strengths, one should not update very strongly toward China being a much bigger deal; this depends on one’s starting point.

“I think people already think China is a big deal, like this is the big underlying thing here. Like if people were just very dismissive of China, then maybe this would be a reason to update.”

Claim: Because people already regard China as a major factor, the argument would be more update-worthy if people had been dismissive of China.

“Yeah, I mean, I do expect that.”

Claim: The speaker expects AI systems to attain superhuman capabilities.

“I think there's a question of how useful is this concept for thinking about this transition to a world with much more advanced AI and I don't find this like a particularly meaningful or helpful concept like I think people introduce some of these notions that on the surface seem useful but then actually when you”

Claim: The speaker considers the concept of superhuman intelligence unhelpful for understanding the transition to much more advanced AI.

“delve into them and it was like very vague and kind of unclear what you're supposed to make of this and you have this notion of AGI which distinguishes from narrow AI in the sense that it's much more general and maybe you know can do everything that a human can do on average I mean AI systems have these very jagged profiles of capability so you have to somehow take some notion of average capabilities and what exactly does that mean it just feels really unclear and then”

Claim: The speaker argues that defining AGI by average human-level capability is unclear because AI systems have uneven, jagged capability profiles.

“You have this notion of ASI, which is AGI in the sense that it's very general, but then it's also better at humans than on every task. you know, is this a meaningful concept? I guess it's coherent.”

Claim: The speaker considers the concept of ASI coherent, describing it as highly general and superior to humans on every task.

“I think this is not a super useful concept because I prefer just thinking about what actually happens in the world.”

Claim: The speaker considers the ASI concept not very useful and prefers analyzing real-world outcomes instead.

“And you could have a drastic acceleration without having. an AI system that can do everything better than humans can do.”

Claim: The world could experience drastic acceleration without any AI system being better than humans at every task.

“I guess you could have no acceleration when you have an ASI that is better than humans at everything, but it's just very expensive or very slow or something.”

Claim: An AI system superior to humans at every task might produce no acceleration if it is very expensive or slow.

“I just prefer thinking about the overall effects on the world and what AI systems are capable of producing those types of effects.”

Claim: The speaker prefers evaluating AI by its overall effects on the world and the outputs that AI systems can produce.

“Well, I agree it will be much greater.”

Claim: The speaker agrees that adding one million people with John von Neumann-level ability would increase growth much more than adding one million people drawn from the normal distribution.

“I think it's, but what I would say is, for example, if you're thinking about the capabilities of goal playing AIs, then the concept of a superhuman goal AI, yeah, you can say like that could be, that is a meaningful concept. But like, if you're developing the AI, it's not a very useful concept.”

Claim: The speaker considers the concept of a superhuman Go-playing AI meaningful for discussing capabilities but not very useful for AI development.

“if you just look at the scaling curve it's just like just goes up and there's some human level somewhere but like the human level is not privileged in any sense so the question is like is it a useful thing to be thinking about and the answer is probably not depends on what you care about um so i'm not saying we shouldn't focus on trying to make the system smarter than humans are like i think that's a good thing to focus on”

Claim: The speaker thinks focusing on the human-level point on an AI scaling curve is probably not useful in general, though its usefulness depends on one's goals.

“if you just look at the scaling curve it's just like just goes up and there's some human level somewhere but like the human level is not privileged in any sense so the question is like is it a useful thing to be thinking about and the answer is probably not depends on what you care about um so i'm not saying we shouldn't focus on trying to make the system smarter than humans are like i think that's a good thing to focus on”

Claim: The speaker argues that human-level capability is not a privileged point on an AI scaling curve.

“if you just look at the scaling curve it's just like just goes up and there's some human level somewhere but like the human level is not privileged in any sense so the question is like is it a useful thing to be thinking about and the answer is probably not depends on what you care about um so i'm not saying we shouldn't focus on trying to make the system smarter than humans are like i think that's a good thing to focus on”

Claim: The speaker considers it worthwhile to focus on developing AI systems that are more capable than humans.

“I mean, I think AI systems will be very diverse, and so it's not super meaningful to ask. something about you know this very diverse range of systems and where we stand in relation to them”

Claim: The speaker expects AI systems to be highly diverse and considers it not very meaningful to ask where humans stand in relation to them as a group.

“I mean, there will be many things that we just will fail to understand. And to some extent, there are many things today that... people don't understand about how the world works and how certain things are made and and then you know how important is it for us to have access or in principle be able to access those considerations and”

Claim: The speaker expects humans to remain unable to understand many things, noting that people already fail to understand many aspects of how the world works and how things are made.

“I think it's not clear to me that that's particularly important that like any individual human should be able to access all the relevant considerations that produce some outcome like that just seems like overkill like why do why do you Do you need that to happen?”

Claim: The speaker is unsure that every individual needs access to all considerations relevant to producing an outcome and regards requiring such access as potentially excessive.

“those things will just not be accessible to you. And then so you have this trade-off between accessibility and maybe how advanced the world is.”

Claim: The speaker sees a trade-off between how accessible a world's underlying considerations are and how technologically advanced that world is.

“So first of all, I think even if you view that along what you might call the intensive margin, in the sense that you just have more of the products you have today, I think there is just a lot of, like there will be a lot of appetite for that. Maybe not quite 100x, that might start hitting some diminishing returns,”

Claim: The speaker expects considerable demand for greater quantities of existing products, but doubts demand would reach 100 times current levels because diminishing returns may emerge.

“GDP per capita on average in the world is 10k a year or something, And there are people who enjoy millions of dollars. And so there's a gap between what people enjoy and like don't seem to be super diminished in terms of marginal utility. And so there's a big room, there's a lot of room on just purely the intensive margin of just consuming the things we consume today.”

Claim: The speaker argues that substantial expansion in consumption of existing goods is possible, pointing to wealthy people who enjoy millions of dollars without apparently experiencing sharply diminished marginal utility.

“But more and then and then there is this maybe much more important dimension along which we will expand, which is extensive margin of what of what is the scope of things that you're consuming.”

Claim: The speaker expects future consumption growth to come importantly, and perhaps more importantly, from expanding the range of goods and services people consume rather than only increasing quantities of existing products.

“And if you look at something like the Industrial Revolution, that seemed to have been the main dimension along which we kind of expanded to consume more.”

Claim: The speaker says the Industrial Revolution seems to have expanded consumption mainly by increasing the variety of available goods and services.

“There's just on any kind of sector that you care about, transportation, medicine, you know, entertainment and food. There's just this massive expansion in terms of variety of things that we're able to consume that is enabled by new technology or new trade routes or new... or new methods of producing things.”

Claim: The speaker says new technologies, trade routes, and production methods have enabled a major increase in the variety of consumable goods and services across sectors including transportation, medicine, entertainment, and food.

“I mean, like, we totally agree with that. I would say that that's just like a kind of qualitative consideration. It doesn't. itself, it isn't self-sufficient to make a prediction about what growth rates are permitted given these BOMO effects versus not. It's just like a qualitative consideration and then you might need to make additional assumptions to be able to make a quantitative prediction. So I think it's a little bit...”

Claim: The speaker agrees that bottlenecks and diminishing returns matter, but argues that this qualitative consideration alone cannot determine quantitatively which growth rates are possible; additional assumptions are needed.

“no if there's one though like if that's a small part of the economy then you could just still get a lot of growth you just automate everything else and that is going to produce a lot of growth”

Claim: If tasks that AI cannot automate account for only a small share of the economy, automating the remaining tasks could still produce substantial economic growth.

“So first of all, you have to be specific about what are these tasks, what are the current share in economic output.”

Claim: Assessing an objection to AI-driven growth requires specifying the relevant tasks and their current share of economic output.

“The second thing is you have to be specific about how bad do you think the complementarities are. So in numerical terms, economists use the concept of elastic substitution to quantify this. So that gives you a numerical estimate of if you just have much more output. on some dimensions but not that much on other dimensions how much does that increase economic output overall and then there's a third question you can also imagine you automate a bunch of the economy well a lot of humans were working on those jobs so now well they don't need to do that anymore because those got automated so they could work on the jobs that haven't yet been automated so for example as i give the example earlier you might imagine a world”

Claim: Quantifying the strength of complementarities involves estimating elastic substitution, which measures how overall economic output changes when output rises more on some dimensions than on others.

“on some dimensions but not that much on other dimensions how much does that increase economic output overall and then there's a third question you can also imagine you automate a bunch of the economy well a lot of humans were working on those jobs so now well they don't need to do that anymore because those got automated so they could work on the jobs that haven't yet been automated so for example as i give the example earlier you might imagine a world image remote work task get automated first and then sensor motor skills lag behind so you might have a world image software engineers like become physical workers instead”

Claim: If AI automates remote-work tasks before sensorimotor tasks, people might reallocate from software-engineering jobs to physical work.

“Of course, in that world, the wages of physical workers will be much higher than their wages are today.”

Claim: If remote-work tasks are automated before sensorimotor tasks and workers shift into physical jobs, physical workers' wages would be much higher than today.

“So that reallocation also produces a lot of extra growth, even if bottom line is maximally powerful. Even if it's literally you just look at all the tasks in the economy and literally take the worst one for productivity growth, you would still get a lot of increase in output because of this reallocation.”

Claim: Worker reallocation to tasks that remain unautomated can produce substantial additional economic growth, even when the least productivity-enhancing task is treated as the limiting task.

“Yeah. So I think one point that I think is useful to make, our experience talking to economists about this is that they will bring up these kind of more qualitative considerations, whereas the arguments that we make are like make specific quantitative predictions about growth rates.”

Claim: The speaker says economists they have spoken with tend to raise qualitative considerations, whereas the speaker's side makes specific quantitative predictions about economic growth rates.

“That was about, there's some estimates of how much computation the human brain does per second and it's about one e15 flop or so. It's a bit unclear, but and then it turns out that an H100 roughly does on that order of computation and so you can ask the question of how long does it take for an H100 to pay itself back.”

Claim: The speaker says estimates put the human brain's computation at roughly 1e15 FLOPs per second, with uncertainty, and that an H100 performs computation on roughly the same order.

“if you run the software of the human brain, you can then deploy that in the economy and earn. and earn say human wages on the order of 50 to 100k a year or whatever in the U.S. And so then it pays itself back because it costs on the order of 30k per H100 and so you get a doubling time of maybe on the order of a year, right?”

Claim: Under the assumptions that an H100 can run human-brain software, that software can earn about $50,000–$100,000 annually in U.S. wages, and an H100 costs about $30,000, the H100 could pay for itself in roughly a year, suggesting an economic doubling time of about a year.

“And so this is like a very quantitatively specific prediction about, you know, and then there's the response, well, you have Baumol effects and then like, okay, well, what does this mean? Like, does this predict it doubles every two years or every five years? Like, you need just more assumptions in order to make this a coherent objection.”

Claim: Qualitative objections to AI-driven growth require additional assumptions to become coherent objections to a specific quantitative growth prediction.

“And so I think a thing that's a little bit, you know, confusing is just that there are these qualitative objections that I agree with. bottlenecks are indeed important, which is part of the reason I'm more skeptical of this software singularity story.”

Claim: The speaker considers bottlenecks important and says this contributes to their skepticism about a software-singularity scenario.

“But I think this is not sufficient for blocking explosive growth.”

Claim: The speaker believes bottlenecks alone are insufficient to prevent explosive economic growth.

“because the O-ring model is taking the product of many, many inputs and then the overall output is the product of very many things.”

Claim: In the O-ring model, overall output is represented as the product of many inputs or factors.

“And so, but actually this is like pretty optimistic from the point of view of having fewer bottlenecks”

Claim: The speaker says the O-ring model is relatively optimistic about the possibility of having fewer bottlenecks.

“but if one of those products doesn't scale, that is that doesn't limit like, yeah, it means you're less. efficient at scaling than you otherwise would be but you can still get”

Claim: A non-scaling factor in a multiplicative production process can make scaling less efficient without necessarily preventing substantial or unbounded scaling in the overall process.

“yet I disagree with him about like the conclusion and I think that we're going to get explosive growth once we have AI that can flexibly substitute for humans.”

Claim: The speaker disagrees with Tyler's conclusion and expects explosive economic growth once AI can flexibly substitute for humans.

“In the AI world, many humans might just be like zero or even minus, actually.”

Claim: The interviewer suggests that in an AI-driven economy, many humans might contribute zero or negative value to output.

“The main objection often is regulation. And I think we've addressed it implicitly in different points, but might as well just explicitly address why won't regulation stop this.”

Claim: The interviewer identifies regulation as a common objection to explosive AI-driven growth and raises the question of whether regulation could stop it.

“so for what it's worth like we do have like a paper where we go over all the arguments for against explosive growth”

Claim: The speaker says their group has written a paper reviewing arguments for and against explosive economic growth.

“you know and regulation i think is the one that seems like strongest that's against um because like the reason it seems strong is because even though we have made arguments I mentioned before about international competition and like variation of policies among jurisdictions and the strong incentives to adopt this technology both for economic and national security reasons. So I think those are pretty compelling when taken together. But even still,”

Claim: The speaker considers regulation the strongest constraint on AI development, although international competition, policy variation across jurisdictions, and economic and national-security incentives favor adoption.

“like the world does have a surprising ability to like coordinate on just not pursuing certain technologies.”

Claim: The world can sometimes coordinate to refrain from pursuing particular technologies.

“to be extremely confident that this is not going to happen like i think it's less likely that we're going to do this for ai than it is for human cloning but because i think human cloning touches on some other taboos and”

Claim: The speaker is not highly confident that global coordination to restrict AI will fail; they consider such coordination less likely for AI than for human cloning, partly because human cloning implicates additional taboos.

“less valuable but and probably less important also for national security in an immediate sense but”

Claim: The speaker thinks human cloning is probably less valuable than AI and probably less immediately important for national security.

“at the same time as i said it's just hard to rule this out so i i wouldn't say like if someone said well i think like there's a 10 or 15 whatever 20 chance that like there will be some kind of global coordination and you know of regulation and that's going to just be very effective maybe it will be enforced through like sanctions on countries that defect or you know and then that is gonna like maybe it doesn't prevent ai from being deployed but maybe just like slow things down enough enough that you never quite get explosive growth like i don't think that's an unreasonable view if it's like 10 chance could be”

Claim: The speaker considers it reasonable to assign roughly a 10–20% chance to effective global coordination on AI regulation, possibly enforced through sanctions on countries that defect, slowing AI deployment enough to avert explosive growth.

“I mean, some things that we've heard from economists, like again, there was this argument that like people sometimes respond to our argument about explosive growth, which is an argument about growth levels. So we're saying we're going to see 30 percent growth per year instead of 3 percent. They respond to that with an objection about levels.”

Claim: The speaker says some economists answer arguments about explosive growth rates, such as 30% rather than 3% annual growth, with objections about absolute productivity levels.

“And like that is just fundamentally the wrong kind of objection. Like we're talking about the rate of change and you're objecting to it by making an argument about the absolute level of productivity. And as I said before,”

Claim: The speaker considers it fundamentally mistaken to answer an argument about the rate of economic change with an objection about the absolute level of productivity.

“it is not an argument that economists themselves would endorse if it was made about a slower rate of growth continuing for a longer time.”

Claim: The speaker says economists would not endorse an absolute-productivity-level objection when applied to a slower growth rate sustained for a longer period.

“it seems more like um like special pleading like”

Claim: The speaker characterizes the objection as seeming like special pleading.

“Yeah, it is often quite fast. In fact, ChatGPT grew extremely fast, right?”

Claim: ChatGPT experienced extremely rapid growth.

“Right, but that was just purely digital service,”

Claim: The speaker characterizes ChatGPT as a purely digital service.

“One reason to be optimistic is if you think the AIs will literally be dropping remote workers or dropping workers in some cases if you have robotics, then companies are already kind of experienced at onboarding humans.”

Claim: The speaker sees a reason for optimism if AI can replace remote or other workers, because companies already have experience onboarding human employees.

“Onboarding humans doesn't take like a very long time. Maybe it takes six months for like a, even in a particularly difficult job for a new worker to like start being productive. Well, that's not that long.”

Claim: The speaker estimates that onboarding a human worker to productivity can take about six months even in a particularly difficult job, and considers six months relatively short.

“So I don't think that would rule out like companies being able to onboard AI workers, assuming that they don't need to make like a ton of new complimentary innovations and discoveries to like take advantage.”

Claim: The speaker thinks companies could onboard AI workers despite the time required to onboard humans, provided that exploiting AI workers does not require many new complementary innovations or discoveries.