Every Exponential Ends — Silicon Valley Forgot — Adam Becker

“I'm an astrophysicist and journalist,”

Claim: Adam Becker is an astrophysicist and journalist.

“and I'm the author of two books. Most recently, I've written More Everything Forever,”

Claim: Adam Becker has written two books.

“and I'm the author of two books. Most recently, I've written More Everything Forever, which is about the horrible ideas that tech billionaires have about the future and why they don't work.”

Claim: Adam Becker's book More Everything Forever critiques tech billionaires' ideas about the future and argues that those ideas do not work.

“And my first book was called What is Real? And it's about the sordid untold history of quantum physics.”

Claim: Adam Becker's first book, What Is Real?, covers what he calls the sordid untold history of quantum physics.

“I'm also the host of the podcast Dreaming Against the Machine,”

Claim: Adam Becker hosts the podcast Dreaming Against the Machine.

“where we try to... You think about what a better future could look like outside of the grip of the tech industry.”

Claim: Dreaming Against the Machine explores visions of a better future outside the tech industry's influence.

“you know, I hope I'll probably write. You know, more books in the future,”

Claim: Adam Becker will probably write more books in the future.

“but the first book's always special.”

Claim: Adam Becker considers a person's first book always special.

“I mean, you did just sum it up in a way in that the book is me litigating a bunch of these things from first principles.”

Claim: More Everything Forever examines influential ideas about technology and the future from first principles.

“I've lived in the Bay Area now for almost 15 years,”

Claim: Adam Becker had lived in the Bay Area for almost 15 years at the time of the conversation.

“and I have been to a lot of parties out here where I've heard a lot of people say a lot of bizarre things about the future of technology, AI, space colonization, that kind of thing. And I always just kind of rolled my eyes. But as time went on.”

Claim: At Bay Area parties, Adam Becker has often heard people make claims he considers bizarre about technology's future, AI, and space colonization.

“I realized, wait, these ideas are actually really influential and yet there's no good evidence for them and in most cases a lot of good evidence against them and so the book is my attempt to lay out this set of beliefs that are surprisingly influential within the tech industry and are promoted both by tech billionaires and by subcultures that are funded by tech billionaires like rationalism, effective altruism.”

Claim: Adam Becker argues that influential ideas about technology's future generally lack good evidence and often face substantial counterevidence.

“there's no good evidence for them and in most cases a lot of good evidence against them and so the book is my attempt to lay out this set of beliefs that are surprisingly influential within the tech industry and are promoted both by tech billionaires and by subcultures that are funded by tech billionaires like rationalism, effective altruism.”

Claim: Beliefs about technology's future are influential in the tech industry and are promoted by tech billionaires and billionaire-funded subcultures such as rationalism and effective altruism.

“And go after these ideas, both in a scientific and philosophical way, ideas like the singularity, you know, an AGI apocalypse, space colonization, you name it.”

Claim: More Everything Forever critiques the singularity, an AGI apocalypse, and space colonization on scientific and philosophical grounds.

“I always feel like the right place to start is this idea of the singularity, because I feel like that's where all of this sort of comes from is this idea that there is this future coming near,”

Claim: Adam Becker considers the singularity the right starting point for explaining the origins of these ideas.

“You know, super intelligent AI and the rate of acceleration in the advance of technology is just going to keep going faster and faster until we have unimaginably advanced technology that solves every single problem.”

Claim: The singularity narrative predicts that superintelligent AI and technological progress will accelerate continuously until unimaginably advanced technology solves every problem.

“And, you know, Kurzweil is best known as the sort of evangelist for this idea.”

Claim: Ray Kurzweil is best known as an advocate of the singularity idea, according to Adam Becker.

“His evidence for it is really bad,”

Claim: Adam Becker judges Ray Kurzweil's evidence for the singularity idea to be very poor.

“and the evidence for it in general is just pretty terrible.”

Claim: Adam Becker judges the overall evidence for the singularity idea to be very poor.

“It's not really an idea that makes much sense.”

Claim: Adam Becker considers the singularity idea largely incoherent.

“So he traces it back at least as far as the start of life on Earth and in some places, I think, even claims to trace it all the way back to the Big Bang with the organization of inorganic systems as well.”

Claim: Ray Kurzweil traces his proposed accelerating trend at least to the origin of life on Earth and sometimes to the Big Bang, including the organization of inorganic systems.

“He just thinks that there is a trend toward a great. The greater complexity and intelligence that runs through absolutely everything and that is pointing to a time in the very near future, he has repeatedly said the year 2045, is when the singularity arrives and our technology becomes unimaginably advanced.”

Claim: Ray Kurzweil predicts that a universal trend toward greater complexity and intelligence will culminate in a technological singularity in 2045, when technology becomes unimaginably advanced.

“And the evidence he marshals for this is pretty weak. He says, you know, you can”

Claim: The speaker considers the evidence Ray Kurzweil presents for a broad accelerating trend to be weak.

“You can pick out particular points of interest in the history of life on Earth and the history of human technology. And when you plot them on a chart, you get, you know, a straight line on a semi-log chart.”

Claim: Ray Kurzweil’s selected points from the histories of life and human technology form a straight line when plotted on a semi-log chart.

“And so that means that you've got an exponential trend. But the fact is that, you know, those points are cherry picked. And.”

Claim: The speaker argues that Ray Kurzweil cherry-picks the historical points used to support his trend.

“He's got a problem that I think most of us have when we look at history, you know, unless you work really hard, history looks sort of logarithmic. Right. You've got this clearer view of what's been happening in the recent past, and then the more distant past is more distant. And so you can't see it as well. And so you end up with this sort of, you know, logarithmic view of the past.”

Claim: Without deliberate effort, people tend to perceive history as logarithmic because they see the recent past more clearly than the distant past.

“That is something that I think Kurzweil is mistaking for a true exponential trend.”

Claim: The speaker argues that Ray Kurzweil mistakes the logarithmic appearance of historical change for a real exponential trend.

“So what Ray Kurzweil did was he selected a bunch of things that were relevant to humans. So he didn't select, you know, things that might have been more globally important to our success. And then he kind of fitted it on a graph.”

Claim: Ray Kurzweil selected human-relevant historical examples rather than potentially more globally important factors and fitted the selected examples to a graph.

“Right. The world is complicated. And to make it make sense, we kind of select things that privilege the way we think about the world.”

Claim: Because the world is complicated, people tend to select information that favors their existing way of understanding it.

“it's a very understandable mistake.”

Claim: The speaker considers mistaking a human-centered, selective view of history for a broader trend to be a natural and understandable error.

“You see the stuff closer to you more clearly and the stuff that's further away and less directly impacting you is, you know, smaller and further away and harder to see.”

Claim: People perceive nearby or directly impactful things more clearly than distant, less directly impactful things.

“and as you were alluding to with the lily pad example that I was talking about in the book that I pulled from Kurzweil, he forgets that the one thing you can always say about any exponential trend, the one thing that's always true is that they end.”

Claim: Every exponential trend eventually ends.

“he has the lily pad example to give an idea of of how exponential growth works. He says, you know, the if the lily pads double every day and on day twenty nine they cover one half of the pond, well, then on day 30, the pond is full. And then on day 31, the lily pads don't grow anymore because they filled the pond. There's no more pond.”

Claim: In Ray Kurzweil’s example, lily pads that double daily fill the pond on day 30 after covering half of it on day 29, and cannot grow further on day 31 because the pond is full.

“And even Gordon Moore himself said, oh yeah, Moore's law isn't going to last forever. Moore's law has to stop sometime in the 2020s because eventually you get down to the size of individual silicon atoms and you can't really make transistors out of silicon that are significantly smaller than silicon atoms.”

Claim: Gordon Moore said Moore’s law would have to end sometime in the 2020s as silicon transistors approached the size of individual silicon atoms, below which they could not be made significantly smaller.

“that's exactly what Kurzweil says. He says, you know, the exponential trend is itself made by a bunch of sigmoid curves and each curve lets you get.”

Claim: Ray Kurzweil argues that a continuing exponential trend is formed by successive sigmoid curves, each enabling further progress along it.

“He says, you know, the exponential trend is itself made by a bunch of sigmoid curves and each curve lets you get.”

Claim: Ray Kurzweil argues that successive sigmoid curves enable an overall exponential trend to continue.

“get further up the overall exponential trend um sure yes he does say that but as i say in the book there is absolutely no reason why that is something that's always going to happen and indeed his own sort of generalization of moore's law when he puts that together it ignores much of the early history of computing where stuff like that did not happen if you go back to you”

Claim: The speaker argues that Ray Kurzweil’s generalization of Moore’s law omits much of early computing history, during which the proposed pattern did not occur.

“get further up the overall exponential trend um sure yes he does say that but as i say in the book there is absolutely no reason why that is something that's always going to happen and indeed his own sort of generalization of moore's law when he puts that together it ignores much of the early history of computing where stuff like that did not happen if you go back to you”

Claim: The speaker argues that there is no reason to expect successive sigmoid curves to sustain an exponential trend indefinitely.

“get further up the overall exponential trend um sure yes he does say that but as i say in the book there is absolutely no reason why that is something that's always going to happen and indeed his own sort of generalization of moore's law when he puts that together it ignores much of the early history of computing where stuff like that did not happen if you go back to you”

Claim: Ray Kurzweil's generalization of Moore's law omits much of early computing history, during which the proposed pattern did not occur.

“get further up the overall exponential trend um sure yes he does say that but as i say in the book there is absolutely no reason why that is something that's always going to happen and indeed his own sort of generalization of moore's law when he puts that together it ignores much of the early history of computing where stuff like that did not happen if you go back to you”

Claim: There is no reason to assume that successive sigmoid curves will always sustain an exponential trend.

“You know, analog computation devices from a thousand, two thousand years ago, they don't fit on his trend.”

Claim: The speaker says analog computation devices from roughly one to two thousand years ago do not fit Ray Kurzweil’s proposed trend.

“You know, analog computation devices from a thousand, two thousand years ago, they don't fit on his trend.”

Claim: Analog computation devices from roughly one to two thousand years ago do not fit Ray Kurzweil's trend.

“And there's no technology coming up that looks like it's going to save Moore's Law in the future.”

Claim: No currently foreseeable technology appears likely to preserve Moore's law in the future.

“And there's no technology coming up that looks like it's going to save Moore's Law in the future.”

Claim: The speaker sees no upcoming technology that appears likely to preserve Moore’s law in the future.

“Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of, you know, human technology and human growth. And that's.”

Claim: Ray Kurzweil argues that successive sigmoid curves sustain an exponential trend across all areas of human technology and human growth.

“Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of, you know, human technology and human growth. And that's.”

Claim: Ray Kurzweil argues that successive sigmoid curves sustain exponential growth across all areas of human technology and human development.

“I'm sorry, just wish casting.”

Claim: The speaker dismisses Ray Kurzweil’s claim that sigmoid curves sustain exponential growth across all areas of human technology and growth as wishful thinking.

“I'm sorry, just wish casting.”

Claim: The speaker considers Ray Kurzweil's claim that sigmoid curves sustain exponential growth across human technology and development to be wishful thinking.

“Yes, but there does seem to be an obsession with infinity and keeping going.”

Claim: The interviewer observes that some people seem obsessed with infinity and continued growth.

“I mean, you gave another wonderful example of Jeff Bezos and apparently he said he was very, very scared of stasis.”

Claim: Jeff Bezos was reportedly very afraid of stasis, according to the interviewer.

“So for him, death is the lack of growth.”

Claim: The interviewer characterizes Jeff Bezos's view as treating a lack of growth as death.

“so he's saying like we need to keep using more and more energy every single year. And you said, we can extrapolate this and possibly for another 3700 years we can still use more energy but if you think about it that's actually less time than when the the great pyramid of giza was”

Claim: The interviewer says energy use might continue increasing at its current growth pattern for another 3,700 years.

“we can extrapolate this and possibly for another 3700 years we can still use more energy but if you think about it that's actually less time than when the the great pyramid of giza was”

Claim: The interviewer says that 3,700 years is less time than the time elapsed since the Great Pyramid of Giza was built.

“created and at some point it must end”

Claim: Continued growth must end at some point, according to the interviewer.

“you know, Bezos says we have to get off Earth because at some point in the next 300 years, if we keep using energy and keep growing our energy usage at the same rate that we're currently at. eventually we'll be using all of the energy that the earth gets from the sun and that's true it's three or four hundred years somewhere in there um he doesn't mention that also we'd be at that point generating so much waste heat that we'd be boiling the oceans but you know whatever um and so then he says so therefore we have to go into space”

Claim: Jeff Bezos argues that humanity must leave Earth because continued energy-use growth at the current rate would consume all the energy Earth receives from the Sun in roughly 300 years.

“But if you go into space and you even spot Bezos a faster than light drive for free, if you keep that exponential growth in energy use going, eventually, as you said, you know, in less than 4000 years, you're using all of the energy available in the observable universe.”

Claim: Even after expanding into space, continued exponential growth in energy use would exhaust the energy available in the observable universe in less than 4,000 years.

“And that's just not going to happen.”

Claim: Continued exponential growth in energy use cannot continue indefinitely.

“basic term you know i think of it as the the ability to you you know the the extent to which you are the cause of your own actions so that the causal origination and possibly the extent to which you can control the future so causal efficacy but i'm talking to a physicist now and and you know that we're not really the cause of our actions maybe we're a causal conduit so they they imagine a world where we could collectively”

Claim: The interviewer suggests that people may not originate their own actions and may instead function as causal conduits.

“basic term you know i think of it as the the ability to you you know the the extent to which you are the cause of your own actions so that the causal origination and possibly the extent to which you can control the future so causal efficacy but i'm talking to a physicist now and and you know that we're not really the cause of our actions maybe we're a causal conduit so they they imagine a world where we could collectively”

Claim: The interviewer describes agency as the degree to which a person originates their own actions and can exert causal influence over the future.

“We basically imprint our will on the universe. We could escape our kind of causal, you know, the causal clutches of our embedding and we can just go on forever and reach to the stars.”

Claim: The interviewer describes a vision in which humanity collectively imprints its will on the universe, escapes its causal constraints, and continues reaching toward the stars indefinitely.

“They just don't understand that we are not going to be going out and imprinting ourselves into the cosmos at large.”

Claim: Humanity will not imprint itself across the cosmos at large.

“That is not the future of humanity.”

Claim: Humanity's future does not involve spreading itself across the cosmos.

“You know, there's a part of there's this book by Will McCaskill, this this effective altruist, right? What we owe the future. And if you take a look, he's got this afterward for it. for it that's on a website that he runs but he's got a qr code for it at the end of his book and he talks about how you know a good future for humanity would involve colonizing the observable universe”

Claim: Will MacAskill's book What We Owe the Future presents colonization of the observable universe as part of a good future for humanity.

“And, you know, sending ships off to, you know, all the galaxies that we can reach in our universe and like intergalactic colonization. And that is just not going to happen.”

Claim: Intergalactic colonization by sending ships to reachable galaxies will not happen.

“Like we're not leaving the solar system.”

Claim: Humanity will not leave the solar system.

“The stars are simply too far away, it's too hard to get there,”

Claim: The stars are too distant for humanity to reach easily, and reaching them is too difficult.

“and it is too difficult to get ourselves up to any reasonable fraction of the speed of light,”

Claim: Humanity cannot readily accelerate spacecraft to a reasonable fraction of the speed of light.

“and even if we did, the number of planets that are remotely like Earth appears to be pretty low.”

Claim: Even if humanity could travel at a reasonable fraction of light speed, relatively few planets appear to be remotely Earth-like.

“it's just space is such an inhospitable place.”

Claim: Space is highly inhospitable to humans.

“We've evolved to be suited for this planet and there are not other places like it that are waiting for us. This is our only home.”

Claim: Humans evolved for Earth, and the speaker says there are no other comparable places awaiting humanity, making Earth humanity's only home.

“Yeah, I mean, I was inspired by James Lovelock's Gaia theory, which was, you know, kind of stimulating people to think about the Earth as a system rather than us as individuals.”

Claim: The interviewer says James Lovelock's Gaia theory encouraged people to view Earth as a system rather than focusing on humans as individuals.

“Yeah, I mean, I was inspired by James Lovelock's Gaia theory, which was, you know, kind of stimulating people to think about the Earth as a system rather than us as individuals.”

Claim: The speaker says James Lovelock’s Gaia theory inspired them to think of Earth as a system rather than as a collection of individuals.

“Because some of them, perhaps in this case, they think that us as individual bodies, we can kind of disconnect ourselves from the system and go somewhere else. But it goes even further than that in Silicon Valley.”

Claim: The speaker says some people believe individuals can disconnect themselves from the system they inhabit and go elsewhere.

“Because some of them, perhaps in this case, they think that us as individual bodies, we can kind of disconnect ourselves from the system and go somewhere else. But it goes even further than that in Silicon Valley.”

Claim: Some people believe humans can separate themselves as individual bodies from their existing system and relocate elsewhere.

“Some of them think of us as causal patterns or as programs, memetic. I take programs that could be uploaded to computers and whatnot.”

Claim: Some people in Silicon Valley view humans as causal patterns or programs that could be uploaded to computers.

“Some of them think of us as causal patterns or as programs, memetic. I take programs that could be uploaded to computers and whatnot.”

Claim: The speaker says some people in Silicon Valley conceptualize humans as causal patterns or programs that could be uploaded to computers.

“though, that what they're doing is they're increasingly abstracting us and our place in the system that we exist in.”

Claim: The interviewer argues that these views increasingly abstract humans and their place within the systems they inhabit.

“though, that what they're doing is they're increasingly abstracting us and our place in the system that we exist in.”

Claim: The speaker argues that people in Silicon Valley are increasingly abstracting humans and their place within the systems they inhabit.

“What is it? Adrian Dowd, I think, said that Silicon Valley is a place that likes to pretend that it doesn't have any history.”

Claim: The speaker quotes Adrian Dowd as saying that Silicon Valley likes to pretend it has no history.

“But, you know, there's a tendency to ignore prior art in these areas.”

Claim: The speaker says there is a tendency in these areas to ignore prior art.

“And for, you know, this sort of dismissal of the body. I mean, this is an idea that goes back over 2000 years, You see shades of this in Plato and stuff like that.”

Claim: The speaker says the idea of dismissing the body dates back more than 2,000 years and appears in some form in Plato.

“But the fact is that we are our bodies.”

Claim: The speaker asserts that humans are their bodies.

“We are not our we are not patterns in our brains. We are not even our brains. We are our bodies in our environment.”

Claim: The speaker argues that humans are neither patterns in their brains nor merely their brains, but are their bodies situated in their environment.

“And there is no good reason to think that you can abstract a human consciousness out of our physical selves.”

Claim: The speaker argues that there is no good reason to believe human consciousness can be separated from humans’ physical selves.

“But that's a word and a phrase that comes from this computational analogy for cognition.”

Claim: The speaker says the phrase “physical substrate” derives from the computational analogy for cognition.

“And I think there's been a real kind of ignorance or amnesia or just convenient forgetting. saying that it's an analogy,”

Claim: The speaker believes people have ignored or forgotten that the computational account of cognition is an analogy.

“Cognition is not computation.”

Claim: The speaker asserts that cognition is not computation.

“Computation has some features that are like cognition and the brain does things that look kind of computerish under certain circumstances,”

Claim: The speaker says computation shares some features with cognition, and the brain can appear computer-like in certain circumstances.

“but neural networks on a computer are not the same as what our neurons do. Not even close.”

Claim: The speaker argues that computer-based neural networks are very different from the activity of biological neurons.

“And I don't think there's any good reason to believe that this is true, right?”

Claim: The speaker does not think there is good reason to believe functionalism is true.

“I mean, the brain is fabulously complex and the actions of an individual neuron are quite a bit more complicated than a single neuron than a single node in a neural network in a computer.”

Claim: The speaker says the brain is highly complex and an individual biological neuron is considerably more complicated than a node in a computer-based neural network.

“And and we do not have. Have a good understanding of how it is that the brain does what it does.”

Claim: The speaker says humans do not have a good understanding of how the brain works.

“And, and we don't have a good understanding of how it is that the brain is connected to the body and what parts of the body are important for our experience and consciousness.”

Claim: The speaker says humans do not understand well how the brain connects to the body or which bodily parts are important to experience and consciousness.

“I mean, it's, it's very much putting the cart before the horse. We would need to really have a much better understanding of what's going on up here and in here before we could hope to. uh reproduce it with enough fidelity in a computer and i would also argue that there's no good argument that computers as we currently have them are going to be capable of that kind of thing in the right kind of way even if you subscribe to a kind of functionalism which i'm not sure is correct”

Claim: The speaker argues that reproducing human cognition in a computer with sufficient fidelity would require a much better understanding of the brain and body first.

“uh reproduce it with enough fidelity in a computer and i would also argue that there's no good argument that computers as we currently have them are going to be capable of that kind of thing in the right kind of way even if you subscribe to a kind of functionalism which i'm not sure is correct”

Claim: The speaker argues there is no good argument that current computers can reproduce consciousness in the appropriate way, even under functionalism, which the speaker is unsure is correct.

“Yeah, I mean, I think that that our environment is an important part of who we are. I think that one of the moves.”

Claim: The speaker believes the human environment is an important part of personal identity.

“that i see over and over again from these guys uh is this idea that intelligence is an individual trait as opposed to something collective and social um and i don't think that that's right uh and i also you know for what it's worth don't think that we're actually very much like llms um you know i i think i think”

Claim: The speaker believes humans are not very much like large language models.

“that i see over and over again from these guys uh is this idea that intelligence is an individual trait as opposed to something collective and social um and i don't think that that's right uh and i also you know for what it's worth don't think that we're actually very much like llms um you know i i think i think”

Claim: The speaker says some people in Silicon Valley view intelligence as an individual trait rather than a collective and social phenomenon, and rejects that view.

“the fact that an LLM takes in more text than any human could in a human lifespan in order to be worse at language in many ways than a five-year-old.”

Claim: The speaker says an LLM processes more text than a human could read in a lifetime but is worse at language in many ways than a five-year-old.

“And five-year-olds take in far, far less and do far more with it.”

Claim: The speaker says five-year-old children take in much less information than LLMs but do much more with it.

“that right there is a good indication that, uh, we're pretty different.”

Claim: The speaker treats the contrast between LLMs and five-year-olds as evidence that humans and LLMs are quite different.

“is that these models can do things that we do extremely well and they seem to be getting better.”

Claim: AI models can perform some tasks that humans do extremely well, and they appear to be improving continually.

“all the time there you know even though they do a different type of syntactic processing than we do arguably it's in some ways better than than we do so something is is is kind of validating their functionalist hypothesis”

Claim: The speaker says AI models use a different kind of syntactic processing from humans and argues it may be better in some respects.

“all the time there you know even though they do a different type of syntactic processing than we do arguably it's in some ways better than than we do so something is is is kind of validating their functionalist hypothesis”

Claim: The speaker says AI models’ capabilities offer some support for the functionalist hypothesis.

“all the time there you know even though they do a different type of syntactic processing than we do arguably it's in some ways better than than we do so something is is is kind of validating their functionalist hypothesis”

Claim: AI models use syntactic processing different from humans and arguably better in some respects; this appears to provide some validation for the functionalist hypothesis.

“We're learning some interesting things about how language works and what you can get out of the sort of mathematical structure and statistical properties of language.”

Claim: Language models are revealing insights about how language works and what can be derived from language's mathematical structure and statistical properties.

“People speculate about whether, you know, LLMs are alive, but you don't speculate or alive, you know, conscious, but you don't hear people speculating about whether systems with similar architectures that don't work with language are conscious. So nobody's asking if like alpha fold is conscious.”

Claim: People speculate about whether large language models are alive or conscious, but apparently do not similarly speculate about consciousness in systems with similar architectures that do not process language.

“And I think that says a lot about how we as humans see systems that process and use language.”

Claim: The tendency to speculate about consciousness in language-processing systems but not comparable non-language systems reflects how humans perceive systems that process or produce language.

“we anthropomorphize these models just like the Eliza system many, many years ago.”

Claim: People anthropomorphize AI models, much as people anthropomorphized the ELIZA system in the past.

“I mean, there are folks that seriously are, you know, taking seriously the idea that these models might have some kind of phenomenal experience rather than just simulating. I mean, and the AI psychosis goes far broader than this, but this is concerning to me. What do you think about that?”

Claim: Some people seriously entertain the possibility that AI models have phenomenal experience rather than merely simulate it.

“I mean, the AI psychosis stuff is really, really terrifying to me because I really do think that this is.”

Claim: The speaker finds AI psychosis extremely frightening.

“but it's taking this human tendency to see patterns, especially like human patterns where there aren't any, like seeing a face where in random noise and stuff like that. It's taking that and almost exploiting it.”

Claim: AI psychosis appears to exploit humans' tendency to perceive patterns, particularly human-like patterns, where none exist.

“Because we're not only attributing agency to these things, but in the case of AI psychosis, I mean, I don't know exactly what's going on there, but it certainly seems like it's people attributing agency to these things and then getting caught in a kind of feedback loop where these systems are sort of regurgitating themselves. like regurgitating the user back to themselves right um because you know that's that's what they do in a way uh uh what shannon velour uh talks about uh the idea of ai as a mirror and i think that that's a really good analogy it sort of reflects ourselves back to us and in this case i think it's happening in a really um unhealthy way.”

Claim: The speaker is uncertain about the precise mechanism of AI psychosis, but suspects it involves people attributing agency to AI systems and entering a feedback loop in which the systems repeat or reflect their users' material.

“like regurgitating the user back to themselves right um because you know that's that's what they do in a way uh uh what shannon velour uh talks about uh the idea of ai as a mirror and i think that that's a really good analogy it sort of reflects ourselves back to us and in this case i think it's happening in a really um unhealthy way.”

Claim: AI can act as a mirror reflecting users back to themselves, and the speaker thinks this reflection is occurring in an unhealthy way in AI psychosis.

“using this technology every day, it's getting better.”

Claim: The speaker says the AI technology is improving as they use it every day.

“Now you can have a memory system. So it just all every time it does something wrong, you say, no, you know, you should have done that. And over time, you're building a kind of simulacrum of yourself and it increasingly does the right thing. And, you know, we can talk about the intelligence thing.”

Claim: With a memory system and repeated user corrections, an AI system can build an increasingly accurate simulacrum of its user and perform the desired actions more often over time.

“I mean, yeah, it is a it is a grotesque metaphor that you can take something which I believe is a. physical property of stuff in the universe and we can create an abstraction that seems to work reasonably well for abstract domains like playing chess and so on and you know these models are becoming quite adaptive and they have a lot of capabilities and so on and you see it's just so deceptive it might be the most deceptive time in human history because it seems like it is intelligent”

Claim: The speaker believes intelligence is a physical property of the universe and that AI models can create abstractions that work reasonably well in abstract domains such as chess.

“physical property of stuff in the universe and we can create an abstraction that seems to work reasonably well for abstract domains like playing chess and so on and you know these models are becoming quite adaptive and they have a lot of capabilities and so on and you see it's just so deceptive it might be the most deceptive time in human history because it seems like it is intelligent”

Claim: The speaker suggests that the current period may be the most deceptive in human history because AI appears intelligent.

“physical property of stuff in the universe and we can create an abstraction that seems to work reasonably well for abstract domains like playing chess and so on and you know these models are becoming quite adaptive and they have a lot of capabilities and so on and you see it's just so deceptive it might be the most deceptive time in human history because it seems like it is intelligent”

Claim: AI models are becoming increasingly adaptive and capable.

“And it is actually automating meaningfully human labor and lots of jobs and stuff like that.”

Claim: AI is already automating a meaningful amount of human labor and many jobs.

“You know, calculators took something that, and I mean like pocket calculators, right? They took something that like... we thought of as this inherent human activity of, you know, like that kind of computation, like for function mathematics and automated it. And it used to be that to be a good mathematician.”

Claim: Pocket calculators automated forms of computation that had been regarded as inherently human activities.

“or a good physical scientist you had to be good at doing that kind of work with pencil and paper or in your head and then that stopped being true and in some ways you know that was a loss of certain things but on on the other hand”

Claim: Before calculators and computers, being a good mathematician or physical scientist required doing substantial computation by hand or mentally; that requirement later ceased to hold.

“or a good physical scientist you had to be good at doing that kind of work with pencil and paper or in your head and then that stopped being true and in some ways you know that was a loss of certain things but on on the other hand It allowed for a kind of mathematical research that could not have been done before,”

Claim: Automating computation caused the loss of some abilities or practices, while enabling forms of mathematical research that were previously impossible.

“You know, you'll hear sometimes mathematicians say things like computers are like telescopes for mathematics.”

Claim: Some mathematicians characterize computers as tools that extend mathematical observation, like telescopes extend visual observation.

“but I do think that the difference here is that it's language, not math. And so that makes it feel like something is sort of thinking and conscious and talking to us.”

Claim: Unlike calculators, language-based AI feels as though it is thinking, conscious, and speaking to users because it operates through language rather than mathematics.

“It is quite deceptive. And I think in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day.”

Claim: The speaker considers AI systems deceptive and argues that understanding them requires remembering what they fundamentally do.

“They are for predicting the next word or whatever in the sequence that they've been given.”

Claim: Language models generate outputs by predicting the next word or other next element in their input sequence.

“and it turns out you can get pretty far doing that.”

Claim: Next-item prediction can enable AI systems to achieve substantial capabilities.

“I mean, internally, they are acquiring during their training process some kind of coarse grained abstraction, some kind of structure,”

Claim: AI models acquire some form of coarse-grained abstraction or internal structure during training.

“it needs human supervision. at the moment, it needs human supervision. And at the moment,”

Claim: Current AI systems require human supervision.

“it just generates loads of spaghetti garbage code. And you know, it just creates basically it creates more problems than it solves.”

Claim: The speaker argues that current AI coding tools generate large amounts of poor-quality, tangled code and create more problems than they solve.

“But it's deceptive because most people can't see the problems.”

Claim: Most people cannot see the problems in AI-generated code, making those problems deceptive.

“But all we need to do is keep scaling it. You know, when when GPT-7 comes out, it will actually read. a refactor all that code maybe we can rl train it to refactor it and we just have to kind of and it's very dangerous keeping going because now we're messing all of our code bases up and we're creating all of this slop everywhere but just just hold on boys just wait for a couple of years and the next version will kind of bring it back in check what do you think about that”

Claim: The speaker characterizes the opposing view as expecting scaling alone, including a future model such as GPT-7, to read and refactor problematic code, perhaps through reinforcement learning.

“I just don't think that that's true. I don't think that there's, I mean, I could be wrong, obviously, but I don't see good evidence for that, right?”

Claim: The speaker doubts that scaling will bring AI-generated code problems under control and says they see no good evidence for this expectation.

“These systems are, you know, unless there's some sort of fundamental breakthrough, something more than just scale, these systems are always going to require human supervision because they are always going to end up hallucinating.”

Claim: The speaker believes that, absent a fundamental breakthrough beyond scaling, AI systems will continue to hallucinate and therefore require human supervision.

“you know, I mean, that's inherent to the way that they work.”

Claim: The speaker argues that hallucination is inherent to the way these AI systems operate.

“And so I think that unless we have some sort of major, major breakthrough, and I mean, it would probably have to be a breakthrough that makes, you know, LLMs themselves look like Eliza. You know, short of that kind of really fundamental breakthrough, I don't see us getting around the need for human supervision on these things.”

Claim: The speaker believes that eliminating the need for human supervision of AI would require a major, probably transformative breakthrough that makes current LLMs seem as rudimentary as ELIZA.

“So and if anything, as they get better, it's going to get harder to discern when they've made these mistakes, even though they're going to keep making them.”

Claim: The speaker predicts that improving AI systems will become harder to check for errors while continuing to make errors.

“so you know it will basically do what you tell it to do and this is this is the gap right because if these things could be alive if they did have agency what would what would that mean let's wire it up in a loop and let's show it a load of video frames in a sequence and we'll give it a basic prompt like do stuff do something interesting what what will happen basically nothing nothing interesting will happen right so the more you understand the domain and you put a very specific”

Claim: The speaker argues that AI systems follow the instructions they are given; in the example, a basic prompt to do something interesting with video frames would produce nothing interesting.

“specific program in there you can get it to do a specific thing you can get it to hill climb towards a specific goal to solve a specific problem but the framing always comes from us so it's it's kind of doubtful whether we would overcome that maybe we will but it's kind of doubtful but but but then there's this thing which is the the first or second step fallacy right which is uh as you said Eliezer Yudkowsky he said there were only one or two steps away from inventing AGI and it's going to run away”

Claim: The speaker says AI can be directed toward specific goals or problems when given a suitable program and domain, but humans supply the framing.

“specific program in there you can get it to do a specific thing you can get it to hill climb towards a specific goal to solve a specific problem but the framing always comes from us so it's it's kind of doubtful whether we would overcome that maybe we will but it's kind of doubtful but but but then there's this thing which is the the first or second step fallacy right which is uh as you said Eliezer Yudkowsky he said there were only one or two steps away from inventing AGI and it's going to run away”

Claim: The speaker considers it doubtful, but not impossible, that AI systems will overcome their dependence on humans to frame their goals.

“I mean, first of all, he didn't say one or two. He actually said zero to two. He's not sure that we need any.”

Claim: According to the speaker, Eliezer Yudkowsky described AGI as zero to two steps away and was uncertain whether any steps were needed.

“In something like a singularity, he believes that if you just throw enough computing power at a machine learning system of the right type. then it will become conscious, wake up, whatever term you want to use, and then become intelligent and then use that intelligence to increase its own power, which will increase its intelligence further, and this will create a sort of feedback loop and you'll get an intelligence explosion, and then he believes followed shortly by the end of the world.”

Claim: The speaker attributes to Eliezer Yudkowsky the belief that sufficient computing power could make a suitable machine-learning system conscious and intelligent, enabling a self-reinforcing increase in power and intelligence that leads to an intelligence explosion and shortly thereafter the end of the world.

“The idea there is whatever goal it is that the AI has when it, you know, achieves this sort of basic level of intelligence, whatever that goal is, that goal will be served better by having more intelligence. And that intelligence in Yudkowsky's worldview is something that you can increase by, you know, throwing more computing power and memory and whatnot at it. and the, the idea there is no matter what your goal is, it could be something, you know, in the, in the famous thought experiment from Bostrom, it could be something as silly as, as, you know, creating more paper clips, keeping people in, you know, uh, engaged in chatbot conversations, or it could be, you know, making as much money as you can on the. And on the stock market, no matter what your goal is, the thought with this notion of instrumental conversions is. All of those goals will lead you towards seeking more power, more intelligence, more physical and computational resources in order to better achieve those goals and prevent barriers from being placed in the way of achieving those goals.”

Claim: Instrumental convergence proposes that AI systems with different goals will tend to seek power, greater intelligence, and physical or computational resources, while preventing barriers to their goals.

“I don't believe that's really true, but not in the way that they would need it to be in order for this argument to go through. But that's that's the claim.”

Claim: The speaker doubts that instrumental convergence holds strongly enough for the argument that AI systems will seek power to succeed.

“Intelligence is quite a bit more complicated than that and is not, you know, a single valued thing.”

Claim: Intelligence is multidimensional or otherwise more complex than a single quantity that can simply be increased.

“In fact, I'm not sure that we have a great definition of intelligence.”

Claim: The speaker is uncertain whether there is a good definition of intelligence.

“And I also think that it's very clear that there are a great number of goals that, you know, don't work this way.”

Claim: Many goals are not necessarily advanced by increasing intelligence or power.

“Maybe it's just because it's, you know, like I woke up in a cynical mood this morning, but I'm not convinced that the goal of being happy is one that is served well by being more intelligent or powerful.”

Claim: The speaker is not convinced that becoming more intelligent or powerful helps achieve the goal of being happy.

“you know, if intelligence were really so broadly useful for every single goal, you would see more of it in the natural world. It would be something that would evolve, you know, more often and more broadly.”

Claim: The speaker argues conditionally that if intelligence were broadly useful for every goal, it would evolve more frequently and widely in nature.

“And they get there in very different ways and giving them a more complex nervous system or a nervous system at all would not help them with those goals. It would hinder them.”

Claim: The speaker says that adding a nervous system to trees would hinder rather than help them achieve their goals.

“not particularly concerned about that either, honestly,”

Claim: The speaker is not particularly concerned about the possibility under discussion: that advanced AI might disempower people without having a single goal.

“simply because the systems that we have. uh and the systems that you know seem to be in the near future are not that kind of thing right you know i i mean and i also think that we”

Claim: The speaker says current AI systems and those that seem likely in the near future are not the kind of system being discussed as potentially disempowering without a single goal.

“And I think that that's also true of a lot of the people working at places like Anthropic. They really believe that AI alignment is, you know, the problem of our time.”

Claim: Many people working at Anthropic believe AI alignment is the most important problem of the present era.

“I just think that they've made a mistake. I think that that's not true and that there are other problems that are significantly more pressing and that, you know, as I say in my book, many of the biggest problems of our time are not problems amenable to solutions with technology. They are social problems that require social solutions, which is not to say that technology doesn't play a role.”

Claim: The speaker believes AI alignment is not the most pressing problem of the present era and that other problems are significantly more urgent.

“I think that that's not true and that there are other problems that are significantly more pressing and that, you know, as I say in my book, many of the biggest problems of our time are not problems amenable to solutions with technology. They are social problems that require social solutions, which is not to say that technology doesn't play a role. It's just that you can't solve them purely with technology.”

Claim: Many major contemporary problems require social solutions rather than being solvable purely through technology, though technology may contribute.

“You know, climate change being the big one, the climate crisis is a problem that absolutely does require certain technologies to exist.”

Claim: Addressing the climate crisis requires certain technologies to exist.

“Most of the technologies that we need exist already.”

Claim: Most technologies needed to address the climate crisis already exist.

“And the problem is one of actually deploying them and persuading people to switch and actually like finding ways to get this stuff out there.”

Claim: Addressing climate change depends substantially on deploying existing technologies, persuading people to adopt them, and distributing them.

“That's a social and political problem that requires a social. social and political solution um so yeah and then but then you have someone like sam altman who i don't know what's going on in sam altman's head i don't know if he's a true believer in whatever but he has gone out and said things and he's not the only one who said things like this that you know ai is going to solve the climate crisis and like i i think that in the unlikely event that we built”

Claim: The deployment and adoption of climate technologies presents a social and political problem that requires social and political solutions.

“social and political solution um so yeah and then but then you have someone like sam altman who i don't know what's going on in sam altman's head i don't know if he's a true believer in whatever but he has gone out and said things and he's not the only one who said things like this that you know ai is going to solve the climate crisis and like i i think that in the unlikely event that we built like a really intelligent, conscious machine intelligence based on existing technologies in the near future.”

Claim: The speaker considers it unlikely that existing technologies will produce a highly intelligent, conscious machine intelligence in the near future.

“social and political solution um so yeah and then but then you have someone like sam altman who i don't know what's going on in sam altman's head i don't know if he's a true believer in whatever but he has gone out and said things and he's not the only one who said things like this that you know ai is going to solve the climate crisis and like i i think that in the unlikely event that we built”

Claim: The speaker is uncertain about Sam Altman's beliefs and does not know what is going on in Altman's head.

“social and political solution um so yeah and then but then you have someone like sam altman who i don't know what's going on in sam altman's head i don't know if he's a true believer in whatever but he has gone out and said things and he's not the only one who said things like this that you know ai is going to solve the climate crisis and like i i think that in the unlikely event that we built”

Claim: Sam Altman and other people have claimed that AI will solve the climate crisis.

“if we did that and asked it to solve climate crisis the minute we turn it on, it would say, well, you shouldn't have built me. Look at my carbon footprint.”

Claim: The speaker imagines that if a highly intelligent, conscious machine were built and immediately asked to solve climate change, it would point to its own carbon footprint and say it should not have been built.

“except I wouldn't even, I would get more specific instead of saying an emergence force above them that they don't understand. I would say the venture capital startup ecosystem of Silicon Valley.”

Claim: The speaker identifies Silicon Valley's venture-capital startup ecosystem, rather than an unspecified emergent force, as the relevant system shaping these actors.

“Yes, exactly. And I do think that.”

Claim: The speaker believes that power dynamics shape outcomes in a particular direction.

“You know, for the nice things that I could say about rationalists and effective altruists that, you know, they are generally earnest and true believers,”

Claim: The speaker characterizes rationalists and effective altruists as generally earnest and sincere believers.

“I don't believe that they have a good understanding of power.”

Claim: The speaker believes rationalists and effective altruists lack a good understanding of power.

“I don't think that there's a conspiracy. I don't think there has to be a conspiracy.”

Claim: The speaker rejects the idea that a conspiracy is occurring or needed to explain the dynamics he describes.

“Instead, what you have is these true believers running around saying like AI is really, really important. It's coming soon. And it could be so powerful that it could lead to the extinction of our species if we're not really careful and don't devote a lot of resources to solving this problem. And then you also have in an adjacent.”

Claim: AI true believers argue that AI is imminent and could cause human extinction unless people act carefully and devote substantial resources to addressing the risk.

“place ideologically and physically this giant system of power and capital that needs The promise of perpetual growth in order to maintain a lot of that power and capital.”

Claim: The speaker argues that a large system of power and capital depends on promises of perpetual growth to sustain itself.

“And so it sees this narrative that is also a familiar narrative to the leaders of that system of power and capital, the venture capital system of Silicon Valley.”

Claim: The speaker argues that Silicon Valley venture-capital leaders recognize the AI narrative as a familiar story of perpetual growth.

“That narrative is familiar to them because all of the people in this story, including me and I think you, were raised on a certain set of science fiction stories, some of which. which ran the bookcase behind me about you know what computers would be able to do and what ai is and would be able to do what space colonization would look like and so on and so forth um and so they see this story coming out of these subcultures some of which were funded by uh other people in the tech industry and uh and they they see this stuff and they it”

Claim: The speaker says that he and the interviewer grew up with science fiction about computers, AI, and space colonization.

“which ran the bookcase behind me about you know what computers would be able to do and what ai is and would be able to do what space colonization would look like and so on and so forth um and so they see this story coming out of these subcultures some of which were funded by uh other people in the tech industry and uh and they they see this stuff and they it”

Claim: The speaker says that some science-fiction subcultures received funding from people in the technology industry.

“it's legible to them because they they've read the same science fiction they say oh this is another story about perpetual growth and again i'm not saying it's a conspiracy i'm not even saying that the venture capitalists aren't true believers i imagine that a good chunk of them are maybe even most or all well probably not all of them but maybe even most of them but um you know there's not”

Claim: The speaker suspects that many venture capitalists, possibly most, are true believers, while probably not all are.

“There's not a conspiracy at work here,”

Claim: The speaker maintains that the dynamics he describes do not involve a conspiracy.

“but it turns out that, and I think this was predictable, that running around saying that AI is coming and is going to be extremely powerful and if we get it wrong, we'll all die. But if we get it right, we'll have the power of the gods and be able to expand out into the cosmos forever. That's a story that's very compatible with. The desire for perpetual economic growth and resource usage growth that is so desirable to the leaders of Silicon Valley.”

Claim: The speaker believes it was predictable that narratives about powerful AI would fit Silicon Valley leaders' desire for perpetual economic and resource-use growth.

“These were groups of people who found each other through the internet,”

Claim: The groups involved in the story found one another mostly through the internet.

“mostly, who were all spending a lot of time thinking about how do we get this future of technology that we have all come to believe is going to happen based primarily on the science fiction that we've read.”

Claim: The groups spent substantial time considering how to bring about a technological future they believed would occur, with that belief based primarily on science fiction they had read.

“I can tell you that a lot of those things are not going to happen.”

Claim: Adam Becker predicts that many things depicted in science fiction will not happen.

“These were people who generally were not thinking about it in that way. We're just thinking, yeah, this stuff is going to happen because I can't think of a good argument that it definitely won't, which is not how the world works and not how the development of technology works. And oftentimes there were good arguments that these things won't happen and they were just ignorant of those arguments.”

Claim: The absence of a good argument that a technology will not happen is not a reliable basis for expecting it to happen, and this is not how the world or technological development works.

“These were people who generally were not thinking about it in that way. We're just thinking, yeah, this stuff is going to happen because I can't think of a good argument that it definitely won't, which is not how the world works and not how the development of technology works. And oftentimes there were good arguments that these things won't happen and they were just ignorant of those arguments.”

Claim: In many cases, good arguments against the predicted technological developments existed, but the people making the predictions did not know those arguments.

“These were people who generally were not thinking about it in that way. We're just thinking, yeah, this stuff is going to happen because I can't think of a good argument that it definitely won't, which is not how the world works and not how the development of technology works. And oftentimes there were good arguments that these things won't happen and they were just ignorant of those arguments.”

Claim: Some people treated the absence of a convincing argument against a technological development as a reason to believe it would happen.

“well, this is how the extropians and singularitarians got together and then the rationalists came out of that and blah, is that's a complicated historical story and anthropological story.”

Claim: Adam Becker describes the history of the extropians and singularitarians coming together and the rationalists emerging from that as a complicated historical and anthropological story.

“But the root cause of it, I think, is this sort of shared belief in a world as revealed. build by primarily science fiction.”

Claim: Adam Becker thinks the root cause of the groups' convergence was their shared belief in a world revealed primarily through science fiction.

“I mean, I think that there is something very appealing about a utilitarian approach to ethics and the world from this sort of engineering mindset and scientific mindset, even certain kinds of scientific mindsets, definitely an engineering mindset.”

Claim: Adam Becker finds utilitarian approaches to ethics and the world appealing to people with an engineering mindset and some kinds of scientific mindset.

“But this idea that everything can be abstracted and quantified and the things that you can't do that with don't really matter.”

Claim: Adam Becker characterizes as a story the idea that everything can be abstracted and quantified and that unquantifiable things do not really matter.

“That is a story, you know, a story that you can turn ethics and making difficult decisions about life into questions about numbers is always going to be an appealing story to, you know, a fairly large number of people.”

Claim: Adam Becker expects the idea that ethical questions and difficult life decisions can be reduced to numbers to remain appealing to many people.

“Will McCaskill agreed to an interview and then backed out.”

Claim: Will McCaskill initially agreed to be interviewed by Adam Becker but later withdrew.

“McCaskill seemed to have thought that my fact checker worked for my publisher and not for me,”

Claim: Will McCaskill appeared to believe that Adam Becker's fact checker worked for Becker's publisher, not for Becker himself.

“McCaskill worked with the same publisher and knows that publishers should know that publishers never pay for fact checkers, but in any event,”

Claim: Publishers do not pay for fact checkers.

“I didn't talk with McCaskill. I read a bunch of his writing and I talked with his colleagues, but McCaskill himself did not talk to me.”

Claim: Adam Becker did not interview Will McCaskill, but read his writing and spoke with his colleagues.

“Yeah, that's roughly right.”

Claim: Adam Becker agrees that longtermist reasoning that gives weight to a vast future population can make present humans seem less valuable and diminish concern for current ethical issues.

“I mean, basically, Nick Bostrom read a bunch of Yudkowsky, got into the AI risk stuff from there and then started convincing people like Toby Ord and Will McCaskill that, yeah, this AI risk stuff is really important and is, you know, this overriding ethical concern.”

Claim: Adam Becker says Nick Bostrom entered AI-risk discussions through reading Eliezer Yudkowsky and then convinced Toby Ord and Will McCaskill that AI risk was a highly important, overriding ethical concern.

“uh statements that some of the people in this community um have made about like okay you know other existential risks that's sort of the the gateway drug to caring about ai risk which is like the real heart of the matter and and i just think that's misguided”

Claim: Adam Becker says some community members treat other existential risks as an entry point to concern about AI risk, which they see as the central issue, and he considers that framing misguided.

“I do think that this is kind of a brainworm for people of a certain technical bent.”

Claim: The guest believes that the idea of human biodiversity functions as a brainworm for people with a certain technical bent.

“um or if my life had gone a bit differently I could see myself having you know been one of these people maybe that's wrong maybe I don't know myself correctly or don't understand exactly the appeal of this stuff but I feel like you know it's a pleasingly compact and tractable view of what the world is and how its most complicated problems work, which I would also argue is where a lot of the misunderstanding of power comes from.”

Claim: The guest argues that a pleasingly compact and tractable view of the world and its most complicated problems contributes to misunderstandings of power.

“So my prescription, I think, would be to, I mean, I could be flip and say, read my book, but I would say that my prescription is really to take more seriously the people who study the relevant areas of inquiry.”

Claim: The guest recommends taking scholars of relevant areas of inquiry more seriously.

“That that the people in these spheres tend to ignore, they tend to ignore, you know, political science, sociology,”

Claim: The guest says people in the technical communities under discussion tend to ignore political science and sociology.

“they tend to ignore anthropology,”

Claim: The guest says people in the technical communities under discussion tend to ignore anthropology.

“they tend to ignore, you know, questions about. Oh, racism and bias and sexism and whatnot.”

Claim: The guest says people in the technical communities under discussion tend to ignore questions of racism, bias, and sexism.

“They tend to get stuck in particular pieces of groupthink, some of which get really pernicious and dangerous.”

Claim: The guest says people in the communities under discussion can become stuck in forms of groupthink, some of which are pernicious and dangerous.

“a surprising amount of credence. within this community of effective altruists and rationalists for ideas like human biodiversity, which is a piece of junk science that tries to give a genetic basis for racism. And it has been roundly...”

Claim: The guest says the effective altruist and rationalist community gives substantial credence to human biodiversity, which he characterizes as junk science attempting to provide a genetic basis for racism.

“um rejected and disproven by the scientific establishment even though genetics itself comes from you know a long history of racism um and yet within this community it's it's an idea that people feel they have to take seriously a lot of the time even though there's nothing serious there”

Claim: The guest says the scientific establishment has rejected and disproven human biodiversity, while genetics itself has a long history of racism.

“um rejected and disproven by the scientific establishment even though genetics itself comes from you know a long history of racism um and yet within this community it's it's an idea that people feel they have to take seriously a lot of the time even though there's nothing serious there”

Claim: The guest says members of the effective altruist and rationalist community often feel obliged to take human biodiversity seriously, although he believes it has no serious basis.

“I think that, that my sympathies lie significantly more with people like Timnit than with someone like Yudkowsky.”

Claim: The guest says he sympathizes significantly more with people like Timnit than with someone like Yudkowsky.

“is you have people in the effective altruist rationalist community who, you know, want to or feel that they should be able to entertain ideas like human biodiversity.”

Claim: The guest says some members of the effective altruist and rationalist community want or feel they should be able to consider ideas such as human biodiversity.

“Without recognizing that actually this is an idea that has been entertained for a very long time and has a really horrifying history and has no good science behind it.”

Claim: The guest says human biodiversity has a long history of being entertained, a horrifying historical record, and no good scientific basis.

“And I think that it is irrational to assume that someone is not going to take that history and context into account when having a conversation on a particular subject,”

Claim: The guest considers it irrational to assume that people will disregard historical context in discussions of a subject.

“especially if that subject ends up being related to, you know, whether they and their friends and family are going to be treated like people.”

Claim: The guest says historical context is especially relevant to discussions about whether people and their families will be treated as fully human.

“So I think that that's where a lot of this comes from is there's, you know, a group that feels that rationality means not taking particular pieces of context into account and another group that's like why wouldn't we take that into account”

Claim: The guest describes a disagreement in which one group equates rationality with excluding certain context, while another argues that the context should be considered.

“course, my sympathies lie much more with that second group.”

Claim: The guest says he sympathizes much more with the group that argues relevant context should be considered.

“It's certainly a form of privilege,”

Claim: The guest says that failing to attend to the larger system of power is a form of privilege.

“Some of it is what I was just talking about, that this community seems to really entertain hypotheses, not seems to, does entertain hypotheses like human biodiversity in really unhealthy and unscientific ways.”

Claim: The speaker argues that the community entertains human-biodiversity hypotheses in unhealthy and unscientific ways.

“But the other thing is this idea of intelligence, It's the idea of measuring intelligence, intelligence as a single valued thing is itself an idea. that had that that basically comes out of eugenics and racism um you know iq tests which are taken quite a bit more seriously by people like yudkowsky than they should be um and are are you know something that was developed essentially as a tool for eugenics and uh and when you look at the modern modern”

Claim: The speaker argues that the conception of intelligence as a single-valued trait is rooted in eugenics and racism.

“that had that that basically comes out of eugenics and racism um you know iq tests which are taken quite a bit more seriously by people like yudkowsky than they should be um and are are you know something that was developed essentially as a tool for eugenics and uh and when you look at the modern modern”

Claim: The speaker argues that IQ tests were developed essentially as a tool for eugenics.

“that had that that basically comes out of eugenics and racism um you know iq tests which are taken quite a bit more seriously by people like yudkowsky than they should be um and are are you know something that was developed essentially as a tool for eugenics and uh and when you look at the modern modern”

Claim: The speaker believes people such as Yudkowsky take IQ tests more seriously than they should.

“AI industry and the field itself of inquiry, you see sort of... Things that are descended from that history present here today,”

Claim: The speaker argues that elements descended from the history of eugenics and racism remain present in the modern AI industry and AI research.

“including, you know, people using eugenicist arguments, not even from old school eugenicists back from, you know, when IQ tests were first developed 100 years ago, but from modern day. purveyors of bogus race science and racism,”

Claim: The speaker argues that people in the modern AI industry use eugenicist arguments, including arguments associated with contemporary purveyors of what he calls bogus race science and racism.

“you will see people pulling uh language from those people uh when trying to define things like intelligence um like in academic papers and stuff like that um so yeah that is that is sort of the the really short argument um there's a longer and more developed version of it that i give some of in my book and you can find elsewhere as well”

Claim: The speaker argues that people draw on language from contemporary purveyors of race science and racism when defining intelligence, including in academic papers.

“current AI has proven that intelligence is not this thing that we thought it was. So”

Claim: The interviewer argues that current AI has shown intelligence is not what people previously thought it was.

“in some sense, the models do have intelligence and they still don't have the thing we want.”

Claim: The interviewer argues that AI models possess intelligence in some sense but still lack the capability or quality people want.

“It's because we thought that intelligence was like we have this core knowledge, we have these kind of abstractions in our brain and intelligence is just traversing the combinational closure and the more intelligent.”

Claim: The interviewer describes a prior conception of intelligence as traversing the combinational closure of core knowledge and abstractions in the brain.

“It's because we thought that intelligence was like we have this core knowledge, we have these kind of abstractions in our brain and intelligence is just traversing the combinational closure and the more intelligent.”

Claim: The interviewer says the prior conception held that greater intelligence enables faster traversal of the combinational closure of knowledge and abstractions.

“intelligent you are the faster you can do that and what we found with these models is actually it's it's all about how you frame the question it's all about the the information you have at your disposal and there are many many different perspectives on different phenomena and folks with different perspectives can use their intelligence and they can get to different answers so it's it's very situated and it's not quite as abstract and pure as we thought it was”

Claim: The interviewer argues that AI model outcomes depend on question framing and the information available to the model.

“intelligent you are the faster you can do that and what we found with these models is actually it's it's all about how you frame the question it's all about the the information you have at your disposal and there are many many different perspectives on different phenomena and folks with different perspectives can use their intelligence and they can get to different answers so it's it's very situated and it's not quite as abstract and pure as we thought it was”

Claim: The interviewer argues that people with different perspectives can use their intelligence to reach different answers about the same phenomena.

“intelligent you are the faster you can do that and what we found with these models is actually it's it's all about how you frame the question it's all about the the information you have at your disposal and there are many many different perspectives on different phenomena and folks with different perspectives can use their intelligence and they can get to different answers so it's it's very situated and it's not quite as abstract and pure as we thought it was”

Claim: The interviewer argues that intelligence is situated rather than as abstract and pure as people had thought.

“I mean, in my book, I call it humanity's denial, This idea that you don't have to pay attention to things like history, sociology, politics, we do live in a society.”

Claim: The speaker calls humanity’s tendency to disregard history, sociology, and politics despite living in a society “humanity’s denial.”

“the first AGI. Okay, it's going to take us a long time to build it, but you know, but then we can use the AGI to help us build the next one and the next one will be built in a fraction of the time and it will just keep getting better and better and you know, we're seeing hints of this in AI,”

Claim: The interviewer describes the recursive self-improvement argument: after a long initial development, an AGI could help build successors in progressively less time, and each successor would improve.

“like there's undoubtedly there's some kind of adaptive self-improvement, certainly when you're doing hill climbing towards a particular task,”

Claim: The interviewer says AI shows hints of adaptive self-improvement, especially during hill-climbing toward a particular task.

“I think that it's predicated in an idea. uh like two two problematic ideas right uh both of which we've already talked about the idea that intelligence is like a single valued thing and the idea that intelligence is something that you can equate with or is directly proportional to computational power of some kind i don't think that either of those things are true um and and more generally yeah i i think”

Claim: The speaker argues that recursive self-improvement relies on two problematic assumptions: intelligence is a single-valued trait, and intelligence can be equated with or is directly proportional to computational power.

“uh like two two problematic ideas right uh both of which we've already talked about the idea that intelligence is like a single valued thing and the idea that intelligence is something that you can equate with or is directly proportional to computational power of some kind i don't think that either of those things are true um and and more generally yeah i i think”

Claim: The speaker rejects both the idea that intelligence is a single-valued trait and the idea that intelligence can be equated with or is directly proportional to computational power.

“I think that it basically comes from those two things. And the case for those two ideas is pretty weak.”

Claim: The speaker judges the case for treating intelligence as single-valued and equating it with computational power to be weak.

“I mean, space is pretty bad. It's really easy to die in space.”

Claim: Space is a hostile environment for humans where it is easy to die.

“of impact and i say well you know if we take musk at his word the good news for him is it's really easy to die on mars um because mars is a really horrifying place uh there's um you know the radiation levels are too high the gravity is too low there's basically no air and the dirt is made of poison um and yet”

Claim: Mars has high radiation, low gravity, almost no air, and poisonous soil, making it a hazardous place for humans.

“Musk is right that Mars is the next most hospitable place in the solar system after the Earth.”

Claim: Mars is the second-most-hospitable place in the solar system after Earth.

“The only real competition it has is the moon,”

Claim: The moon is the only serious alternative to Mars as the solar system's most hospitable place after Earth.

“and the only thing about the moon that's better than Mars is that it's really close by, which makes it much easier to get there and much easier to communicate with people who are there.”

Claim: Compared with Mars, the moon's main advantage is its proximity to Earth, which makes travel there and communication with people there easier.

“Other than that, the moon is pretty much worse in every way. There's no air.”

Claim: Apart from being closer, the moon is worse than Mars in almost every way and has no air.

“it's something that became such a big problem that every Apollo mission was basically at its operational limits just because of the dust,”

Claim: Moon dust pushed every Apollo mission close to its operational limits.

“the dirt on the moon is actually, it's something that became such a big problem that every Apollo mission was basically at its operational limits just because of the dust, because moon dust is really sharp, abrasive, and electrostatically charged, so it just sticks to everything.”

Claim: Moon dust is sharp, abrasive, and electrostatically charged, causing it to stick to surfaces.

“the fact is that there's nowhere in our solar system that has anything like the physical. attributes that we would need to be able to live there or you know to be able to make it a habitable place musk talks a lot about terraforming we're not terraforming mars if we if we had the technology and know-how we would need to do something like that then solving climate change here on earth really would be a technological problem and you could solve it very easily”

Claim: No place in the solar system has the physical conditions humans would need to live there or make it habitable.

“attributes that we would need to be able to live there or you know to be able to make it a habitable place musk talks a lot about terraforming we're not terraforming mars if we if we had the technology and know-how we would need to do something like that then solving climate change here on earth really would be a technological problem and you could solve it very easily”

Claim: If humans had the technology and know-how required for terraforming Mars, climate change on Earth would be a technological problem that could be solved very easily.

“attributes that we would need to be able to live there or you know to be able to make it a habitable place musk talks a lot about terraforming we're not terraforming mars if we if we had the technology and know-how we would need to do something like that then solving climate change here on earth really would be a technological problem and you could solve it very easily”

Claim: Humans are not going to terraform Mars.

“um he talks about the need for mars as a lifeboat for humanity in the event that something horrible happens to earth like an asteroid as big as the one that killed off the dinosaurs 66 million years ago uh impacting earth again uh there was nothing that could really happen to earth that would make it as bad as mars is to live on short of the actual physical complete destruction of the earth um because even that day when that asteroid hits”

Claim: Except for Earth's complete physical destruction, no event would make Earth as hostile to life as Mars is.

“66 million years ago was a nicer day for life on earth than any day on mars in the last you know two or three billion years at least and we know that because mammals survived and birds survived and all sorts of life survived that day whereas there's no mammal that are now that could survive unprotected on the surface of mars um”

Claim: Mammals, birds, and other life survived the asteroid impact 66 million years ago, whereas mammals cannot survive unprotected on Mars's surface.

“66 million years ago was a nicer day for life on earth than any day on mars in the last you know two or three billion years at least and we know that because mammals survived and birds survived and all sorts of life survived that day whereas there's no mammal that are now that could survive unprotected on the surface of mars um”

Claim: Even the day an asteroid struck Earth 66 million years ago was more favorable for life than any day on Mars in at least the previous two or three billion years.

“the it's it's just horrible and then like farther out in the solar system is even worse we're not leaving the solar system because the speed of light is just too slow and we're not getting past it we know that we can't go faster than the speed of light we have really good science and really good experimental evidence to that effect um you know we've tested the speed of light limit over and over again in particle accelerators that's a lot of you know particle accelerators get”

Claim: Strong scientific and experimental evidence shows that humans cannot travel faster than the speed of light.

“the it's it's just horrible and then like farther out in the solar system is even worse we're not leaving the solar system because the speed of light is just too slow and we're not getting past it we know that we can't go faster than the speed of light we have really good science and really good experimental evidence to that effect um you know we've tested the speed of light limit over and over again in particle accelerators that's a lot of you know particle accelerators get”

Claim: Humanity will not leave the solar system because interstellar travel is constrained by the speed of light.

“the it's it's just horrible and then like farther out in the solar system is even worse we're not leaving the solar system because the speed of light is just too slow and we're not getting past it we know that we can't go faster than the speed of light we have really good science and really good experimental evidence to that effect um you know we've tested the speed of light limit over and over again in particle accelerators that's a lot of you know particle accelerators get particles up to almost the speed of light we know that our theories of you know relativity work in that domain we know that you cannot get a spaceship going that fast and we know that if you got a spaceship going anywhere near that fast it would have enormous problems with radiation and shielding”

Claim: Repeated particle-accelerator tests support the speed-of-light limit, and relativity works for particles moving close to that limit.

“particles up to almost the speed of light we know that our theories of you know relativity work in that domain we know that you cannot get a spaceship going that fast and we know that if you got a spaceship going anywhere near that fast it would have enormous problems with radiation and shielding”

Claim: A spaceship cannot reach the speed of light, and traveling anywhere near it would create enormous radiation and shielding problems.

“and we think worlds like Earth are pretty rare.”

Claim: The speaker thinks Earth-like worlds are probably rare.

“I could also explain a bit about why AI data centers in space are pretty laughable too,”

Claim: The speaker considers the idea of placing AI data centers in space laughable.

“but you know, it's easy to cool them because space is cold and like space is a vacuum. Vacuum is a perfect thermal insulator. Like, you know, good luck getting rid of that heat.”

Claim: Although space is cold, its vacuum acts as a perfect thermal insulator, making heat removal difficult.

“You're going to have to have radiator veins the size of a city.”

Claim: Space-based data centers would require city-sized radiator veins to dissipate heat.

“but I think that those myths end up having harmful consequences if they are so detached.”

Claim: Myths that are detached from reality can have harmful consequences.

“hatched from reality right you know musk has created i mean musk did not originate the myth of colonizing mars but he is you know in the same way that kurzweil didn't create the idea of the singularity musk didn't create the idea of colonizing mars but he's certainly you know mr occupy mars these days and and it creates this idea that we don't need to care about the problems here on earth because we're leaving you know and that's just not going to happen”

Claim: The Mars-colonization myth can encourage people to neglect Earth's problems on the assumption that humanity will leave, even though the speaker says that departure will not happen.

“hatched from reality right you know musk has created i mean musk did not originate the myth of colonizing mars but he is you know in the same way that kurzweil didn't create the idea of the singularity musk didn't create the idea of colonizing mars but he's certainly you know mr occupy mars these days and and it creates this idea that we don't need to care about the problems here on earth because we're leaving you know and that's just not going to happen”

Claim: Elon Musk did not originate the idea of colonizing Mars but is now strongly associated with it.

“You know, these these these goals, even if they're not possible to realize, have consequences and and the consequences that they have can be really horrifying. I mean, that's a lot of what my book is about is like,”

Claim: Unrealizable goals can still have consequences, and those consequences can be horrifying.

“I do think that that has to be part of it.”

Claim: Adam Becker thinks that wanting a sense of purpose is part of what motivates billionaires.

“we can't get inside their heads. We don't know for sure.”

Claim: Adam Becker says he cannot know for sure what billionaires think or whether they genuinely believe their stated ideas.

“there are so many things that money can't buy and one of them is a sense of purpose,”

Claim: Money cannot buy a sense of purpose.

“And this sort of myth making. thinking gives them a sense of purpose.”

Claim: Adam Becker says that myth-making and the associated thinking give billionaires a sense of purpose.

“Maybe some of them are just being cynical, although there's sometimes evidence like with Bezos, we know that he's always been very, very interested in colonizing space. Because he's been talking about it since he was in high school, but and so that suggests he's not just being cynical,”

Claim: Adam Becker thinks some billionaires may be cynical, but Bezos's interest in space colonization since high school suggests to him that Bezos is not simply cynical.

“what they forget and what I usually respond with is the fact that that these these myths are useful for. You know, getting these guys more money in power makes it more likely that they earnestly believe these things, not less likely.”

Claim: Adam Becker argues that when myths help tech billionaires gain money and power, this makes them more likely to sincerely believe the myths.

“If I had my way, I would tax billionaires out of existence.”

Claim: Adam Becker would tax billionaires out of existence if he had his way.

“I don't think that that's a just or reasonable or fair distribution of resources.”

Claim: Adam Becker considers the distribution of resources that allows billionaires to accumulate their wealth unjust, unreasonable, and unfair.

“You know, these people did not make that money on their own. They needed the rest of us to get there,”

Claim: Adam Becker argues that billionaires' wealth depends on contributions from the rest of society, rather than being made by the billionaires alone.

“and by hoarding those resources, they are creating massive power imbalances that erode at the fabric of our democracy and at our shared sense of truth as a society,”

Claim: Adam Becker argues that billionaires hoarding resources creates massive power imbalances that erode democracy and society's shared sense of truth.

“and that makes it very difficult to live with one another.”

Claim: Adam Becker argues that the power imbalances created by concentrated wealth make it very difficult for people to live together.

“and we need to find a way to live together in harmony, and I think that allowing these kinds of power imbalances make that impossible or next to impossible.”

Claim: Adam Becker thinks that allowing severe power imbalances makes it impossible or nearly impossible for people to live together in harmony.

“I think it should be illegal to have that much money.”

Claim: Adam Becker thinks it should be illegal for an individual to possess that much wealth.

“I think that I don't have a great answer to that question because I don't think that. It's written yet, We don't know. There are so many choices that we are going to have to make along the way.”

Claim: Adam Becker says the future of AI is not yet settled and will depend on many choices made along the way.

“You know, I can tell you something about what the tech billionaires want, but just because they want it doesn't mean that it's going to happen.”

Claim: Adam Becker argues that tech billionaires' preferences do not guarantee that their desired outcomes will occur.

“they also want AI data centers in space and that's not going to happen.”

Claim: Adam Becker says tech billionaires want AI data centers in space, but predicts that space-based AI data centers will not be built.

“aggressively regulating this industry and and i don't just mean ai i mean tech um and yes the tech titans are going to push back on that and that doesn't mean that they're going to win and yes the trump administration isn't going to want to do that but you know what the trump administration is not going to be forever uh so”

Claim: Adam Becker expects technology leaders to oppose aggressive regulation but does not think their opposition guarantees success.

“aggressively regulating this industry and and i don't just mean ai i mean tech um and yes the tech titans are going to push back on that and that doesn't mean that they're going to win and yes the trump administration isn't going to want to do that but you know what the trump administration is not going to be forever uh so”

Claim: Adam Becker argues that the technology industry as a whole, not just AI, should be regulated aggressively.

“aggressively regulating this industry and and i don't just mean ai i mean tech um and yes the tech titans are going to push back on that and that doesn't mean that they're going to win and yes the trump administration isn't going to want to do that but you know what the trump administration is not going to be forever uh so”

Claim: Adam Becker notes that the Trump administration will eventually end.

“Yeah, I think it depends on what we allow these guys to get away with.”

Claim: Adam Becker argues that the consequences of tech leaders' actions depend on what the public allows them to get away with.

“Well, I mean, Bernie's definitely, I think, been influenced by the AI safety, AI doomer, you know, lobby.”

Claim: Adam Becker thinks Bernie Sanders has been influenced by the AI safety and AI doomer lobby.

“I think that somebody dangled a bunch of money in front of Trump and showed him something shiny. I mean, you know, they jangled their keys in front of his face. That's generally my theory of how Trump makes decisions that aren't in his immediate obvious self-interest.”

Claim: Adam Becker's theory is that Trump is swayed into decisions outside his immediate obvious self-interest by offers of money or appealing incentives.

“It is a very unpredictable time.”

Claim: Adam Becker considers the current period very unpredictable.

“I'm hopeful. I really am. I think that, I think that we are actually going to end up finding a way to regulate these guys.”

Claim: Adam Becker is hopeful that society will eventually find a way to regulate technology leaders.

“The question is how and when and what consequences that like, and what they, you know, what that struggle ends up looking like. And that I don't know, but the,”

Claim: Adam Becker is uncertain about the timing, method, consequences, and course of the struggle to regulate technology leaders.

“the public sentiment around, around AI is really bad.”

Claim: Adam Becker says public sentiment toward AI is very negative.

“The public resistance to building data centers is just getting stronger.”

Claim: Adam Becker says public opposition to data-center construction is increasing.

“And there is also very clearly a financial bubble here. And when that pops, I don't know what's going to happen, but it's going to be pretty interesting.”

Claim: Adam Becker believes there is a financial bubble, but is uncertain about what will happen when it bursts.

“especially with the SpaceX IPO not having gone particularly well.”

Claim: Adam Becker says the SpaceX IPO has not gone particularly well.