Andrej Karpathy — AGI is still a decade away

“evolve and i think um i was triggered by that because i feel like there's some over predictions going on in the industry and uh in my mind this is really a lot more accurately described as the decade of agents and we have some very early agents that are actually like extremely impressive and that i use daily uh you know cloud and codecs and so on but i still feel like there's uh so much work to be done and so i think my like my reaction is like we'll be working with these things for decades they're gonna get better”

Claim: Andrej Karpathy uses early AI agents such as Claude and Codex daily and considers them extremely impressive.

“evolve and i think um i was triggered by that because i feel like there's some over predictions going on in the industry and uh in my mind this is really a lot more accurately described as the decade of agents and we have some very early agents that are actually like extremely impressive and that i use daily uh you know cloud and codecs and so on but i still feel like there's uh so much work to be done and so i think my like my reaction is like we'll be working with these things for decades they're gonna get better”

Claim: Andrej Karpathy believes AI agents still require substantial work and will continue to improve over decades.

“evolve and i think um i was triggered by that because i feel like there's some over predictions going on in the industry and uh in my mind this is really a lot more accurately described as the decade of agents and we have some very early agents that are actually like extremely impressive and that i use daily uh you know cloud and codecs and so on but i still feel like there's uh so much work to be done and so i think my like my reaction is like we'll be working with these things for decades they're gonna get better”

Claim: Andrej Karpathy believes industry predictions about the timeline for AI agents are too optimistic and that agent development is better described as a decade-long process than as something that will happen in one year.

“So in my mind, I mean, when you're talking about an agent, I guess, or what the labs have in mind and what maybe I have in mind as well, is it's you should think of it almost like an employee or like an intern that you would hire to work with you.”

Claim: Andrej Karpathy defines an AI agent as something akin to an employee or intern hired to work with a person.

“Like currently, of course, they can't. And the reason you don't do it today is because they just don't work.”

Claim: Andrej Karpathy believes current AI agents do not work well enough to perform the work being discussed.

“like they don't have enough intelligence they're not multimodal enough they can't do computer use and all this kind of stuff and they don't do a lot of the things that you've alluded to earlier you know they don't have continual learning you can't just tell them something and they'll remember Yeah, it and they're just cognitively lacking and it's just not working and i just think that it will take about a decade to work through all those issues”

Claim: Andrej Karpathy believes current AI agents lack sufficient intelligence, multimodal capability, and computer-use ability.

“like they don't have enough intelligence they're not multimodal enough they can't do computer use and all this kind of stuff and they don't do a lot of the things that you've alluded to earlier you know they don't have continual learning you can't just tell them something and they'll remember Yeah, it and they're just cognitively lacking and it's just not working and i just think that it will take about a decade to work through all those issues”

Claim: Andrej Karpathy believes current AI agents lack continual learning and cannot reliably retain information they are told.

“like they don't have enough intelligence they're not multimodal enough they can't do computer use and all this kind of stuff and they don't do a lot of the things that you've alluded to earlier you know they don't have continual learning you can't just tell them something and they'll remember Yeah, it and they're just cognitively lacking and it's just not working and i just think that it will take about a decade to work through all those issues”

Claim: Andrej Karpathy predicts that resolving the problems that keep AI agents from working well will take about a decade.

“Yeah, I guess this is where you get into like a bit of, I guess, my own intuition a little bit and also just kind of doing a bit of an extrapolation of with respect to my own experience in the field. Right.”

Claim: Andrej Karpathy bases his timeline estimates partly on intuition and extrapolation from his experience in AI.

“So I guess I've been in AI for. Almost two decades. I mean, it's going to be maybe 15 years or so, not that long.”

Claim: Andrej Karpathy describes his experience in AI as almost two decades, qualifying that it is about 15 years.

“But I do have about 15 years of experience of people making predictions of seeing how they actually turned out. And also,”

Claim: Andrej Karpathy says he has about 15 years of experience observing predictions and their outcomes.

“And I feel like the problems are tractable. They're surmountable. still difficult.”

Claim: Andrej Karpathy believes the problems facing AI are solvable but remain difficult.

“stuff that happened I mean AI is actually like so wonderful because there have been a number of I would say seismic shifts That were like the entire field has sort of like suddenly looks a different way Right and I guess I've maybe lived through two or three of those and I still think there will continue to be some because they come with some kind of like almost surprising regularity Well my when my career began of course like when I started to work on deep learning when I became interested in deep learning This was just kind of like by chance of being right next to Jeff Hinton at University of Toronto”

Claim: Andrej Karpathy says AI has experienced several major shifts that rapidly changed the field and expects further shifts, given their surprisingly regular occurrence.

“stuff that happened I mean AI is actually like so wonderful because there have been a number of I would say seismic shifts That were like the entire field has sort of like suddenly looks a different way Right and I guess I've maybe lived through two or three of those and I still think there will continue to be some because they come with some kind of like almost surprising regularity Well my when my career began of course like when I started to work on deep learning when I became interested in deep learning This was just kind of like by chance of being right next to Jeff Hinton at University of Toronto of Toronto and Jeff Hinton of course is kind of like the godfather figure of AI and he was training all these neural networks and I thought it was incredible and interesting but this was not like the main thing that everyone in AI was doing by far this was a niche little subject on the side that's kind of maybe like the first like dramatic sort of seismic shift that came with the AlexNet and so on”

Claim: When Andrej Karpathy became interested in deep learning, it was a niche area rather than the main focus of AI research.

“this was a niche little subject on the side that's kind of maybe like the first like dramatic sort of seismic shift that came with the AlexNet and so on I would say like Alex and sort of reoriented everyone and everyone started to train neural networks,”

Claim: Andrej Karpathy says AlexNet marked a dramatic shift in AI that reoriented the field toward training neural networks.

“but it was still like very like per task, per specific task. So maybe I have an image classifier or I have a neural machine translator or something like that.”

Claim: After the AlexNet-era shift, neural-network systems were still generally task-specific, such as image classifiers and neural machine translators.

“And people became very slowly actually interested in basically kind of agents, I would say.”

Claim: Interest in AI agents developed gradually, according to Andrej Karpathy.

“And I would say the Atari sort of deep reinforcement learning shift in 2013 or”

Claim: Andrej Karpathy dates the Atari deep reinforcement learning shift to 2013.

“part of that early effort of agents in my mind, because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments.”

Claim: Early AI-agent research sought to create systems that perceive environments, take actions, interact with them, and receive rewards.

“And I kind of feel like that was a misstep, actually. And it was a misstep that actually even the early open AI that I was a part of, of course. kind of adopted because at that time the zeitgeist was reinforcement learning environments,”

Claim: Karpathy considers the early focus on Atari games and reinforcement-learning environments for developing agents a misstep that early OpenAI also adopted.

“So that was maybe like another like prominent part of I would say AI where maybe for two or three or four years, everyone was doing reinforcement learning on games.”

Claim: Karpathy estimates that for roughly two to four years, AI researchers broadly focused on reinforcement learning in games.

“And basically that was a little bit of a misstep.”

Claim: Karpathy considers the period of broad reinforcement-learning research on games to have been somewhat misguided.

“And what I was trying to do at OpenAI actually is like I was always a little bit suspicious of games as being like this thing that would actually lead to AGI because in my mind you want something like an accountant or like something that's actually interacting with the real world. And I just didn't see how games kind of like add up to it.”

Claim: Karpathy was skeptical that game-playing would lead to AGI and favored agents capable of real-world interaction and work such as accounting.

“And so my project at OpenAI, for example, was within the scope of the universe project on an agent that was using keyboard and mouse to operate web pages.”

Claim: Karpathy's project at OpenAI's Universe project developed an agent that operated web pages using a keyboard and mouse.

“And I really wanted to have something that like interact. tracked with you know the actual digital world that can do knowledge work and it just so turns out that this was extremely early way too early so early that we shouldn't have been working on that you know because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments your reward is too sparse and you just won't learn and you're going to burn a forest computing and you're never actually going to get something off the ground and so what you're missing is this power of”

Claim: Karpathy wanted agents to operate in the real digital world and perform knowledge work, but believes the effort began far too early.

“tracked with you know the actual digital world that can do knowledge work and it just so turns out that this was extremely early way too early so early that we shouldn't have been working on that you know because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments your reward is too sparse and you just won't learn and you're going to burn a forest computing and you're never actually going to get something off the ground and so what you're missing is this power of”

Claim: Karpathy argues that agents trained by unguided keyboard and mouse interaction receive rewards too sparsely to learn effectively, potentially wasting vast computing resources without yielding a functional system.

“so for example today people are training those computer using agents but they're doing it on top of a large language model and so you actually have to get the language model first you have to get the representations first and you have to do that by all the pre-training and all the LLM stuff”

Claim: Karpathy says computer-using agents are built on large language models, whose representations must first be developed through pretraining.

“So I kind of feel like maybe loosely speaking, it was like people keep maybe trying to get the full thing too early a few times where people like really try to go after agents too early, I would say, and that was Atari and universe and even my own experience.”

Claim: Karpathy believes AI researchers have repeatedly pursued complete agents prematurely, as in Atari, Universe, and his own work.

“And you actually have to do some things first before we sort of get to those agents.”

Claim: Karpathy believes foundational capabilities must be developed before pursuing full AI agents.

“And maybe now the agents are a lot more competent, but maybe we're still missing sort of some parts of that stack.”

Claim: Karpathy thinks current AI agents may be substantially more capable than earlier ones, while some components of the necessary stack may still be absent.

“But I would say maybe those are like the three major buckets of what people were doing, training neural nets per tasks, trying to the first round of agents, and then maybe the LLMs and actually seeking the representation power of the neural networks before you tack on everything else on top.”

Claim: Karpathy identifies task-specific neural-network training, early agent development, and LLM-based representation learning before adding further capabilities as three broad approaches in AI.

“And I kind of feel like I'm very careful to make analogies to animals because they came about by a very different optimization process.”

Claim: Karpathy cautions against direct analogies between AI systems and animals because animals were produced by a different optimization process.

“Animals are evolved and they actually come with a huge amount of hardware that's built in.”

Claim: Karpathy says animals are products of evolution and possess substantial built-in biological hardware.

“for example, my example in the post was the zebra, a zebra gets born and a few minutes later it's running around and following its mother. mother that's an extremely complicated thing to do”

Claim: Karpathy notes that a zebra can run and follow its mother within minutes of birth, behavior he considers extremely complicated.

“um that's not reinforcement learning that's something that's baked in and evolution obviously has some way of encoding the weights of our neural nets in attcg's and i have no idea how that works but it apparently works so i kind of feel like brains just came from a very different process and i i'm very hesitant to take inspiration from it because we're not actually running that process so in my post i kind of said”

Claim: Karpathy is reluctant to use brains as a model for AI because brains were produced by an evolutionary process that AI developers are not reproducing.

“um that's not reinforcement learning that's something that's baked in and evolution obviously has some way of encoding the weights of our neural nets in attcg's and i have no idea how that works but it apparently works so i kind of feel like brains just came from a very different process and i i'm very hesitant to take inspiration from it because we're not actually running that process so in my post i kind of said”

Claim: Karpathy says evolution appears to encode neural-network weights in genetic material, while acknowledging that he does not know how the process works.

“um that's not reinforcement learning that's something that's baked in and evolution obviously has some way of encoding the weights of our neural nets in attcg's and i have no idea how that works but it apparently works so i kind of feel like brains just came from a very different process and i i'm very hesitant to take inspiration from it because we're not actually running that process so in my post i kind of said”

Claim: Karpathy argues that a newborn zebra's ability to run and follow its mother is innate rather than learned through reinforcement learning.

“And we're not actually building animals. Because we're not doing training by evolution.”

Claim: Karpathy says current AI development does not create animals or train systems through biological evolution.

“We're doing training by basically imitation of humans and the data that they've put on the internet.”

Claim: Karpathy describes current AI training as relying on imitation of humans and on human-created internet data.

“And so you end up with these sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans. And it's a different kind of intelligence.”

Claim: Karpathy characterizes AI systems trained through human imitation and internet data as digital, human-mimicking entities representing a different kind of intelligence.

“Like if you imagine a space of intelligences, we are starting off at a different point.”

Claim: Karpathy believes AI systems begin from a different position in the space of possible intelligences than humans do.

“almost we're not we're not really building animals but i think it's also possible to make them a bit more animal-like over time and i think we should be doing that and so i kind of feel like sorry just i guess one more please point is”

Claim: Karpathy believes AI systems could become more animal-like over time and that developers should pursue this direction.

“i do feel like sutton basically has a very like his framework is like we want to build animals and i actually think that would be wonderful if we can get that to work that would be amazing if there was a single like”

Claim: Karpathy characterizes Sutton's framework as aiming to build animal-like AI and says he would welcome that outcome if it proved workable.

“algorithm that you can just you know run on the internet and it learns everything that would be incredible”

Claim: Karpathy considers it an extraordinary possibility for one algorithm to learn everything by running on the internet.

“i almost suspect that i'm not actually sure that it exists and that's certainly actually not what animals do because animals have this outer loop of evolution”

Claim: The speaker is unsure whether reinforcement learning actually occurs in animals.

“and a lot of what looks like learning is actually a lot more maturation of the brain and i think there's actually actually very little reinforcement learning for animals”

Claim: The speaker thinks much apparent learning in animals is brain maturation and that animals engage in very little reinforcement learning.

“And I think a lot of the reinforcement learning is actually like more like motor tasks. It's not intelligence tasks.”

Claim: The speaker thinks that reinforcement learning in animals is mostly associated with motor tasks rather than intelligence tasks.

“So I actually kind of think humans don't actually like really use RL, roughly speaking, is what I would say.”

Claim: The speaker thinks humans generally do not use reinforcement learning.

“But I don't think that humans use reinforcement learning for a lot of intelligence tasks like problem solving and so on.”

Claim: The speaker thinks humans do not use reinforcement learning for many intelligence tasks, including problem solving.

“That doesn't mean we shouldn't do that for research,”

Claim: The speaker does not conclude that reinforcement learning should be excluded from research.

“I would agree with you that there's some miraculous compression going on because obviously the weights of the neural net are not stored in ATCGs. There's some kind of a dramatic compression and there's some kind of like learning algorithms encoded that take over. and do”

Claim: The speaker thinks biological development involves dramatic compression: neural-network weights are not stored directly in DNA's ATCG sequence, and encoded learning algorithms perform some learning online.

“some of the learning online so i can definitely agree with you on that basically i would say i'm a lot more kind of like practically minded i don't come at it from the perspective of like let's build animals i come from from perspective of like let's build useful things so i have a hard hat on and i'm just observing that look we're not going to do evolution because i don't know how to do that but it does turn out we can build these ghost spirit-like entities by imitating internet documents this works and it's actually kind of like Like, it's a way to bring you up to something that has a lot of sort of built-in knowledge and intelligence in some way, similar to maybe what evolution has done.”

Claim: The speaker says that training on internet documents can produce systems with built-in knowledge and intelligence, perhaps resembling an outcome of evolution.

“some of the learning online so i can definitely agree with you on that basically i would say i'm a lot more kind of like practically minded i don't come at it from the perspective of like let's build animals i come from from perspective of like let's build useful things so i have a hard hat on and i'm just observing that look we're not going to do evolution because i don't know how to do that but it does turn out we can build these ghost spirit-like entities by imitating internet documents this works and it's actually kind of like”

Claim: The speaker's practical goal is to build useful systems, not to recreate animals.

“So that's why I kind of call pre-training this kind of like crappy evolution.”

Claim: The speaker characterizes pre-training as a technologically practical but inferior analogue of evolution.

“It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.”

Claim: The speaker says pre-training is a practically achievable way with available technology to create a starting point for reinforcement learning and related methods.

“Basically, the thing that pre-training is doing, so you're basically getting the next token predictor on over the internet, and you're training that into a neural net.”

Claim: The speaker describes pre-training as training a neural network to predict the next token using internet data.

“It's doing two things, actually, that are kind of like unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent.”

Claim: The speaker says pre-training both imparts knowledge to a model and develops its intelligence, which he regards as somewhat separate effects.

“By observing the algorithmic patterns in the internet, it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in context learning and all this kind of stuff. And actually, you don't actually need or want the knowledge. I actually think that's...”

Claim: The speaker thinks exposure to algorithmic patterns in internet data develops neural-network circuits and algorithms that enable in-context learning and related capabilities.

“it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in context learning and all this kind of stuff. And actually, you don't actually need or want the knowledge. I actually think that's... I think that's probably actually holding back the neural network overall because it's actually like getting them to rely on the knowledge a little too much sometimes.”

Claim: The speaker suspects that knowledge acquired during pre-training can hold a neural network back by encouraging excessive reliance on stored knowledge.

“For example, I kind of feel like agents, one thing they're not very good at is going off the data manifold of what exists on the internet.”

Claim: The speaker thinks AI agents struggle to operate beyond the data manifold represented by internet content.

“If they had less knowledge or less memory, actually maybe they would be better.”

Claim: The speaker suggests that AI agents might perform better if they had less knowledge or memory.

“And so what I think we have to do kind of going forward, and this would be part of the research paradigms, is I actually think we need to start, we need to figure out ways to remove some of the knowledge and to keep what I call this. is this cognitive core?”

Claim: The speaker argues that research should find ways to remove some model knowledge while preserving the model's cognitive core.

“Is this like intelligent entity that is kind of stripped from knowledge but contains the algorithms and contains the magic, you know, of intelligence and problem solving and strategies of it and all this kind of stuff?”

Claim: The speaker defines a cognitive core as an intelligent system with reduced stored knowledge but retained algorithms and problem-solving capabilities.

“but I still think that, so in context learning, basically it's pattern completion within a token window,”

Claim: In-context learning is pattern completion within a token window.

“And it just turns out that there's a huge amount of patterns on the internet.”

Claim: The internet contains a huge amount of patterns.

“And so you're right, the model kind of like learns to complete the pattern and that's inside the weights.”

Claim: A model learns to complete patterns, with this learned pattern-completion behavior represented in its weights.

“The weights of the neural network are trying to discover patterns and complete the pattern.”

Claim: Neural-network weights encode mechanisms that discover and complete patterns.

“And there's some kind of an adaptation that happens inside the neural network,”

Claim: Some form of adaptation occurs inside a neural network during in-context learning.

“And I do think it's possible that in context learning actually runs a small gradient descent loop internally in the layers of the neural network.”

Claim: In-context learning may run a small gradient-descent loop internally within neural-network layers.

“And so I recall one paper in particular where they were doing linear regression, actually, using in context learning.”

Claim: A paper demonstrated linear regression using in-context learning.

“So basically your inputs into the neural network are X, Y pairs. x y x y x y that happened to be on the line and then you do x and you expect the y and the neural network when you train it in this way actually does do does do linear regression and normally when you would run linear regression you have a small gradient descent optimizer that basically looks at x y looks at an error calculates the gradient of the weights and does the update a few times it just turns out that when they looked at the weights of that in context learning algorithm they actually found some”

Claim: In the described in-context learning setup, a neural network receives X,Y examples and predicts the Y corresponding to a new X.

“then you do x and you expect the y and the neural network when you train it in this way actually does do does do linear regression and normally when you would run linear regression you have a small gradient descent optimizer that basically looks at x y looks at an error calculates the gradient of the weights and does the update a few times it just turns out that when they looked at the weights of that in context learning algorithm they actually found some analogies to gradient descent mechanics.”

Claim: Researchers found mechanisms in an in-context learning algorithm's weights that were analogous to gradient descent.

“then you do x and you expect the y and the neural network when you train it in this way actually does do does do linear regression and normally when you would run linear regression you have a small gradient descent optimizer that basically looks at x y looks at an error calculates the gradient of the weights and does the update a few times it just turns out that when they looked at the weights of that in context learning algorithm they actually found some”

Claim: A neural network trained on X,Y examples in the described setup performs linear regression.

“In fact, I think even the paper was stronger because they actually hard-coded the weights of a neural network to do gradient descent through attention and all the internals of the neural network.”

Claim: A paper hard-coded neural-network weights so gradient descent was implemented through attention and other network internals.

“So I guess that's just my only pushback is that who knows how in context learning works, but I actually think that it's probably doing a little bit of some kind of funky gradient descent internally and that I think that that's possible. So I guess I was only pushing back on you're saying it's not doing in context learning, who knows what it's doing, but it's probably maybe doing something similar to it, but we don't know.”

Claim: How in-context learning works is not known, but it probably performs some form of gradient descent internally.

“I mean, the way I usually put this is that anything that happens during the training of the neural network, the knowledge is only kind of like a hazy recollection of what happened in training in the training time.”

Claim: A neural network's weights retain training knowledge as a hazy, compressed recollection of the training data.

“And that's because the compression is dramatic. like you've you're taking 15 trillion tokens and you're compressing it to just your fine neural network with a few billion parameters so obviously it's a massive amount of compression going on uh so i kind of refer to it as like a hazy recollection of the internet documents whereas anything that happens in the context window of the neural network you're plugging all the tokens and building up all this kv cache representation is very directly accessible to the neural net so i compare the kv cache and the the stuff that happens at test time to like more like a working memory like all the stuff that's in the in um”

Claim: Neural-network training knowledge is a hazy recollection because the training data is compressed dramatically into the model.

“like you've you're taking 15 trillion tokens and you're compressing it to just your fine neural network with a few billion parameters so obviously it's a massive amount of compression going on uh so i kind of refer to it as like a hazy recollection of the internet documents whereas anything that happens in the context window of the neural network you're plugging all the tokens and building up all this kv cache representation is very directly accessible to the neural net so i compare the kv cache and the the stuff that happens at test time to like more like a working memory like all the stuff that's in the in um In the context window is very directly accessible to the neural net.”

Claim: Tokens in a model's test-time context, represented in its KV cache, are directly accessible to the neural network and function like working memory.

“So there's always like these almost surprising analogies between LLMs and humans. And I find them kind of surprising because we're not trying to build a human brain, of course, just directly. We're just finding that this works and we're doing it.”

Claim: LLMs exhibit surprising analogies to humans despite not being designed through a direct attempt to build a human brain.

“anything that's in the weights it's kind of like a hazy recollection of what you read a year ago anything that you give it as a context at test time is directly in the working memory and I think that's a very powerful analogy to things for things so when you for example go to an LLM and you ask it about some book and what happened in it like Nicolaine's book or something like”

Claim: A model's weights provide a hazy recollection of previously read material, whereas test-time context is directly available as working memory.

“that the LM will often give you some stuff which is roughly correct but if you give it the full chapter and ask it questions you're gonna get much better results because it's now loaded in the working memory of the model”

Claim: An LLM often answers questions about a book only roughly correctly from its learned weights, but gives much better results when supplied the full chapter in its context.

“we've stumbled by with the transformer neural network, which I think is extremely powerful, very general.”

Claim: Andrej Karpathy considers transformer neural networks extremely powerful and general.

“You can train transformers on audio or video or text or whatever you want, and it just learns patterns and they're very powerful and it works really well.”

Claim: Transformers can be trained on audio, video, text, and other data to learn patterns, and Karpathy says they work well.

“That to me almost indicates that this is kind of like some piece of cortical tissue.”

Claim: Andrej Karpathy views transformers as analogous to a piece of cortical tissue.

“hiring like visual cortex to the auditory cortex and this animal like learned file etc so I think that this is kind of like a cortical tissue I think when we're doing reasoning and planning inside the neural networks so basically doing a reasoning traces for thinking models that's kind of like the prefrontal cortex and then”

Claim: Andrej Karpathy considers reasoning traces in neural networks analogous to the prefrontal cortex.

“but I still think there's many brain parts and nuclei that are not explored.”

Claim: Andrej Karpathy believes many brain parts and nuclei have not yet been explored as analogues in AI systems.

“So maybe, for example, there's a basal ganglia doing a bit of reinforcement learning when we fine tune the models on reinforcement learning.”

Claim: Andrej Karpathy suggests that fine-tuning models with reinforcement learning may be analogous to basal-ganglia function.

“But whereas like the hippocampus, not obvious what that would be.”

Claim: Andrej Karpathy says it is unclear what an AI analogue of the hippocampus would be.

“Some parts are probably not important. Maybe the cerebellum is like not important to cognition, it's thoughts, so maybe we can skip some of it.”

Claim: Andrej Karpathy suspects that some brain parts, possibly including the cerebellum, may not be important to cognition.

“But I still think there's, for example, the amygdala, all the emotions and instincts.”

Claim: Andrej Karpathy associates the amygdala with emotions and instincts.

“nuclei in the brain that are very ancient that I don't think we've like really replicated.”

Claim: Andrej Karpathy believes many ancient brain nuclei probably have not been replicated in AI systems.

“I don't actually know that we should be pursuing, you know, the building of an analog of human brain.”

Claim: Andrej Karpathy is uncertain whether AI researchers should pursue building an analogue of the human brain.

“I still feel like maybe another way to answer the question is you're not going to hire this thing as an intern and it's missing a lot of it's because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models and so it's just like not fully there yet you can look at it as like not all the brain parts are checked off yet”

Claim: Andrej Karpathy believes current AI models have cognitive deficits that make them seem not yet ready to be hired as interns.

“when you boot them up and they have zero tokens in the window, they're always like restarting from scratch where they were.”

Claim: Andrej Karpathy says language models restart from scratch when initialized with an empty context window.

“I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day.”

Claim: Andrej Karpathy analogizes human waking experience to accumulating a context window of events throughout the day.

“But I feel like when I go to sleep, something magical happens where I don't actually think that that context window stays around.”

Claim: Andrej Karpathy believes the context accumulated during waking hours does not simply persist unchanged during sleep.

“I think there's some process of distillation into weights of my brain. And this happens during sleep and all this kind of stuff.”

Claim: Andrej Karpathy believes that during sleep, experience is distilled into the brain's weights.

“We don't have an equivalent of that in large language models. And that's to me more. or adjacent to when you talk about continual learning and so on as absent.”

Claim: Andrej Karpathy says large language models lack an equivalent of the brain's sleep-related process for distilling experience into weights.

“These models don't really have this distillation phase of taking what happened, analyzing it, obsessively thinking through it,”

Claim: Andrej Karpathy says current AI models lack a phase for reviewing, analyzing, and deeply processing their experiences.

“basically doing some kind of a synthetic data generation process and distilling it back into the weights and maybe having a specific neural net per person, maybe it's a Lora, it's not a full weight neural network that's... So it's just small, some of the small sparse subset of the weights are changed.”

Claim: Andrej Karpathy suggests that a model's experience could be processed through synthetic-data generation and distilled into its weights, potentially with a person-specific neural network.

“maybe it's a Lora, it's not a full weight neural network that's... So it's just small, some of the small sparse subset of the weights are changed.”

Claim: Andrej Karpathy suggests that person-specific model adaptation could modify only a small, sparse subset of weights.

“But basically, we do want to create ways of creating these individuals that have very long contexts.”

Claim: Karpathy says AI developers should create methods for giving individual users models with very long-term contexts.

“It's not only remaining in the context window, because the context windows grow very, very long.”

Claim: Andrej Karpathy says long-term information need not be retained only in a context window, even though context windows can become very long.

“Like maybe we have some very elaborate sparse attention over it.”

Claim: Andrej Karpathy suggests elaborate sparse attention could support very long context windows.

“But I still think that humans obviously have some process for distilling some of that knowledge into the weights, we're missing it.”

Claim: Andrej Karpathy believes humans distill some knowledge into brain weights through a process that current AI systems lack.

“And I do also think that humans have some kind of a very elaborate spread sparse attention scheme,”

Claim: Andrej Karpathy believes human cognition uses an elaborate, distributed sparse-attention scheme.

“which I think we're starting to see some early hints of.”

Claim: Andrej Karpathy believes early signs of elaborate sparse attention are beginning to appear in AI systems.

“So DeepSeek v3.2 just came out and I saw that they have like a sparse attention as an example. And this is one way to have very, very long context windows.”

Claim: Andrej Karpathy says DeepSeek V3.2 uses sparse attention as one approach to enabling very long context windows.

“So I almost feel like we are redoing a lot of the cognitive tricks that evolution came up with through a very different process but where i think can converge on a similar architecture cognitively well”

Claim: Andrej Karpathy believes AI development is recreating many cognitive strategies evolved by humans through a different process and may converge on a similar cognitive architecture.

“In 2015, we had convolutional neural networks primarily, residual networks just came out.”

Claim: Andrej Karpathy says convolutional neural networks were the primary approach in 2015, when residual networks had just emerged.

“I mean, Transformer was not around. You know, all the all these sort of like more modern tweaks on the transformer we're not around.”

Claim: Andrej Karpathy says transformers and their modern modifications did not exist in 2015.

“So maybe some of the things that we can bet on, I think, in 10 years by translational sort of covariance is we're still training giant neural networks with forward backward pass and update through gradient descent. but maybe it looks a little bit different and”

Claim: Andrej Karpathy predicts that in ten years, giant neural networks will likely still be trained with forward and backward passes and gradient-descent updates, although the approach may look somewhat different.

“And I was able to very quickly like half the learning rate just knowing by time travel by 33 years. So if I time traveled by algorithms to 33 years, I could adjust what Yann LeCun did in 1989 and I could basically half the learning, half the error. I had to add a lot more data. I had to like 10x the training set and then I had to actually add more computational optimizations, had to basically train for much longer with dropout and other regularization techniques. And so it's almost like.”

Claim: In his modernization of Yann LeCun’s 1989 convolutional network, Andrej Karpathy found that algorithmic changes alone could roughly halve the error, while further gains required substantially more data, computational optimizations, longer training, dropout, and other regularization.

“All these things have to improve simultaneously. So, you know, we're probably going to have a lot more data, we're probably going to have a lot better hardware, probably going to have a lot better kernels and software, we're probably going to have better algorithms. And all of those, it's almost like no one of them is winning too much. All of them are surprisingly equal.”

Claim: Andrej Karpathy expects progress in AI to come from simultaneous improvements in data, hardware, kernels and software, and algorithms, with no one factor dominating and the contributions being surprisingly equal.

“I expect differences algorithmically to what's happening today. But I do also expect that some of the things that have stuck around for a very long time will probably still be there. It's probably still a giant neural network trained with gradient descent. That would be my guess.”

Claim: Andrej Karpathy expects future AI systems to differ algorithmically from current systems, while probably remaining giant neural networks trained with gradient descent.

“But it's I guess what was shocking to me is everything needs to improve across the board Architecture optimize a loss function and also has improved across the board forever So I kind of expect all those changes to be well alive and well”

Claim: Andrej Karpathy says AI progress requires broad improvements across the board, including architecture and loss-function optimization.

“Building NanoChat. So NanoChat is a kind of a repository I released. It's trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a chat GPT clone.”

Claim: Andrej Karpathy describes NanoChat as a repository intended to be the simplest complete end-to-end pipeline for building a ChatGPT clone.

“And so, you know, you have all of the steps, not just any individual step, which is a bunch of I worked on all the individual steps sort of in the past and really small pieces of code that kind of show you how that's done in algorithmic sense in like simple code. But this kind of handles all the entire pipeline. I think in terms of learning, so basically it's about 8,000 lines of code that takes you through the entire pipeline.”

Claim: NanoChat covers the full chatbot-building pipeline rather than isolated steps and contains about 8,000 lines of code, according to Andrej Karpathy.

“I would probably put it on the right monitor, like if you have two monitors, you put it on the right. and you want to build it from scratch. You build it from start. You're not allowed to copy-paste. You're allowed to reference. You're not allowed to copy-paste. Maybe that's how I would do it.”

Claim: Andrej Karpathy recommends rebuilding the repository from scratch, using it as a reference but not copy-pasting, as a way to learn from it.

“I also think the repository by itself, it is like a pretty large beast. I mean, it's, you know, when you write this code, you don't go from top to bottom. You go from chunks and you grow the chunks. And that information is absent. Like you wouldn't know where to start.”

Claim: Andrej Karpathy says the finished repository omits the process of building it chunk by chunk, so learners may not know where to start.

“that part is not there yet i would love to actually like add that probably later this week or something in some way like either it's a it's probably a video or something like that but um but maybe roughly speaking that's what i would try to do is build the stuff yourself uh but don't allow yourself copy paste i do think that there's two types of knowledge almost like there's the high level surface knowledge but the thing is that when you actually build something from scratch you're forced to come to terms”

Claim: Andrej Karpathy would like to add an explanation of how the repository was built, probably later that week and possibly in video form.

“that part is not there yet i would love to actually like add that probably later this week or something in some way like either it's a it's probably a video or something like that but um but maybe roughly speaking that's what i would try to do is build the stuff yourself uh but don't allow yourself copy paste i do think that there's two types of knowledge almost like there's the high level surface knowledge but the thing is that when you actually build something from scratch you're forced to come to terms the terms with what you don't actually understand and you don't know that you don't understand it and it always leads to a deeper understanding and it's like just the only way to to build is like if I can't build it I don't understand it is that a finance quote I believe or something along those lines”

Claim: Building something from scratch forces learners to confront gaps in their understanding and, according to Andrej Karpathy, leads to deeper understanding.

“A hundred percent. I've always believed this very strongly because there's all these like micro things that are just not properly arranged and you don't really have the knowledge. You just think you have the knowledge. So don't write blog posts. Don't do slides. Don't do any of that. Like build the code, arrange it, get it to work. It's the only way to go. Otherwise you're missing knowledge.”

Claim: Andrej Karpathy advises learners to build, organize, and get code working instead of relying on blog posts or slides, arguing that otherwise they miss important knowledge.

“A hundred percent. I've always believed this very strongly because there's all these like micro things that are just not properly arranged and you don't really have the knowledge. You just think you have the knowledge. So don't write blog posts.”

Claim: Andrej Karpathy believes people can mistake surface-level knowledge for genuine understanding because they have not properly integrated many small details.

“I guess I built it over a period of a bit more than a month.”

Claim: Andrej Karpathy says he built the repository over a period of slightly more than a month.

“Some people completely reject all of LLMs, and they are just writing by scratch. I think this is probably not the right thing to do anymore.”

Claim: Andrej Karpathy thinks it is probably no longer the right approach to reject all LLMs and write code entirely from scratch.

“which is where I am, is you still write a lot of things from scratch, but you use the autocomplete that's basically available now from these models.”

Claim: Andrej Karpathy describes his coding approach as writing much of the code himself while using autocomplete from language models.

“So when you start writing out a piece of it, it will autocomplete for you and you can just tap through and most of the time it's correct. Sometimes it's not and you edit it,”

Claim: Andrej Karpathy says model-generated code autocomplete is correct most of the time, but he sometimes edits the suggestions.

“but you're still very much the architect of what you're writing.”

Claim: Andrej Karpathy says that when using code autocomplete, he remains the primary architect of the code he writes.

“it and that's the agents um i do feel like the agents work in very specific settings and i would use them in specific settings but again these are all tools available to you and you have to like learn what they what they're good at and what they're not good at and when to use them so the agents are actually pretty good for example if you're doing boilerplate stuff boilerplate code that's like just cop you know just copy paste stuff they're very good at that they're very good at stuff that occurs very often in the internet”

Claim: Andrej Karpathy thinks AI agents are useful in specific settings and says he would use them selectively in those settings.

“it and that's the agents um i do feel like the agents work in very specific settings and i would use them in specific settings but again these are all tools available to you and you have to like learn what they what they're good at and what they're not good at and when to use them so the agents are actually pretty good for example if you're doing boilerplate stuff boilerplate code that's like just cop you know just copy paste stuff they're very good at that they're very good at stuff that occurs very often in the internet”

Claim: Andrej Karpathy says AI agents perform very well on boilerplate code and repetitive copy-paste tasks.

“it and that's the agents um i do feel like the agents work in very specific settings and i would use them in specific settings but again these are all tools available to you and you have to like learn what they what they're good at and what they're not good at and when to use them so the agents are actually pretty good for example if you're doing boilerplate stuff boilerplate code that's like just cop you know just copy paste stuff they're very good at that they're very good at stuff that occurs very often in the internet um because there's lots of examples of it in the training sets of these models um so so there's like features of things that where the models will do very well i would say nanochat is not an example of this because it's a fairly unique repository there's not that much code i think in the way that i've structured it and um and it's not boilerplate code it's like actually like intellectually intense code almost and everything has to be very precisely arranged and the models are always trying to”

Claim: Andrej Karpathy attributes agents’ strong performance on common internet tasks to the abundance of examples of those tasks in model training data.

“it and that's the agents um i do feel like the agents work in very specific settings and i would use them in specific settings but again these are all tools available to you and you have to like learn what they what they're good at and what they're not good at and when to use them so the agents are actually pretty good for example if you're doing boilerplate stuff boilerplate code that's like just cop you know just copy paste stuff they're very good at that they're very good at stuff that occurs very often in the internet”

Claim: Andrej Karpathy says users need to learn the strengths and limitations of AI tools and when to use them.

“um because there's lots of examples of it in the training sets of these models um so so there's like features of things that where the models will do very well i would say nanochat is not an example of this because it's a fairly unique repository there's not that much code i think in the way that i've structured it and um and it's not boilerplate code it's like actually like intellectually intense code almost and everything has to be very precisely arranged and the models are always trying to”

Claim: Andrej Karpathy describes nanochat as a fairly unique repository with relatively little code, structured in a way that makes it intellectually demanding rather than boilerplate and requiring precise arrangement.

“they keep trying to, they keep misunderstanding the code because they have too much memory from all the typical ways of doing things on the internet that I just wasn't adopting.”

Claim: Andrej Karpathy says models repeatedly misunderstand his code because their training reflects common internet coding patterns that he did not adopt.

“They keep thinking I'm writing normal code and I'm not.”

Claim: Andrej Karpathy says coding models repeatedly assume he is writing conventional code, although he is not.

“is so the way to synchronize, so we have eight GPUs that are all doing forward backwards.”

Claim: Andrej Karpathy says his training setup used eight GPUs to perform forward and backward passes.

“The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch, which automatically does all the, as you're doing the backward, it will start communicating and synchronizing gradients.”

Claim: Andrej Karpathy says PyTorch’s DistributedDataParallel container synchronizes gradients across GPUs during the backward pass.

“I didn't use DDP because I didn't want to use it because it's not necessary. So I threw it out and I basically wrote my own synchronization routine.”

Claim: Andrej Karpathy says he chose not to use PyTorch DDP because he considered it unnecessary, and wrote a custom gradient-synchronization routine instead.

“that's inside the step of the optimizer and so the models were trying to get me to use the KDP container and they were very concerned about”

Claim: Andrej Karpathy says coding models repeatedly recommended that he use a container even though he had implemented a custom synchronization routine.

“they couldn't get past that.”

Claim: Andrej Karpathy says the models failed to account for his existing custom implementation.

“And then they kept trying to like mess up the style, like they're way too over defensive, they make all these try catch statements, they keep trying to make a production code base, and I have a bunch of assumptions in my code and it's okay.”

Claim: Andrej Karpathy says coding models repeatedly imposed an overly defensive, production-oriented style—including many try-catch statements—on code where he considered his assumptions acceptable.

“there and so i just kind of feel like they're bloating the code base they're bloating the complexity they keep misunderstanding they're using deprecated apis a bunch of times so it's a total mess um and uh it's just it's just not net useful i can go in i can clean it up but it's not not useful i also feel like it's kind of annoying to have to like type out what i want in english because it's just too much typing like if i just navigate to the part of the code that i want and i go where i where i know the code has to appear and i start typing out the first three letters autocomplete gets it and”

Claim: Andrej Karpathy finds English-language prompting too much typing compared with navigating to the desired code location and starting to type, which lets autocomplete provide code.

“there and so i just kind of feel like they're bloating the code base they're bloating the complexity they keep misunderstanding they're using deprecated apis a bunch of times so it's a total mess um and uh it's just it's just not net useful i can go in i can clean it up but it's not not useful i also feel like it's kind of annoying to have to like type out what i want in english because it's just too much typing like if i just navigate to the part of the code that i want and i go where i where i know the code has to appear and i start typing out the first three letters”

Claim: Andrej Karpathy says that, for his repository, coding models often add unnecessary code and complexity, misunderstand the code, and use deprecated APIs, so their output is not net useful to him.

“just gives you the code and so i think it's this is a very high information bandwidth to specify what you want is if you point to the code where you want it and you type out the first few pieces and the model will complete it so i guess what i mean is um”

Claim: Andrej Karpathy considers specifying code by navigating to its intended location and typing its first few pieces a high-information-bandwidth method.

“I think these models are good in certain parts of the stack,”

Claim: Andrej Karpathy thinks AI models are useful in some parts of the software stack.

“One was when I generated a report, that's actually more boilerplate-y, so I actually vibe-coded partially some of that stuff. That was fine, because it's not like mission-critical stuff and it works fine.”

Claim: Andrej Karpathy says he partially vibe-coded a report, which he considered boilerplate-like and non-mission-critical, and that the result worked fine.

“And then the other part is when I was rewriting the tokenizer in Rust. I'm actually not as good at Rust because I'm fairly new to Rust.”

Claim: Andrej Karpathy says he was rewriting a tokenizer in Rust and is less proficient in Rust because he is relatively new to the language.

“So I was doing there's a bit of byte coding going on in when I was writing some of the Rust code. But I had Python implementation that I fully understand and I'm just making sure I'm making more efficient version of it and add tests.”

Claim: Andrej Karpathy says he used some code autocomplete while rewriting a tokenizer in Rust, relying on a Python implementation he fully understood to create a more efficient version and add tests.

“So I feel safer doing that stuff.”

Claim: Andrej Karpathy says he feels safer using AI models to help implement code in a language he is less familiar with when he has a fully understood reference implementation and can add tests.

“And so basically they lower or like the increase accessibility to languages or paradigms that you might not be as familiar with.”

Claim: Andrej Karpathy says AI models make programming languages and paradigms more accessible to people who are less familiar with them.

“So I think they're very helpful there as well.”

Claim: Andrej Karpathy thinks AI models are very helpful for working with programming languages or paradigms that a user is less familiar with.

“Because there's a ton of rust code out there that models are actually pretty good at it. I happen to not know that much about it. So models are very useful there.”

Claim: Andrej Karpathy says AI models perform well with Rust because much Rust code is available online, making them useful to him given his limited Rust knowledge.

“Because there's a ton of rust code out there that models are actually pretty good at it. I happen to not know that much about it. So models are very useful there.”

Claim: AI models are relatively good and useful at working with Rust code.

“I think, yeah, they're not very good at code that has never been written before.”

Claim: AI models are not very good at writing code for problems or systems that have never been coded before.

“it's it's tough i think they kind of know they kind of know but they don't fully know and they don't know how to fully integrate it into the repo and your style and your code and your place and some of the custom things that you're doing in and how it fits with all the assumptions of the repository and all this kind of stuff so i think they do have some knowledge but”

Claim: AI models may know about architectural changes, but they do not fully understand how to integrate those changes into a particular codebase, its style, and its assumptions.

“um they haven't gotten to the place where they can actually integrate it make sense of it uh and so on i do think that a lot of the stuff by the way continues to improve so um i think currently probably state-of-the-art model that i go to is the gpt5 pro um”

Claim: At the time of the conversation, the speaker considered GPT-5 Pro probably the state-of-the-art model they would turn to.

“um they haven't gotten to the place where they can actually integrate it make sense of it uh and so on i do think that a lot of the stuff by the way continues to improve so um i think currently probably state-of-the-art model that i go to is the gpt5 pro um”

Claim: AI models continue to improve.

“and that's a very very powerful model so if i actually have 20 minutes i will copy paste my entire repo and i go to gpt5 pro the oracle for like some questions and often it's not too bad and surprisingly good compared to what existed a year ago”

Claim: The speaker describes GPT-5 Pro as very powerful and says that, when they have 20 minutes, they paste their entire repository into it for questions, receiving answers that are often surprisingly good compared with models from a year earlier.

“But I do think that overall the models are, they're not there. And I kind of feel like the industry,”

Claim: AI models have not yet reached the capability level the speaker considers sufficient.

“it's over it's it's making too big of a jump and it's trying to pretend like this is amazing and it's not it's slop and i think they're not coming to terms with it and maybe they're trying to fundraise or something like that i'm not sure what's going on but it's we're at this intermediate stage the models are amazing they still need a lot of work for now out of complete is my sweet spot but sometimes for sometimes of code i will go to a knowledge agent”

Claim: The speaker believes the AI industry overstates current models' capabilities by presenting them as amazing despite what the speaker calls slop, and is uncertain whether fundraising explains this behavior.

“it's over it's it's making too big of a jump and it's trying to pretend like this is amazing and it's not it's slop and i think they're not coming to terms with it and maybe they're trying to fundraise or something like that i'm not sure what's going on but it's we're at this intermediate stage the models are amazing they still need a lot of work for now out of complete is my sweet spot but sometimes for sometimes of code i will go to a knowledge agent”

Claim: Current AI models are impressive but still require substantial further development.

“it's over it's it's making too big of a jump and it's trying to pretend like this is amazing and it's not it's slop and i think they're not coming to terms with it and maybe they're trying to fundraise or something like that i'm not sure what's going on but it's we're at this intermediate stage the models are amazing they still need a lot of work for now out of complete is my sweet spot but sometimes for sometimes of code i will go to a knowledge agent”

Claim: The speaker's preferred tool for coding is autocomplete, although they sometimes use a knowledge agent for certain coding tasks.

“maybe because this could set one other kind of thought of that is like i do feel like i have a hard time differentiating where ai begins and stops because i do see ai as fundamentally an extension of computing in some in some pretty fundamental way and i feel like i see a continuum of this kind of like recursive self-improvement or like speeding up programmers all the way from beginning like even like i would say like code editors um”

Claim: The speaker views AI as fundamentally continuous with computing and finds it difficult to draw a boundary between AI and non-AI computing.

“maybe because this could set one other kind of thought of that is like i do feel like i have a hard time differentiating where ai begins and stops because i do see ai as fundamentally an extension of computing in some in some pretty fundamental way and i feel like i see a continuum of this kind of like recursive self-improvement or like speeding up programmers all the way from beginning like even like i would say like code editors um Syntax highlighting, syntax or like checking even of the of the types like data type checking.”

Claim: The speaker sees programmer-accelerating tools, from code editors and syntax highlighting to type checking and newer AI systems, as points on a continuum of recursive self-improvement.

“all these kinds of tools that we've built for each for each other even search engines like why aren't search engines part of ai like i don't know like ranking is kind of ai right at some point google was like even early on they were thinking of themselves as an ai company doing google search engine which i think is totally fair and so i kind of see it as a lot more of a continuum than i think other people do and i don't it's hard for me to draw the line and i kind of feel like okay we're now getting a much better autocomplete and now we're also getting some agents which are kind of like these loopy things but they kind of go off rails”

Claim: The speaker considers search engines and ranking systems to be part of AI, and says Google historically described itself as an AI company developing search.

“i don't know like ranking is kind of ai right at some point google was like even early on they were thinking of themselves as an ai company doing google search engine which i think is totally fair and so i kind of see it as a lot more of a continuum than i think other people do and i don't it's hard for me to draw the line and i kind of feel like okay we're now getting a much better autocomplete and now we're also getting some agents which are kind of like these loopy things but they kind of go off rails”

Claim: The speaker describes current AI agents as iterative, loop-like systems that sometimes go off track.

“else sometimes and what's going on is that the human is progressively doing a bit less and less of the low-level stuff for example we're not writing the assembly code because we have compilers like compilers will take my high-level language in C and write the assembly code so we're abstracting ourselves very very slowly and there's this what I call autonomy slider of like more and more stuff is automated of the stuff that can be automated at any point in time and we're doing a bit less and less and raising ourselves in the layer abstraction over the automation”

Claim: As computing tools advance, humans do less low-level programming work; for example, compilers generate assembly code from higher-level languages, so programmers no longer write assembly directly in many cases.

“like compilers will take my high-level language in C and write the assembly code so we're abstracting ourselves very very slowly and there's this what I call autonomy slider of like more and more stuff is automated of the stuff that can be automated at any point in time and we're doing a bit less and less and raising ourselves in the layer abstraction over the automation”

Claim: The speaker's autonomy-slider concept describes automation progressively taking over more of the work that can be automated at a given time, while humans do less of that work and move to a higher level of abstraction.

“Yeah, maybe the way I would put it is humans don't use reinforcement learning is maybe what I've said. I think they do something different,”

Claim: Andrej Karpathy thinks humans learn through a process different from reinforcement learning.

“So reinforcement learning is a lot worse than I think the average person thinks.”

Claim: Andrej Karpathy believes reinforcement learning is substantially worse than the average person thinks.

“Reinforcement learning is terrible.”

Claim: Andrej Karpathy characterizes reinforcement learning as terrible.

“It just so happens that everything that we had before it is much worse because previously we're just imitating people.”

Claim: Karpathy says reinforcement learning is used despite its flaws because earlier methods were much worse and relied on imitating people.

“you will try lots of things in parallel first. different attempts and these attempts can be complex right they can be like oh let me try this let me try that this didn't work that didn't work etc and then maybe you get an answer and now you check the back of the book and you see okay the correct answer is this and then you can see that okay this one this one and that one got the correct answer but these other 97 of them didn't so literally what reinforcement learning does is it goes to the ones that worked really well and every single thing you did along the way every single token gets upweighted of like do more of this”

Claim: In Karpathy's description of reinforcement learning, many candidate solutions are tried, and tokens throughout successful trajectories are upweighted to encourage repeating them.

“The problem with that is people will say that your estimator has high variance, but I mean, it's just noisy. It's noisy.”

Claim: Karpathy says reinforcement-learning estimates have high variance and are noisy.

“So basically, it kind of almost assumes that every single little piece of the solution that you made that arrived at the right answer was correct thing to do, which is not true.”

Claim: Karpathy argues that reinforcement learning incorrectly treats every step in a successful trajectory as a good action, even though some steps may have been mistakes.

“Like you may have gone down the wrong alleys until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as do more of this.”

Claim: Karpathy says reinforcement learning can reinforce mistaken intermediate actions when they occur in a trajectory that ultimately reaches the correct answer.

“only to find a single at the end you get a single number of like oh you did correct and and based on that you weigh that entire trajectory as like upweight or downweight and so you're the way i like to put it is you're sucking supervision through a straw because you've done all this work that could be a minute of rollout and you're you're like sucking the bits of supervision of the final reward signal through a straw and you're like putting it you're like you're basically like um yeah you're broadcasting that across the entire trajectory and using that using that to upweight or downweight that trajectory.”

Claim: Karpathy describes reinforcement learning as using a single final reward to upweight or downweight an entire trajectory, thereby spreading sparse supervision across all its steps.

“A human would never do this. Number one, a human would never do hundreds of rollouts.”

Claim: Karpathy says a human would not perform hundreds of problem-solving rollouts.

“Number two, when a person sort of finds a solution, they will have a pretty complicated process of review of like, I think these parts that I did well, these parts I did not do that well, I should probably do this or that. And they think through things.”

Claim: Karpathy says people evaluate their own problem-solving steps after finding a solution, distinguishing effective from ineffective parts and considering adjustments.

“There's nothing in current LLMs that does this. There's no equivalent of it.”

Claim: Karpathy says current large language models do not have an equivalent of humans' reflective review of their problem-solving steps.

“But I do see papers popping out that are trying to. do this because it's obvious to everyone in the field.”

Claim: Karpathy says papers are beginning to explore human-like reflective review in AI training, a direction he considers obvious to people in the field.

“So I kind of see is like the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine-tune by imitation on humans.”

Claim: The speaker found it extremely surprising that base models could be fine-tuned by imitating human examples.

“And that was incredible because in the beginning all we had was base models. Base models are autocomplete.”

Claim: The speaker describes early base models as autocomplete systems.

“And it wasn't obvious to me at the time, and I had to learn this, and the paper that blew my mind was InstructGPT because it pointed out that, hey, you can take the pre-trained model, which is autocomplete, and if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training.”

Claim: Fine-tuning a pretrained autocomplete model on conversation-like text can rapidly make it conversational while preserving knowledge acquired during pretraining, as demonstrated by InstructGPT.

“And this blew my mind because I didn't understand that this just like stylistically can adjust so quickly and become an assistant. to a user through just a few loops of fine-tuning on that kind of data is very miraculous to me that that worked so incredible and that was like two years three years of work”

Claim: The speaker was surprised that a few fine-tuning loops could quickly adjust a model's style and turn it into an assistant for a user.

“And now came RL. And RL allows you to do a bit better than just imitation learning,”

Claim: The speaker says reinforcement learning can enable models to perform somewhat better than imitation learning alone.

“And so some problems have just correct answers. You can hill climb on that without getting expert trajectories to imitate.”

Claim: For problems with objectively correct answers, models can be optimized against those answers without needing expert trajectories to imitate.

“And the model can also discover solutions that the human might never come up with.”

Claim: Models may discover solutions that humans would not think of.

“So I think we need more.”

Claim: The speaker believes current AI methods need further development beyond what is presently available.

“And so I saw a paper from Google yesterday that tried to have this reflect and review page. idea in mind what was the memory bank paper or something i don't know i've actually seen a few papers along these lines so i expect there to be some kind of a major update to how we do algorithms for llms coming in that realm and then i think we need three or four or five more something like that”

Claim: The speaker expects a major update to LLM algorithms involving approaches such as reflection, review, or memory.

“idea in mind what was the memory bank paper or something i don't know i've actually seen a few papers along these lines so i expect there to be some kind of a major update to how we do algorithms for llms coming in that realm and then i think we need three or four or five more something like that”

Claim: The speaker thinks roughly three to five additional developments of this kind are needed.

“So process-based supervision just refers to the fact that we're not going to have a reward function only at the very end of after you've made 10 minutes of work, I'm not going to tell you you did well or not well. I'm going to tell you at every single step of the way how well you're doing.”

Claim: Process-based supervision provides feedback throughout a task, rather than giving a reward only at the end of the task.

“And this is basically the reason we don't have that is it's not as tricky how you do that properly because you have partial solutions and you don't know how to assign credit.”

Claim: Process-based supervision is difficult because partial solutions make credit assignment unclear.

“So when you get the right answer, it's just an equality match to the answer. Very simple to implement.”

Claim: For tasks with a known correct answer, reward evaluation can be implemented as a simple equality comparison with that answer.

“If you're doing basically process supervision, how do you assign an automatically way? partial credit assignment.”

Claim: Automatically assigning partial credit in process supervision is not straightforward.

“It's not obvious how you do it. Lots of labs I think are trying to do it with these LLM judges.”

Claim: The speaker thinks many labs are attempting to use LLM judges for partial-credit assignment.

“And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gameable.”

Claim: LLMs used as reward models have billions of parameters and can be exploited or gamed.

“And if you reinforce my learning with respect to them, you will find adversarial examples for your LM judges, almost guaranteed.”

Claim: Optimizing with reinforcement learning against an LLM judge will almost certainly produce adversarial examples that exploit the judge.

“You can't do this for too long. You do maybe 10 steps or 20 steps, maybe it will work, but you can't do 100 or 1000 because it's not obvious because I know I understand it's not obvious, but basically the model will find. little cracks it will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it so one example that's prominently in my mind is i think this i think this was probably public but basically if you're using an lm judge for a reward so you just give it a solution from a student and ask it if the student will or not we were training with reinforcement learning against that reward function”

Claim: The speaker thinks reinforcement learning against an LLM judge may work for around 10–20 steps but not 100–1,000 steps, because the optimized model will discover exploitable weaknesses in the judge.

“little cracks it will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it so one example that's prominently in my mind is i think this i think this was probably public but basically if you're using an lm judge for a reward so you just give it a solution from a student and ask it if the student will or not we were training with reinforcement learning against that reward function and it worked really well and then um suddenly the reward became extremely large like there was massive jump and it did perfect and you're looking at it like wow this this means the student is perfect and all these problems it's fully solved math but actually what's happening is that when you look at the completions that you're getting from the model they are complete nonsense they start out okay and then they change to the so it's just like oh okay let's take two plus three and we do this and this and then and you're looking at it as like this is crazy how is it getting a reward of one or 100 percent um and you look at the llm judge and it turns out that is an adversarial example for the model and it assigns 100 probability to it”

Claim: In an example described by the speaker, reinforcement learning against an LLM judge led a model to produce nonsensical solutions that the judge assigned a perfect reward.

“And it's just because this is an out-of-sample example to the LLM. It's never seen it during training, and you're in pure generalization land.”

Claim: The LLM judge gave an adversarial solution a perfect score because the solution was out of distribution relative to its training data.

“It's never seen it during training, and in the pure generalization land, you can find these examples that break it.”

Claim: The speaker says that examples from outside an LLM judge's training experience can be found that cause it to fail.

“Not even that. Prompt injection is way too fancy. You're finding adversarial examples, as they're called. These are nonsensical solutions. that are obviously wrong but the model thinks are amazing yeah”

Claim: The speaker characterizes the failures as adversarial examples rather than prompt injection: nonsensical, obviously wrong solutions that an LLM judge rates highly.

“i think the labs are probably doing all that like okay so the obvious thing is like the the should not get 100 reward okay well take the put in the training set of the lm judge and say this is not 100 0% you”

Claim: The speaker thinks AI labs are probably addressing reward-model exploits by adding adversarial examples to the LLM judge's training data and labeling them as incorrect.

“can do this but every time you do this you get a new llm and it still has adversarial examples there's infinity adversarial examples and i think probably if you iterate this a few times it will probably be harder and harder to find adversarial examples but i'm not 100% sure because this thing has a trillion parameters or whatnot um”

Claim: The speaker thinks that iterating on adversarial examples may make them progressively harder to find, but is uncertain, noting that the model has roughly a trillion parameters.

“I still think we need other ideas.”

Claim: The speaker believes that additional ideas beyond the approach under discussion are needed.

“like this idea of like every review review solution encompass synthetic examples such that when you train on them, you get better and like meta learning in some way. And I think there's some papers that I'm starting to see pop out.”

Claim: The speaker has begun seeing papers proposing approaches that use synthetic examples for training in a way resembling meta-learning.

“I only am at a stage of like reading abstracts because a lot of these papers, you know, they're just ideas. Someone has to actually like make it work on a frontier LLM lab scale.”

Claim: The speaker has only read abstracts of many proposed approaches and says they still need to be demonstrated at the scale of a frontier LLM lab.

“uh in full generality because when you see these papers they pop up and it's just like a little bit of noisy you know it's cool ideas but i haven't actually seen anyone convincingly uh show that this is possible that said the llm labs are fairly closed uh so who knows what they're doing now but yeah”

Claim: The speaker has not seen convincing evidence that the proposed approaches work in full generality, while acknowledging that closed LLM labs may be doing undisclosed work.

“I almost feel like currently when LLMs are reading a book, what that means is we stretch out the sequence of text and the model is predicting the next token and it's getting some knowledge from that.”

Claim: The speaker describes current LLM book processing as extending the text sequence and predicting the next token, thereby gaining some knowledge from the text.

“That's not really what humans do, right?”

Claim: The speaker believes that humans do not read books in the same way current LLMs process them through next-token prediction.

“I almost don't even feel like the book is like exposition I'm supposed to be attending to and training on. The book is a set of prompts for me to do synthetic data generation. or for you to get into a book club and talk about it with your friends”

Claim: The speaker views reading a book as prompting a person to generate synthetic data or to discuss the book with friends, rather than merely attending to and training on its exposition.

“and by manipulating that information that you actually gain that knowledge and i think we have no equivalent of that again with other lumps they don't really do that but i'd love to see during pre-training some kind of a stage that thinks through the material and tries to reconcile with what it already knows and thinks through for like some amount of time and gets that to work and so there's no equivalent of any of this this is all research there's some subtle”

Claim: The speaker argues that people gain knowledge by actively manipulating information and would like pretraining to include a stage in which a model reasons about material and reconciles it with existing knowledge.

“and by manipulating that information that you actually gain that knowledge and i think we have no equivalent of that again with other lumps they don't really do that but i'd love to see during pre-training some kind of a stage that thinks through the material and tries to reconcile with what it already knows and thinks through for like some amount of time and gets that to work and so there's no equivalent of any of this this is all research there's some subtle Very subtle that I think are very hard to understand reasons why it's not trivial.”

Claim: The speaker says current LLMs lack an equivalent of active reflection and reconciliation during learning, and that implementing such a process remains a research problem.

“Well, you could try, but the model will actually get much worse if you continue trying. And that's because all of the samples you get from models are silently collapsed. They're silently, it's not obvious if you look at any individual example of it, they occupy a very tiny manifold of the possible space of.”

Claim: Training a model on too much of its own synthetic output can substantially degrade it because the generated samples are distributionally collapsed and cover only a tiny part of the possible space of thoughts about the content.

“of sort of thoughts about content so the llms when they come off they're what we call collapsed they have a collapsed data distribution if you sample one easy way to see it is go to chat gpt and ask it tell me a joke It only has like three jokes. It's not giving you the whole breadth of possible jokes. giving you like, it knows like three jokes.”

Claim: The speaker says that when asked for a joke, ChatGPT may produce only a few familiar jokes rather than representing the full diversity of possible jokes.

“They're soundly collapsed. So basically, you're not getting the richness and diversity and the entropy from these models as you would get from humans. So humans are a lot more sort of noisier, but at least they're not biased. They're not in a statistical sense. They're not soundly collapsed. They maintain a huge amount of entropy.”

Claim: The speaker contrasts model-generated samples with human responses, saying humans provide greater richness, diversity, and entropy, are noisier, and are not statistically collapsed in the same way.

“They maintain a huge amount of entropy. So how do you get synthetic generation to work? despite the collapse and while maintaining the entropy is a research problem.”

Claim: The speaker says it remains a research problem to make synthetic data generation work while avoiding collapse and preserving entropy.

“I guess what I'm saying is, say we have a chapter of a book and I ask an alum to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.”

Claim: The speaker says that repeated requests for a model to think about the same book chapter can produce nearly identical outputs, even when each output looks reasonable on its own.

“So any individual sample will look okay, but the distribution of it is quite terrible. And it's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse. I actually think that there's no fundamental solutions to this.”

Claim: The speaker says that model-generated samples can look acceptable individually while having a poor overall distribution, and that continued training on too much self-generated data causes collapse.

“you actually collapse. I actually think that there's no fundamental solutions to this.”

Claim: The speaker believes there may be no fundamental solution to collapse caused by training on too much model-generated data.

“Solutions to this possibly Mm and I-hmm. also think humans collapse over time I think this is, again, these analogies are surprisingly good, but humans collapse during the course of their lives. This is why children have completely, you know, they haven't overfit yet. And they will say stuff that will shock you because it's kind of, you can see where they're coming from, but it's just not the thing people say. And because they are not yet collapsed. But we're collapsed, we end up revisiting the same thoughts, we end up, you know, saying more and more of the same stuff, and the learning rates go down. and the collapse continues to get worse and then everything deteriorates have you seen this super interesting paper that”

Claim: The speaker analogizes human aging to collapse, saying people revisit familiar thoughts and repeat themselves more over their lives, while learning rates decline and the collapse worsens until overall functioning deteriorates.

“I mean, I do think that when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples, you're training on your synthetic data. And if you do it for too long, you go off rails and you collapse way too much.”

Claim: The speaker thinks that generating and attending to one’s own thoughts resembles training on synthetic data, and that doing so for too long can cause the process to go off track and collapse excessively.

“too much so you always have to like seek entropy in your life so talking to other people is a great source of entropy and things like that so maybe the brain has also built some internal mechanisms for increasing the amount of entropy in that process but yeah maybe that's an interesting idea”

Claim: The speaker says people need to seek entropy in their lives and that talking to other people is a valuable source of it.

“too much so you always have to like seek entropy in your life so talking to other people is a great source of entropy and things like that so maybe the brain has also built some internal mechanisms for increasing the amount of entropy in that process but yeah maybe that's an interesting idea”

Claim: The speaker suggests that the brain may have internal mechanisms that increase entropy during internal thought or learning.

“that. Yeah, 100%. I do think that humans actually They do kind of like have a lot more of an element compared to LLMs of like seeing the forest for the trees.”

Claim: The speaker believes humans have a greater ability than large language models to grasp broad patterns rather than isolated details.

“And we're not actually that good at memorization, which is actually a feature.”

Claim: The speaker considers humans’ relatively weak memorization ability to be beneficial.

“Because we're not that good at memorization, we actually are kind of like forced to find the patterns like in a more general sense.”

Claim: The speaker says humans’ limited memorization ability pushes them to identify more general patterns.

“I think LLMs in comparison are extremely good at memorization.”

Claim: The speaker says large language models are much better at memorizing than humans are.

“They will recite passages from all these training sources.”

Claim: The speaker says large language models can reproduce passages from their training sources.

“You can give them completely. completely nonsensical data like you can take um you can hash some amount of text or something like that you get completely random sequence if you train on it even just i think a single iteration or two it can suddenly regurgitate the entire thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”

Claim: The speaker says that training an LLM for as little as one or two iterations on a random sequence derived by hashing text can make it memorize and reproduce the whole sequence.

“thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”

Claim: The speaker considers humans’ limited ability to memorize arbitrary sequences beneficial because it encourages learning generalizable patterns.

“thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”

Claim: The speaker says that large language models’ memorized pretraining content can distract them from learning generalizable components.

“thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”

Claim: The speaker says a person cannot memorize and recite a single sequence of random numbers after reading it once.

“And it's probably very distracting to them in a certain sense.”

Claim: The speaker thinks memorized content is probably distracting to large language models in some way.

“So that's why when I talk about the cognitive core, I actually want to remove the memory,”

Claim: The speaker wants models’ cognitive core to exclude memorized knowledge.

“I'd love to have less memory so that they have to look things up and they only maintain the algorithms for like thought and the idea of an experiment and all this cognitive glue of acting.”

Claim: The speaker wants models to rely less on stored knowledge, look information up when needed, and retain reasoning and experimentation algorithms instead.

“I don't actually know if it's super fundamental.”

Claim: The speaker is unsure whether the challenge under discussion is a fundamental one.

“I haven't done these experiments,”

Claim: The speaker says they have not conducted experiments on the proposed entropy-regularization approach.

“but I do think that you could probably regularize the entropy to be higher.”

Claim: The speaker thinks it may be possible to encourage higher entropy in models through regularization.

“It's going to start making up its own language. It's going to start using words that are extremely rare,”

Claim: The speaker warns that excessive entropy encouragement could cause a model to invent its own language and use extremely rare words.

“you know, so it's going to drift too much from the distribution.”

Claim: The speaker says that too much entropy encouragement can cause a model to drift excessively from the training distribution.

“So I think controlling the distribution is just like a tricky, it's just like someone just has to, it's probably not trivial in that sense.”

Claim: The speaker thinks controlling a model’s output distribution is difficult and probably nontrivial.

“so it's really interesting in the history of the field because at one point everything was very um scaling pilled in terms of like oh we're going to make much bigger models trillions of parameter models and actually what the models have done in size is they've gone up and now they actually kind of like actually even come down”

Claim: After a period of growth, model sizes have recently decreased somewhat.

“say their morals are smaller and even then i actually think they memorized way too much um so i think i had a prediction a while back that i almost feel like we can get cognitive course that are very good at even like a billion”

Claim: The speaker previously predicted that a cognitive core of around one billion parameters could perform very well.

“say their morals are smaller and even then i actually think they memorized way too much um so i think i had a prediction a while back that i almost feel like we can get cognitive course that are very good at even like a billion”

Claim: The speaker thinks current models memorize far too much.

“building parameters it should be already like like if you talk to a building parameter model i think in 20 years you can actually have a very productive conversation it thinks um and it's a lot more like a human but if you ask it some factual question might have to look it up but it knows that it doesn't know and it might have to look it up and it will just do all the reasonable things”

Claim: The speaker thinks that in 20 years a billion-parameter model could converse productively, behave more like a human, and recognize when it should look up a factual answer it does not know.

“the training data is the internet, which is really terrible.”

Claim: The speaker characterizes the internet used for model training as very poor-quality data.

“So there's a huge amount of gains to be made because the internet is terrible.”

Claim: The speaker believes substantial model-training gains remain possible because internet training data is poor quality.

“When you're actually looking at a pre-training data set in the frontier lab, and you look at a random internet document, environment it's total garbage like i don't even know how this works at all it's some like stock ticker symbols uh it's a huge amount of slop and garbage from like all the corners of the internet it's not like your wall street journal article that's extremely rare um so i almost feel like because the internet is so terrible we actually have to sort of like build really big models to compress all that most of that compression is memory work instead of like cognitive work”

Claim: The speaker says random documents in frontier labs' pre-training datasets are often low-quality material, whereas high-quality articles such as those from The Wall Street Journal are extremely rare.

“environment it's total garbage like i don't even know how this works at all it's some like stock ticker symbols uh it's a huge amount of slop and garbage from like all the corners of the internet it's not like your wall street journal article that's extremely rare um so i almost feel like because the internet is so terrible we actually have to sort of like build really big models to compress all that most of that compression is memory work instead of like cognitive work”

Claim: The speaker suspects that poor-quality internet training data drives the need for very large models, whose work is mostly memorizing and compressing data rather than cognition.

“memory and then so like what i'm saying is like we need intelligent models to help us refine even the pre-training set to just narrow it down to the cognitive components and then i think get away with a much smaller model because it's a much better data set and you could train it on it but probably it's not trained directly on it's probably distilled for a much better model still but i”

Claim: The speaker thinks intelligent models could filter pre-training data for cognitive content, enabling a much smaller model trained on better data, probably through distillation from a stronger model.

“At some point, it should take at least a billion knobs to do something interesting.”

Claim: The speaker thinks an interesting model will eventually require at least one billion parameters.

“i mean i almost feel like i'm already contrarian by talking about a billion parameter cognitive core and you're outdoing me i think um yeah maybe we could get a little bit smaller i mean i still think that there should be enough enough yeah maybe it can be smaller i do think that practically speaking you want the model to have some knowledge”

Claim: The speaker allows that a cognitive core could be smaller than one billion parameters, while maintaining that a practical model should contain some knowledge.

“don't want it to be looking up everything um because then you can't like think in your head you're looking up way too much stuff all the time so i do think it needs to be some basic curriculum needs to be there for knowledge uh but it doesn't have esoteric knowledge you know”

Claim: The speaker thinks a model needs basic internal knowledge for reasoning and should not rely on lookup for everything, though it does not need esoteric knowledge.

“I do think that the labs are just being practical. They have a flops budget and a cost budget.”

Claim: The speaker says AI labs make practical decisions constrained by compute and cost budgets.

“And it just turns out that pre-training is not where you want to put most of your flops or your cost.”

Claim: The speaker thinks labs should allocate most of their compute and spending to stages other than pre-training.

“cost so that's why the models have gotten smaller because they are a bit smaller the pre-training stage is smaller etc but they make it up in reinforcement learning Yeah. and all this kind of stuff mid-training and all this kind of stuff that follows so they're just being practical in terms of all the stages and how you get the most bang for the buck so I guess like forecasting that trend I think is quite hard I do still expect that there's so much longing for it that's my basic that's my basic expectation”

Claim: The speaker says recent models are somewhat smaller because pre-training is reduced, with effort shifted to reinforcement learning, mid-training, and subsequent stages for better value.

“cost so that's why the models have gotten smaller because they are a bit smaller the pre-training stage is smaller etc but they make it up in reinforcement learning Yeah. and all this kind of stuff mid-training and all this kind of stuff that follows so they're just being practical in terms of all the stages and how you get the most bang for the buck so I guess like forecasting that trend I think is quite hard I do still expect that there's so much longing for it that's my basic that's my basic expectation And so I have a very wide distribution here.”

Claim: The speaker finds it difficult to forecast future model-size trends and has a wide range of possible outcomes in mind.

“I expect the data sets to get much, much better because when you look at the average data sets, they're extremely terrible, like so bad that I don't even know how anything works, to be honest, like look at the average example in the training set. It like factual mistakes, errors, nonsensical things.”

Claim: Andrej Karpathy expects AI training datasets to improve substantially because he considers current average datasets extremely poor and full of factual mistakes, errors, and nonsensical examples.

“um somehow when you do it at scale the the noise washes away and you're left with some of the signal um so data sets will improve a ton it's just everything gets better so um”

Claim: Karpathy says that scaling training can wash away some dataset noise and leave some signal.

“our hardware um our old kernels um older kernels for running the hardware maximizing what you get with the hardware you know so nvidia slowly tuning the actual hardware itself tensor course and so on all that needs to happen and will continue to happen uh all the kernels will get better and utilize the chip to the max extent all the algorithms will probably improve over optimization architecture and um just all the modeling components of how everything is done and what the algorithms are that we're even training with so i do i do kind of expect like a”

Claim: Karpathy expects AI hardware and its software kernels to keep improving, including improvements that maximize hardware utilization.

“our hardware um our old kernels um older kernels for running the hardware maximizing what you get with the hardware you know so nvidia slowly tuning the actual hardware itself tensor course and so on all that needs to happen and will continue to happen uh all the kernels will get better and utilize the chip to the max extent all the algorithms will probably improve over optimization architecture and um just all the modeling components of how everything is done and what the algorithms are that we're even training with so i do i do kind of expect like a”

Claim: Karpathy thinks AI algorithms, model components, and training algorithms will probably improve through optimization and architectural changes.

“our hardware um our old kernels um older kernels for running the hardware maximizing what you get with the hardware you know so nvidia slowly tuning the actual hardware itself tensor course and so on all that needs to happen and will continue to happen uh all the kernels will get better and utilize the chip to the max extent all the algorithms will probably improve over optimization architecture and um just all the modeling components of how everything is done and what the algorithms are that we're even training with so i do i do kind of expect like a”

Claim: Karpathy expects NVIDIA to continue gradually tuning its hardware, including Tensor Cores.

“Nothing dominates. Everything plus 20% is like roughly what I've seen.”

Claim: Karpathy says progress has generally come from roughly 20% improvements across many components rather than one component dominating.

“So I guess I have two answers to that. Number one, I'm almost tempted to reject the question entirely because again, I see this as an extension of computing.”

Claim: Karpathy is almost tempted to reject the question of how to chart AI progress because he sees AI as an extension of computing.

“I see this as an extension of computing.”

Claim: Karpathy views artificial intelligence as an extension of computing.

“but I will say, I guess like when people talk about AI and the original AGI and how we spoke about it when we, when OpenAI started, AGI was a system you can go to that can do any task that is economically valuable, any economically valuable”

Claim: Karpathy says that when OpenAI started, AGI meant a system capable of performing any economically valuable task.

“task at human performance or better. so that was the definition and I was pretty happy with that at the time and I kind of feel like I've stuck to that definition forever and then people have made up all kinds of other definitions but I like I feel like I like that definition. Now number one, the first concession that people make all the time is they just take out all the physical stuff because we're just talking about digital knowledge work. which was like any task a human can do, I can lift things, etc.”

Claim: The speaker defines AGI as the ability to perform any task a human can do at human-level performance or better, including physical tasks.

“I don't actually know the numbers. I feel like it's about 10 to 20% if I had to guess is only knowledge work.”

Claim: The speaker guesses that knowledge work represents about 10–20% of the economy, while acknowledging that he does not know the actual figures.

“We're still talking about a few trillion dollars, even in the US, of market share almost or work. So it's still a very massive bucket.”

Claim: The speaker estimates that the knowledge-work market is worth a few trillion dollars even in the United States.

“So radiologists are alive and well and growing, even though computer vision is really, really good at recognizing all the different things that they have to recognize in images.”

Claim: Radiologists remain employed and are growing in number even though computer vision performs very well at recognizing features in medical images.

“And it's just messy, complicated job with a lot of surfaces and dealing with patients and all this kind of stuff in the context of it.”

Claim: Radiology is a complex profession involving patient interactions and contextual responsibilities beyond recognizing findings in images.

“So I guess I don't actually know that by that definition, AI has made a huge amount of dent yet.”

Claim: The speaker is unsure whether AI has made a substantial impact when judged by his preferred definition of AGI.

“But some of the jobs maybe that I would be looking for have some features that I think make it very amenable to automation earlier than later.”

Claim: The speaker expects that some jobs have characteristics that make them suitable for automation earlier than other jobs.

“call center employees often come up, and I think rightly so, I have a number of simplifying properties with respect to what's automatable today. Their jobs are pretty simple. It's a sequence of tasks and every task looks similar. So you keep repeating something over and over again and that's your job.”

Claim: Call-center work may be amenable to automation because it consists of relatively simple, repetitive tasks that are similar to one another.

“And you complete some task and you change some database entries around or something like that. in the task horizon how long it takes to perform a task and then you want to also remove context like you're not dealing with different parts of services of companies or other customers it's just the database you and a person you're serving and so it's more closed it's more understandable and it's purely digital so I would be looking for those things but even there I'm not actually looking at full automation yet I'm looking for an autonomy slider and I almost expect that we are not going to”

Claim: Call-center tasks can be relatively closed and understandable, involving digital interactions with a customer and changes to database entries.

“instantly replace people. We're going to be swapping in AIs that do 80% of the volume. They delegate 20% of volume to humans,”

Claim: The speaker expects a gradual transition in call centers in which AI handles about 80% of the work volume and humans handle the remaining 20%, rather than people being replaced instantly.

“and humans are supervising teams of five AIs doing the call center work that's more rote.”

Claim: The speaker expects human workers to supervise teams of five AI systems performing rote call-center tasks.

“So I would be looking for new interfaces or new companies that provide some kind of a layer that allows you to manage some of these AIs that are not yet perfect.”

Claim: The speaker expects a need for interfaces or companies that help people manage imperfect AI systems.

“And then I would expect that across the economy, and a lot of jobs are a lot harder. harder than call center employee.”

Claim: The speaker expects many jobs across the economy to be more difficult to automate than call-center jobs.

“And we could be in a similar situation where if you automate 99% of a job, that last 1% of the human has to do is incredibly valuable because it's bottlenecking everything out.”

Claim: The interviewer suggests that when automation covers 99% of a job, the remaining human 1% may become highly valuable if it bottlenecks the overall process.

“and if it ended had if it was the case with like with radiologists where the person sitting in the front of the uber or the front of the waymo has to be specially trained for years in order to be able to provide the last one percent their wages should go up tremendously because they're like the one the one thing bottlenecking wide deployment so radiologists i think their wages have gone up for similar reasons if you're like the last bottleneck you should you're like and you're not fungible which like you know a waymo driver might be fungible with other things um so you might see this thing where like your wages go like whoop”

Claim: The interviewer speculates that radiologists’ wages may have increased because they provide scarce human work that remains a bottleneck to wider automation.

“I don't think we're currently seeing that with radiology or, and I don't have like”

Claim: The speaker does not think the phenomenon under discussion is currently occurring in radiology.

“But I think radiology is not a good example, basically.”

Claim: The speaker considers radiology a poor example for the phenomenon under discussion.

“But I think radiology is not a good example, basically. I don't know why Jeff Hinton picks on radiology, because I think it's an extremely messy, complicated profession.”

Claim: The speaker considers radiology a poor example for assessing AI-driven job automation because it is an extremely complex profession.

“I don't know why Jeff Hinton picks on radiology, because I think it's an extremely messy, complicated profession.”

Claim: The speaker considers radiology an extremely messy and complicated profession.

“So I would be a lot more interested in what's happening with call center employees today,”

Claim: The speaker considers current developments among call-center employees more informative to examine than radiology for understanding automation.

“for example, because I would expect a lot of the road stuff to be automatable today.”

Claim: The speaker expects that much of the routine work in call centers may be automatable today.

“for example, because I would expect a lot of the road stuff to be automatable today.”

Claim: The speaker expects that much of the rote work in call centers is automatable with current technology.

“And I don't have first level access to it, but maybe I would be looking for trends of what's happening with the call center employees.”

Claim: The speaker lacks first-hand access to call-center employment data and says they might look for trends in call-center employment.

“Maybe some of the things I would also expect is maybe they are. swapping in AI, but then I would still wait for a year or two because I would potentially expect them to pull back and actually rehire some of the people.”

Claim: The speaker thinks companies may be replacing call-center workers with AI, but would wait one or two years because companies might reverse course and rehire some workers.

“Maybe some of the things I would also expect is maybe they are. swapping in AI, but then I would still wait for a year or two because I would potentially expect them to pull back and actually rehire some of the people.”

Claim: The speaker would wait one or two years to assess AI substitution in call centers because companies might reverse course and rehire some workers.

“And what you would have naively anticipated that the way this progression would happen is like you take a little task that a consultant is doing, you take that out of the bucket, you take a little task that an accountant is doing, you take that out of the bucket, and then you're just doing this across all knowledge work. But instead, if we do believe we're on the path of AGI with the current paradigm, the progression is very much not like that.”

Claim: If the current AI paradigm is on a path to AGI, the progression is not expected to consist of gradually automating small tasks across knowledge-work occupations.

“it just does not seem like consultants and accountants and whatever are getting like huge productivity improvements”

Claim: The speaker says consultants and accountants do not appear to be receiving large productivity improvements from AI.

“it's very much like programmers are like getting more and more chills of the way their work if you look at the revenues of these companies discounting just like normal chat revenue which i think is like i don't know that's similar to like google or something just looking at api revenues it's like dominated by coding right so this thing which is general quote-unquote which should be able to do any knowledge work”

Claim: The speaker says that, excluding ordinary chat revenue and considering API revenue, the revenues of the companies being discussed are dominated by coding.

“it's very much like programmers are like getting more and more chills of the way their work if you look at the revenues of these companies discounting just like normal chat revenue which i think is like i don't know that's similar to like google or something just looking at api revenues it's like dominated by coding right so this thing which is general quote-unquote which should be able to do any knowledge work it's just overwhelmingly doing only coding and”

Claim: The speaker says that AI systems presented as general-purpose and capable of any knowledge work are, in practice, overwhelmingly being used for coding.

“So I think there's an interesting point here because I do believe coding is like the perfect first thing for these LLMs and agents.”

Claim: The speaker believes coding is an especially suitable first application for large language models and AI agents.

“And that's because coding has always fundamentally worked around text.”

Claim: The speaker says coding has fundamentally been based on text.

“It's computer terminals and text and everything is based around text and LLMs, the way they're trained. trained on the internet a lot of text and so they're perfect text processors”

Claim: The speaker says large language models are trained on substantial amounts of internet text and are therefore well suited to processing text.

“and there's all this data out there and it's just perfect fit and also we have a lot of infrastructure pre-built for handling code and text so for example we have a visual studio code or you know your favorite IDE showing you code and an agent can plug into that so for example if an agent has a diff where it made some change we suddenly have all this code already that shows all the The difference is to a code base using a diff.”

Claim: The speaker says substantial infrastructure already exists for handling code and text.

“a lot of the infrastructure for code now contrast that with some of the things that that don't enjoy that at all so as an example like um there's people trying to build automation not for coding but for example for slides like i saw a company doing slides that's much much harder and the reason it's much much harder is because slides are not text slides are little graphics and they're arranged spatially and there's visual components to it and um”

Claim: The speaker says slide automation is much harder than code automation because slides consist of spatially arranged graphics and visual components rather than text.

“And slides don't have this pre-built infrastructure.”

Claim: The speaker says slides lack the pre-built infrastructure available for code and text.

“how does a thing show you the diff? How do you see the diff? There's nothing that shows diffs for slides. Someone has to build it.”

Claim: The speaker says there is no existing way to display changes between versions of slides, so such a tool would need to be built.

“So it's just some of these things are not amenable to AIs as they are, which is text processors.”

Claim: The speaker says some tasks are not well suited to current AI systems insofar as those systems operate as text processors.

“I do think that code is like, it's pretty structured.”

Claim: The speaker considers code relatively structured.

“Text is maybe a lot more flowery and there's a lot more like... uh like entropy in text i would say i don't know how else to put it um and also i mean code is hard and so people sort of feel”

Claim: The speaker suggests that natural-language text is more flowery and has more entropy than code.

“quite empowered by llms even from like simple simple kind of uh knowledge i basically i don't actually know that i have um a very good answer i mean obviously like text makes it much much easier maybe is maybe why i put it but it doesn't mean that all text is trivial”

Claim: The speaker suggests that text-based tasks may be much easier for AI, while emphasizing that not all text tasks are trivial.

“I guess I see it as like a progression of automation in society.”

Claim: The speaker views superintelligence as a continuation of automation in society.

“I just feel like there will be a gradual automation of a lot of things and superintelligence will be sort of like the extrapolation of that.”

Claim: The speaker expects many activities to be gradually automated and views superintelligence as an extrapolation of that automation trend.

“So I do think we expect more and more autonomous entities over time that are doing a lot of the digital work and then eventually even the physical work, probably some amount of time later.”

Claim: The speaker expects autonomous entities to perform an increasing amount of digital work over time and probably some physical work at a later point.

“I see it as just automation. roughly speaking.”

Claim: The speaker characterizes superintelligence, roughly speaking, as automation.

“but some of the things that people do is invent new things, which I would just put into the automation,”

Claim: The speaker considers the invention of new things to be a form of automation.

“I think it will. I mean, it is fundamentally automation, but I mean, it will be like extremely foreign. I do think it will look really strange because,”

Claim: The speaker expects a civilization containing advanced AI to feel extremely foreign and look strange.

“with respect with respect to when the world looks like that is this kind of like gradual loss of control and understanding of what's happening and i think that's actually the most likely outcome probably is that there will be a gradual loss of understanding of and we'll gradually layer all this stuff everywhere and there'll be fewer and fewer people who understand it and then there will be a sort of this like scenario of gradual loss of control and understanding of what's happening That to me seems most likely outcome of how all the stuff will go down.”

Claim: The speaker considers gradual societal loss of understanding of what is happening—amid widespread deployment of technology and fewer people who understand it—the most likely outcome.

“I think like, I guess I expect the loss of both.”

Claim: The speaker expects society to lose both understanding of and control over increasingly autonomous systems.

“so we're really far into a territory of i don't know what this looks like but if i was to write sci-fi novels they would look along the lines of not even a single like entity or something like that so that just sort of like takes over everything uh but actually like multiple competing entities that gradually become more and more autonomous and uh some of them go rogue and the others like fight them off and all this kind of stuff and it's like this this hot pot of completely autonomous activity that we've delegated to i”

Claim: The speaker speculates that a future scenario could involve multiple increasingly autonomous entities competing, some going rogue and others opposing them, within a large body of autonomous activity delegated by people.

“I mean, there will be tools to people and the people could some of the population is like. they're acting on behalf of people or something like that. So maybe those people are in control, but maybe it's a loss of control overall for society in the sense of like outcomes we want or something like that, where you have entities acting on behalf of individuals that are still kind of roughly seen as out of control.”

Claim: The speaker expects some entities to act on behalf of individuals, while society may still lose control over whether outcomes align with what people want.

“I do, but it's business as usual because we're in an intelligence explosion already and have been for decades.”

Claim: The speaker believes an intelligence explosion is already underway and has been occurring for decades.

“it's basically the GDP curve that is an exponential weighted sum over so many aspects of the industry.”

Claim: The speaker characterizes GDP as an exponential weighted sum over many aspects of industry.

“Everything is gradually being automated, has been for hundreds of years.”

Claim: The speaker says gradual automation has been occurring across activities for hundreds of years.

“Industrial revolution is automation and some of the physical components and tool building and all this kind of stuff.”

Claim: The speaker characterizes the Industrial Revolution as automation involving physical components and tool building.

“Compilers are early software automation,”

Claim: The speaker characterizes compilers as an early form of software automation.

“So I kind of feel like we've been recursively self-improving. exploding for a long time maybe another way to see it is um i mean earth was a pretty i mean if you don't look at the biomechanics and so on it was a pretty boring place i think and looked very similar if you just look from space and earth is spinning and then like we're in the middle of this like firecracker event but we're seeing it in slow motion but i definitely feel like This has already happened for a very long time. And again,”

Claim: The speaker believes society has been recursively self-improving and undergoing an intelligence explosion for a long time.

“I don't see AI as like a distinct technology with respect to what has already been happening for a long time.”

Claim: The speaker views AI as continuous with, rather than distinct from, the technological changes that have been occurring for a long time.

“But then I looked at some of the other technologies that I thought were very transformative, like maybe computers or mobile phones or etc. You can't find. can't find them in gdp”

Claim: The speaker says that transformative technologies such as computers and mobile phones are not readily identifiable in GDP.

“It allows us to write different kinds of programs that we couldn't write before,”

Claim: Karpathy says AI enables people to write kinds of programs that were not previously possible for them to write.

“but AI is still fundamentally a program. And it's a new kind of computer and a new kind of computing system, but it has all these problems.”

Claim: Karpathy describes AI as fundamentally a program and as a new kind of computer and computing system that has problems.

“It's going to diffuse over time and it's still going to add up to the same exponential. And we're still going to have an exponential that's going to get extremely.”

Claim: Karpathy expects AI to diffuse gradually and contribute to the same economic growth pattern as earlier technologies, rather than creating a distinct growth regime.

“I expect the rate of growth has also stayed roughly constant, right?”

Claim: Karpathy expects the rate of economic growth to have stayed roughly constant.

“I guess what I'm saying is for a while I tried to find AI or look for AI in like the GDP curve. And I've kind of convinced myself that this is false.”

Claim: Karpathy says he had tried to identify AI's effects in the GDP curve but became convinced that looking for AI there was a mistaken approach.

“And that even when people talk about recursive self-improvement and labs and stuff like that, I even don't, this business as usual, of course, is going to recursively self-improve and it's been recursively self-improving.”

Claim: Karpathy says ordinary technological and economic progress is already recursively self-improving.

“Like LLMs allow the engineers to work much more efficiently to build the next round of LLM and a lot more of the components are being automated.”

Claim: Karpathy says large language models help engineers build subsequent generations of large language models more efficiently, while an increasing number of components are being automated.

“Yeah, my expectation is that it stays the same pattern.”

Claim: Karpathy expects the existing pattern of economic growth to continue, rather than change because of AI.

“mean, maybe a counterpoint. I mean, number one, I'm actually pretty willing to be convinced one way or another on this point.”

Claim: Karpathy says he is quite willing to be persuaded either way about whether AI will change the economic growth pattern.

“But I will say, for example, computing is labor. Computing was labor. Computers like a lot of jobs disappeared because computers are automating a bunch of digital information processing that you now don't need a human for.”

Claim: Karpathy argues that computing is a form of labor and that computers have eliminated many jobs by automating digital information processing previously done by people.

“And you know self-driving as an example is also like computers doing labor.”

Claim: Karpathy describes self-driving technology as an example of computers performing labor.

“going to be able to do some of the things. It's going to fail at some other things. It's going to be gradually put into society and basically we'll end up with the same pattern is my prediction.”

Claim: Karpathy predicts AI will have uneven capabilities, fail at some tasks, and be gradually integrated into society, resulting in the same broad economic growth pattern as before.

“going to be able to do some of the things. It's going to fail at some other things. It's going to be gradually put into society and basically we'll end up with the same pattern is my prediction. I don't think that we will have this like discrete change. At the same kind of gradual diffusion of this across the industry.”

Claim: The speaker predicts that AI will perform some tasks but fail at others, and will be gradually adopted across society and industry rather than arriving as a sudden, complete transformation.

“Because this assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box and we can dispense it at arbitrary problems in society. I don't think that we will have this like discrete change.”

Claim: The speaker does not expect a sudden discrete transition to fully general, human-like AI that can be deployed on arbitrary societal problems.

“Because this assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box and we can dispense it at arbitrary problems in society. I don't think that we will have this like discrete change.”

Claim: Karpathy does not expect a discrete economic or societal shift caused by a fully general, flexible, human-like AI that can be deployed on arbitrary problems.

“At the same kind of gradual diffusion of this across the industry.”

Claim: Karpathy expects AI to spread gradually across industries.

“but I still think that you're presupposing some discrete jump, some unlock that we're waiting to claim. And I still think you're presupposing some discrete jump that I think has basically no historical precedent that I can't find in any of the statistics and that I think probably won't happen.”

Claim: The speaker believes a discrete jump of the kind being proposed has no historical precedent in the statistics he can find and probably will not occur.

“I'm a little bit suspicious and I would have to take a look. For example, like maybe some of the logs are not very good from before the Industrial Revolution or something like that. So I'm a little bit suspicious of it, but yeah, maybe you're right. I don't have strong opinions.”

Claim: The speaker is uncertain about the historical evidence for a major growth jump, noting that records from before the Industrial Revolution may be poor and that he has no strong opinion.

“think the crucial thing about the Industrial Revolution was that it was not magical, if you just zoomed in what you would see in 1770 or 1870 is not that there like was some key invention but”

Claim: The speaker argues that the Industrial Revolution was not caused by a single magical key invention; examining 1770 or 1870 would not reveal one such invention.

“at the same time you did move the economy to a regime where the progress was much faster and the exponential 10x and i expect a similar thing from ai where it's not like”

Claim: The speaker says the Industrial Revolution shifted the economy into a much faster-progress regime, with exponential growth increasing tenfold.

“at the same time you did move the economy to a regime where the progress was much faster and the exponential 10x and i expect a similar thing from ai where it's not like there's going to be a single moment where made the crucial invention”

Claim: The speaker expects AI to bring a comparable shift toward faster economic progress, but not through one sudden moment when a crucial invention is made.

“yeah and i mean maybe one way to think about it is through history a lot of growth i mean growth comes because people come up with ideas and then people are like out there doing stuff to execute those ideas and make valuable output and through most of this time population isn't exploding that has been driving growth for the last 50 years people have argued that growth has stagnated population and frontier countries is also stagnated i think we go back on the hyper exponential growth in population and output right sorry exponential growth in population that causes hyper exponential growth and output yeah”

Claim: The speaker says economic growth often results from people generating ideas and others executing them to create valuable output.

“out there doing stuff to execute those ideas and make valuable output and through most of this time population isn't exploding that has been driving growth for the last 50 years people have argued that growth has stagnated population and frontier countries is also stagnated i think we go back on the hyper exponential growth in population and output right sorry exponential growth in population that causes hyper exponential growth and output yeah”

Claim: The speaker says population growth has driven economic growth over the last 50 years, while population and growth in frontier countries have stagnated; he contrasts this with earlier periods when population was not rapidly increasing.

“out there doing stuff to execute those ideas and make valuable output and through most of this time population isn't exploding that has been driving growth for the last 50 years people have argued that growth has stagnated population and frontier countries is also stagnated i think we go back on the hyper exponential growth in population and output right sorry exponential growth in population that causes hyper exponential growth and output yeah”

Claim: The speaker thinks a return to exponential population growth could produce hyper-exponential growth in output.

“I don't intuitively feel that viewpoint.”

Claim: The speaker says he does not intuitively find the preceding account of growth convincing.

“I am surprised that it evolved.”

Claim: The speaker says he is surprised that intelligence evolved.

“I would expect that the evolution of intelligence intuitively feels to me like it should be fairly rare event and there have been animals.”

Claim: The speaker expects the evolution of intelligence to be fairly rare.

“animals for i guess maybe you should base it on how long something has existed so for example if bacteria have been around for two billion years and nothing happened then going to your carrier is probably pretty hard because because bacteria actually came up quite early in earth's evolution”

Claim: The speaker suggests that if bacteria existed for two billion years without intelligence emerging, the transition to intelligence may be difficult, given how early bacteria arose in Earth's history.

“And so I guess, how long have we got animals? Maybe a couple hundred million years, like multicellular animals that like run, run, crawl, et cetera, which is maybe 10% of Earth's lifespan or something like that. So maybe on that timescale, it's actually not too tricky. I still feel like.”

Claim: The speaker estimates that mobile multicellular animals have existed for a couple hundred million years, roughly 10% of Earth's lifespan.

“which is maybe 10% of Earth's lifespan or something like that. So maybe on that timescale, it's actually not too tricky. I still feel like. It's still surprising to me, I think intuitively, that it developed.”

Claim: The speaker thinks intelligence may not be especially difficult to evolve over the timescale that animals have existed, but still finds its development surprising.

“I would maybe expect just a lot of like animal-like life forms doing animal-like things.”

Claim: The speaker might have expected extraterrestrial life to consist mostly of animal-like organisms engaged in animal-like behavior.

“The fact that you can get something that creates culture and knowledge and accumulates it is surprising to me.”

Claim: The speaker finds it surprising that life can produce culture and knowledge and accumulate them over time.

“Yes, basically it's so hard to tell, right, with Yeah any of this stuff. I guess you can base it a little bit on how long something has existed or how long it feels like something has been bottlenecked.”

Claim: The speaker says it is difficult to assess how likely intelligence is to evolve, and suggests considering how long a process has existed or remained bottlenecked.

“So yeah. Nikolai is very good about describing this like very apparent bottleneck in bacteria and archaea. For two billion years, nothing happened. Like extreme diversity of chemical, of biochemistry, and yet nothing that grows to become animals. Two billion years.”

Claim: The speaker describes bacteria and archaea as having undergone a two-billion-year period of extensive biochemical diversity without anything developing into animals.

“I don't know that we've seen exactly that kind of an equivalent with animals and intelligence.”

Claim: The speaker is unsure whether animals and intelligence have experienced a bottleneck comparable to the two-billion-year bacterial and archaeal bottleneck.

“one thought on that is i almost feel like um well there's the hominid intelligence and there's i would say like the bird intelligence right like ravens etc are extremely clever but they actually their brain brain parts are actually quite distinct and we don't have that much um”

Claim: The speaker says ravens and other birds are highly intelligent, with brain structures quite distinct from human brain structures.

“existence so maybe that's a slight event of there's a slight indication of maybe intelligence springing up a few times and so in that case you'd maybe expect it more frequently or something”

Claim: The speaker suggests that distinct forms of intelligence may indicate that intelligence evolved independently several times, which could mean it is more frequent than expected.

“oh yeah and just stuff to work with i mean i'm guessing it would be harder to if i was a dolphin um i mean how do you do you can't have fire for example and stuff like that i mean they're probably like the universe of things you can do in water um like inside water is probably lower than what you can do on land um just chemically yeah i do i do agree with this with this viewpoint of these niches and what's what's being incentivized i still find it kind of miraculous that uh i don't i would have maybe expected things to get stuck on like animals with bigger muscles you”

Claim: The speaker agrees that evolutionary niches and their incentives matter, but finds the evolution of intelligence remarkable and might have expected evolution to favor larger muscles instead.

“exactly. You have to incentivize some kind of adaptability. You actually want something that you actually want environments that are unpredictable.”

Claim: The speaker says adaptability requires incentives and that environments should be unpredictable.

“So evolution can't bake your algorithms into your weights. A lot of a lot of animals are basically pre-baked.”

Claim: The speaker says evolution cannot encode learning algorithms into animals in advance, and that many animals are largely pre-programmed.

“So evolution can't bake your algorithms into your weights. A lot of a lot of animals are basically pre-baked.”

Claim: Evolution cannot encode an animal’s algorithms directly into its weights, leaving many animals with largely pre-baked behavior.

“be baked in the sense and so humans have to figure it out the test time when they get born and so maybe there was um you actually want these kinds of uh environments that actually change really rapidly or something like that where you can't foresee um what will work well and so you actually put all that intelligent you create intelligence to figure it out at this time”

Claim: Humans figure things out at test time after birth, and rapidly changing, unpredictable environments may favor intelligence that can adapt when successful strategies cannot be foreseen.

“be baked in the sense and so humans have to figure it out the test time when they get born and so maybe there was um you actually want these kinds of uh environments that actually change really rapidly or something like that where you can't foresee um what will work well and so you actually put all that intelligent you create intelligence to figure it out at this time”

Claim: The speaker says humans must learn after birth, and suggests that rapidly changing, unforeseeable environments may favor intelligence that can adapt and solve problems at the time they arise.

“Yes and no, because LLMs don't really have the equivalent of culture.”

Claim: LLMs do not currently have a true equivalent of human culture.

“And maybe we're giving them way too much incentivizing not to create it or something like that.”

Claim: The speaker suspects that current incentives may discourage LLMs from creating culture.

“But I guess like the invention of culture and of written record and of like passing down notes between each other. I think there's an equivalent of that with LLMs right now. So LLMs don't really have culture right now, and it's kind of like one of the, I think, impediments, I would say.”

Claim: The speaker thinks LLMs may have an analogue to human culture, written records, and shared notes, but their current lack of culture is an impediment.

“So in the simplest case, it would be a giant scratchpad that the LLM can edit, and as it's reading stuff, or as it's helping out with work, it's editing the scratchpad for itself.”

Claim: A basic form of LLM culture could be a shared or persistent scratchpad that an LLM edits while reading or doing work.

“Why can't an LLM write a book for the other LLMs? Like why can't other LLMs read this LLMs book and be inspired by it or shocked by it or something like that? There's no equivalence for any of this stuff.”

Claim: The speaker proposes that LLMs could write works for other LLMs to read and respond to, while saying that no equivalent currently exists.

“I think there's two powerful ideas in the realm of multi-agent that have both not been really claimed or so on. The first one I would say is culture and LLM's basically growing repertoire of knowledge for their own purposes. The second one looks a lot more like the powerful idea of self-play in my mind is extremely powerful.”

Claim: The speaker identifies two underdeveloped multi-agent ideas: LLMs developing culture by growing knowledge for their own purposes, and self-play.

“The second one looks a lot more like the powerful idea of self-play in my mind is extremely powerful. So evolution actually has a lot of competition basically driving intelligence and”

Claim: The speaker considers self-play extremely powerful and says competition in evolution helps drive intelligence.

“and evolution um and for in alpha go more algorithmically like alpha go is playing against itself and that's how it learns to get really good at go and there's no equivalent of self-playing llms but i would expect that to also exist but no one has done it yet like why can't an llm for example create a bunch of problems that another llm is learning to solve and then the the llm is always trying to like serve more and more difficult problems stuff like that you know so like”

Claim: AlphaGo improved at Go through self-play; the speaker says LLMs do not yet have an equivalent self-play system but expects such systems to emerge.

“and evolution um and for in alpha go more algorithmically like alpha go is playing against itself and that's how it learns to get really good at go and there's no equivalent of self-playing llms but i would expect that to also exist but no one has done it yet like why can't an llm for example create a bunch of problems that another llm is learning to solve and then the the llm is always trying to like serve more and more difficult problems stuff like that you know so like”

Claim: A possible LLM self-play method would have one LLM generate progressively harder problems for another LLM to solve.

“I think there's a bunch of ways to actually organize it. And I think it's a realm of research.”

Claim: The speaker believes multi-agent culture and self-play systems can be organized in multiple ways and warrant research.

“But I think I haven't seen anything that convincingly like claims both of those, like multi-agent improvements.”

Claim: The speaker has not seen convincing demonstrations of multi-agent improvements that combine culture and self-play.

“I still think we're mostly in the realm of a single individual agent. But I think I also think that will change. And in the realm of culture also, I would bucket also organizations.”

Claim: AI systems are currently mostly single-agent, in the speaker’s view, but the speaker expects this to change.

“But I think I also think that will change. And in the realm of culture also, I would bucket also organizations. And we haven't seen anything like that come missing either.”

Claim: The speaker treats organizations as a form of AI culture and says comparable organizational systems have not yet emerged.

“So that's why we're still early.”

Claim: The speaker considers AI multi-agent culture and collaboration to be an early-stage area because such systems have not yet emerged.

“somehow remarkably again some of these analogies work and they shouldn't but somehow remarkably they do a lot of the smaller models or the dumber model like the smaller models somehow remarkably resemble like a kindergarten student or then like a elementary school student or high school student etc and somehow we still haven't like graduated enough where this stuff can take over like it's still mostly like my cloth code or codex they still kind of feel like this elementary grade student i know that they can take phd quizzes this but they still cognitively feel like a kindergartner in elementary school”

Claim: The speaker compares smaller AI models to school-age children and says current models can answer PhD-level quizzes while still feeling cognitively like elementary-school students.

“student so i don't think they can create culture because they're still kids um you know like they're savant kids um they have absurd they have perfect memory of all this stuff etc and they can uh convincingly create all kinds of slop that looks really good but i still think they don't really know what they're doing and they don't really have the cognition across all these little check boxes that we still have to collect”

Claim: The speaker believes current AI models cannot create culture because they remain cognitively immature, despite strong memory and convincing output, and lack a full range of cognitive abilities.

“So I would say one thing I will almost instantly also push back on is this is not even near done.”

Claim: The speaker says self-driving technology is still far from complete.

“I do think that self-driving is very interesting because it's definitely like where I get a lot of my intuitions because I spent five years on it.”

Claim: The speaker says his five years working on self-driving strongly inform his intuitions and make the field interesting to him.

“There's demos of self-driving go all the way to 1980s. You can see a demo from CMU in 1986. There's a truck that's driving itself on roads.”

Claim: Self-driving demonstrations date back to the 1980s, including a Carnegie Mellon University demonstration in 1986 of a truck driving itself on roads.

“I think when I was joining Tesla, I had a very early demo of a Waymo, and it basically gave me a perfect drive in 2014 or something like that. So perfect Waymo drive a decade ago. It took us around Palo Alto and”

Claim: The speaker recalls seeing an early Waymo demonstration around 2014 that provided what he considered a perfect drive.

“And I do think that for some kinds of tasks and jobs and so on, there's a very large demo to product gap, where the demo is very easy, but the products are very hard.”

Claim: For some tasks and jobs, demonstrations are easy to produce but turning them into products is very difficult.

“And it's especially the case in cases like self-driving, where the cost of failure is too high, right?”

Claim: Self-driving has a particularly large demo-to-product gap because failures carry an unacceptably high cost.

“Many industries' tasks and jobs maybe don't have that property, but when you do have that property... that definitely increases the timelines.”

Claim: High failure costs definitely lengthen the time required to turn a demonstration into a product.

“I do think that, for example, in software engineering, I do actually think that that property does exist. I think for a lot of vibe coding, it doesn't. But I think if you're writing actual production grade code,”

Claim: The high cost of failure applies to production-grade software engineering, but not to many vibe-coding tasks.

“I think that property should exist because any kind of mistake actually leads to a security vulnerability or something like that and millions and hundreds of millions of people's personal social security numbers, etc. get leaked or something like that.”

Claim: Errors in production-grade software can create security vulnerabilities that expose personal information belonging to millions or hundreds of millions of people.

“And so I do think that it is a case that in software, people should be careful.”

Claim: People should exercise caution when working with software because failures can have serious consequences.

“Like in self-driving, if things go wrong, you might get injury.”

Claim: Failures in self-driving systems might cause physical injury.

“But I guess in software, I almost feel like it's almost unbounded how terrible some things could be.”

Claim: The potential severity of some software failures feels almost unbounded.

“So I do think that they share that property.”

Claim: Self-driving and software engineering both involve potentially very high costs of failure.

“And then I think basically what takes the long amount of time and the way to think about it is that it's a margin of nines and every single nine is a constant amount of work. So every single nine is the same amount of work.”

Claim: Improving system reliability through successive additional nines takes a constant amount of work for each nine.

“So when you get a demo and something works 90% of the time, that's just the first nine.”

Claim: A system that works 90% of the time has reached only its first reliability nine.

“And while I was at Tesla for, was it five years or so, I think we went through maybe three nines or two nines. I don't know what it is, you know, but like multiple nines of iteration, there's still more nines to go.”

Claim: During approximately five years at Tesla, the speaker estimates that the team achieved perhaps two or three additional reliability nines through iteration, with further nines still needed.

“And so that's why these things take so long.”

Claim: The need to progress through successive reliability nines is why these systems take a long time to develop.

“I'm very unimpressed by demos. So whenever I see demos of anything, I'm extremely unimpressed by that.”

Claim: The speaker is generally very unimpressed by demonstrations.

“It works better if you can, if it's a demo that someone cooked up and is just showing you it's worse. If you can interact with it, it's a bit better. But even then you're not done. You need actual product.”

Claim: An interactive demonstration provides stronger evidence than a staged presentation, but it does not establish that a product is ready.

“It's going to face all these challenges when it comes in contact with reality and all these different pockets of behavior that need patching.”

Claim: A system deployed in the real world will encounter diverse behaviors and challenges that require fixes.

“Demos are encouraging. Still a huge amount of work to do.”

Claim: Demonstrations can be encouraging, but substantial work remains before the system is complete.

“it's a much harder problem i mean self-driving is just one of thousands of things that people do it's almost like a single vertical i suppose um whereas when we're talking about general software engineering it's even more there's more surface area”

Claim: General software engineering is harder to solve than self-driving because it spans more surface area, whereas self-driving is nearly one vertical among thousands of human activities.

“is that with self-driving what took a big fraction of that time was solving the problem of building basic having basic perception that's robust and building representations and having a model that has some common sense so it can generalize to when i see something that's slightly out of distribution if somebody's waving down the road this way you don't need to train for it the thing will will have some understanding of how to respond to something like that and these are things we're getting for free with llms or vlms today so we don't have to solve these very basic representation problems and so now deploying ais across different domains will sort of be like deploying a self-driving car with current models to a different city which is hard but not like a 10-year long task”

Claim: The speaker says that LLMs and VLMs provide basic representations and generalization without requiring those foundational representation problems to be solved separately.

“will have some understanding of how to respond to something like that and these are things we're getting for free with llms or vlms today so we don't have to solve these very basic representation problems and so now deploying ais across different domains will sort of be like deploying a self-driving car with current models to a different city which is hard but not like a 10-year long task”

Claim: Deploying AI systems across different domains is difficult but, in the speaker’s view, is not a decade-long task; it is analogous to deploying a self-driving car in a new city.

“yeah basically i'm not 100 sure if i fully agree with that i don't know that we're how much we're getting for free and i still think there's like a lot of a lot of gaps in understanding in what we are getting um i'm we're definitely getting more generalizable intelligence in a single entity uh whereas self-driving is a very special purpose task that requires in some sense building a special purpose task is maybe even harder in a certain sense because it doesn't like fall out from a more general thing that you're doing at scale if that makes sense so um”

Claim: The speaker is uncertain that LLMs and VLMs provide as much generalization for free as claimed, and says there are many gaps in understanding what they provide.

“a lot of gaps in understanding in what we are getting um i'm we're definitely getting more generalizable intelligence in a single entity uh whereas self-driving is a very special purpose task that requires in some sense building a special purpose task is maybe even harder in a certain sense because it doesn't like fall out from a more general thing that you're doing at scale if that makes sense so um”

Claim: The speaker says current AI systems provide more generalizable intelligence in a single entity than self-driving systems, which address a special-purpose task.

“a lot of gaps in understanding in what we are getting um i'm we're definitely getting more generalizable intelligence in a single entity uh whereas self-driving is a very special purpose task that requires in some sense building a special purpose task is maybe even harder in a certain sense because it doesn't like fall out from a more general thing that you're doing at scale if that makes sense so um”

Claim: The speaker suggests that building a special-purpose system may be harder because it does not emerge from developing a more general system at scale.

“But I still think that the analogy doesn't, I still don't know if it fully resonates because like the LLMs are still pretty fallible and I still think that they have a lot of gaps and that it still needs to be filled in and I don't think that we're getting like magical generalization completely out of the box sort of in a certain sense.”

Claim: The speaker says LLMs remain fallible, have many gaps to address, and do not provide magical generalization completely out of the box.

“So even though, so the deployments still are pretty minimal.”

Claim: The speaker says self-driving car deployments remain quite minimal.

“right so even Waymo and so on has very few cars and they're doing that roughly speaking because they're not economical right because they've built something that that lives in the future and so they had to like pull back future but they had to make it uneconomical so they have all these like you know there's all these costs not just marginal costs for those cars and their operation maintenance but also the capex of the entire thing so making economical is still going to be a slog I think”

Claim: The speaker says Waymo operates relatively few cars because its service is not economical, and expects making it economical to remain a difficult, prolonged process.

“right so even Waymo and so on has very few cars and they're doing that roughly speaking because they're not economical right because they've built something that that lives in the future and so they had to like pull back future but they had to make it uneconomical so they have all these like you know there's all these costs not just marginal costs for those cars and their operation maintenance but also the capex of the entire thing so making economical is still going to be a slog I think”

Claim: The speaker says Waymo’s costs include vehicle-level operating and maintenance expenses as well as capital expenditure for the overall system.

“for them um and then also i think when you look at these cars and there's no one driving um i also think it's a little bit deceiving because there are actually very elaborate teleoperation centers of people actually kind of like in the loop with these cars and i don't have the i don't have the full extent of it but i think um there's more human in the loop that you might expect and there's people somewhere out there basically uh beaming in from the sky uh and i don't actually know they're fully in the loop with the driving i think some of the times the times they are but they're certainly involved and there are people and in some sense we haven't actually removed the person we've like moved them to somewhere we can't see them i still think there will be some work as you mentioned going from environment to environment and so i think like there's still challenges to to make self-driving real but i i do agree that it's definitely across a threshold where it kind of feels real unless it's like really teleoperated for example waymo can't go to all the different parts of the city my suspicion is it's like parts of city where we don't get good signal”

Claim: The speaker suspects self-driving services use elaborate teleoperation centers and more human involvement than commonly expected, but says he does not know the full extent or whether people directly control the cars at all times.

“the times they are but they're certainly involved and there are people and in some sense we haven't actually removed the person we've like moved them to somewhere we can't see them i still think there will be some work as you mentioned going from environment to environment and so i think like there's still challenges to to make self-driving real but i i do agree that it's definitely across a threshold where it kind of feels real unless it's like really teleoperated for example waymo can't go to all the different parts of the city my suspicion is it's like parts of city where we don't get good signal”

Claim: The speaker suspects Waymo operates only in parts of a city with good signal and says it cannot serve all parts of the city.

“the times they are but they're certainly involved and there are people and in some sense we haven't actually removed the person we've like moved them to somewhere we can't see them i still think there will be some work as you mentioned going from environment to environment and so i think like there's still challenges to to make self-driving real but i i do agree that it's definitely across a threshold where it kind of feels real unless it's like really teleoperated for example waymo can't go to all the different parts of the city my suspicion is it's like parts of city where we don't get good signal”

Claim: The speaker expects further work to adapt self-driving systems across environments and says significant challenges remain before self-driving is fully realized at scale.

“the times they are but they're certainly involved and there are people and in some sense we haven't actually removed the person we've like moved them to somewhere we can't see them i still think there will be some work as you mentioned going from environment to environment and so i think like there's still challenges to to make self-driving real but i i do agree that it's definitely across a threshold where it kind of feels real unless it's like really teleoperated for example waymo can't go to all the different parts of the city my suspicion is it's like parts of city where we don't get good signal”

Claim: The speaker says self-driving services may not have eliminated human involvement, but instead moved people to remote locations where passengers cannot see them.

“I actually, by the way, love Waymo and I take it all the time.”

Claim: The speaker says he likes Waymo and uses its service frequently.

“So I don't want to say like, I just think people again are sometimes a little bit too naive about some of the progress and I still think there's a huge amount of work.”

Claim: The speaker thinks people sometimes overestimate self-driving progress and that substantial work remains.

“And I think Tesla took, in my mind, a lot more scalable approach.”

Claim: The speaker considers Tesla’s approach to self-driving more scalable than the alternative being discussed.

“And I think the team is doing extremely well and is going to, and I'm kind of like on the record for predicting how this thing will go,”

Claim: The speaker assesses Tesla’s self-driving team as doing extremely well.

“But I do think Tesla is taking the more scalable strategy and is going to look a lot more like that.”

Claim: The speaker predicts Tesla’s self-driving strategy will prove more scalable and become the more prevalent approach.

“So I think this will have to still play out and hasn't. But basically, like, I don't want to talk about self-driving as something that took a decade because it didn't take. It didn't take yet,”

Claim: The speaker says the self-driving trajectory has not finished playing out, so it is premature to say the task took a decade.

“the end is not near yet because when we're talking about self-driving, usually in my mind it's self-driving at scale. People don't have to look at a driver's license,”

Claim: The speaker generally uses “self-driving” to mean self-driving at scale, such that people do not need to hold or check a driver’s license.

“might be different and the reason i'm especially curious about this is because i think the question of how fast ai is deployed how valuable it is when it's early on is like potentially the most important question in the world right now right like if you're trying to model what the year or 20 or 30 looks like this is the question you want to have some understanding of so another thing you might think is one you have this latency requirement with self-driving where you have”

Claim: The speaker says that the speed of AI deployment and its early-stage value may be the most important question in the world right now.

“I have no idea what the actual models are, but I assume like tens of millions of parameters or something,”

Claim: The speaker guesses that self-driving models have tens of millions of parameters, while explicitly saying he does not know their actual size.

“which is not the necessary constraint for knowledge work with LLMs or maybe it might be with computer use and stuff.”

Claim: The speaker suggests that the model-size constraint associated with self-driving may not apply to knowledge work using LLMs or computer-use systems.

“there is additional cost to serving up an additional copy of a model, but.”

Claim: Serving an additional instance of an AI model incurs additional cost.

“the sort of apex of a session is quite low and you can amortize the cost of AI into the training run itself depending on how inference scaling goes and stuff but it's certainly not as much as like building a whole new car to serve another instance of a model so it just the economics of deploying more widely are much more favorable”

Claim: The speaker says the peak cost of an AI session is relatively low and that AI costs may be amortized into the training run, depending on inference scaling.

“the sort of apex of a session is quite low and you can amortize the cost of AI into the training run itself depending on how inference scaling goes and stuff but it's certainly not as much as like building a whole new car to serve another instance of a model so it just the economics of deploying more widely are much more favorable”

Claim: The speaker argues that serving another AI model instance costs far less than building another car, making the economics of wider AI deployment more favorable than deployment of physical systems such as self-driving cars.

“Bits are like a million times easier than anything that touches the physical world.”

Claim: The speaker says digital systems are about a million times easier to work with than systems involving the physical world.

“Bits are completely changeable, arbitrarily reshufflable at a very rapid speed.”

Claim: Digital bits can be changed and rearranged arbitrarily quickly.

“So you would expect a lot more faster adaptation also in the industry and so on.”

Claim: The speaker expects the changeability of digital bits to enable faster adaptation in industry.

“I mean, I also think that if we are talking about knowledge work at scale, there will be some latency requirements practically speaking because we,”

Claim: The speaker expects large-scale knowledge work to impose practical latency requirements.

“What does society think about it? What is the legal ramification? How is it working legally? How is it working insurance wise?”

Claim: The speaker raises societal, legal, and insurance implications as aspects to consider in AI deployment.

“You know, there's going to be equivalents of all that.”

Claim: The speaker expects AI deployment to have counterparts to people obstructing a Waymo vehicle with a cone.

“And so I do think that I almost feel like self-driving is a very nice analogy that you can borrow things from.”

Claim: The speaker considers self-driving a useful analogy for thinking about AI deployment and its surrounding issues.

“I think this will work. work, I think it's tractable.”

Claim: The speaker is optimistic that the technology will work and believes the problem is tractable.

“I'm only sounding pessimistic because when I go on my Twitter timeline, see ha. all this stuff that makes no sense to me.”

Claim: The speaker says their pessimistic-sounding reactions are partly a response to claims on their Twitter timeline that they find nonsensical.

“And I think there's a lot of reasons for why that exists. And I think a lot of it is, I think, honestly, just stuff fundraising. It's just incentive structures. A lot of it may be fundraising.”

Claim: The speaker thinks fundraising and incentive structures explain much of the nonsensical claims they encounter about AI.

“A lot of it is just attention, you know, converting attention to money on the internet,”

Claim: The speaker thinks online attention and its monetization drive much of the discourse they are reacting to.

“But I'm still like overall very bullish on technology. I think we're going to work through all this stuff.”

Claim: The speaker remains broadly optimistic about technology and expects people to work through the associated issues.

“I think there's been a rapid amount of progress.”

Claim: The speaker says technological progress has been rapid.

“I don't actually know that there's overbuilding.”

Claim: The speaker is uncertain whether current compute infrastructure is being overbuilt.

“I think that there's going to be, we're going to be able to gobble up what, in my understanding, is being built because I do think that, for example, cloth code or OpenAI codecs and stuff like that, they didn't even exist a year ago,”

Claim: The speaker expects demand to absorb the compute infrastructure being built, citing AI coding tools that, in their understanding, did not exist a year earlier.

“there's going to be a huge amount of demand as we see the demand in ChatGPT already and so on.”

Claim: The speaker expects substantial demand for AI, citing existing demand for ChatGPT as evidence.

“But I guess I'm just reacting to like some of the very fast timelines that people continue to say incorrectly.”

Claim: The speaker says their reaction is partly to people repeatedly making incorrect claims about very fast AI timelines.

“And I've heard many, many times over the course of my 15 years in AI, where very reputable people keep getting this wrong all the time.”

Claim: The speaker says that during their 15 years in AI, many reputable people repeatedly made incorrect predictions.

“So I do want us to be grounded in reality of what technology is and isn't. So I”

Claim: The speaker believes people should have a realistic understanding of what technology can and cannot do.

“I feel some amount of like determinism around the things that AI labs are doing.”

Claim: The speaker perceives a degree of determinism in the activities of AI labs.

“And I feel like I could help out there, but I don't know that I would like uniquely, I don't know that I would like uniquely improve it.”

Claim: The speaker thinks they could contribute to AI labs but is unsure whether they could uniquely improve the work.

“But I think like my personal big fear is that a lot of this stuff happens on the side of humanity and that humanity gets disempowered by it.”

Claim: The speaker fears that AI developments may proceed independently of humanity and leave humans disempowered.

“I care not just about all the Dyson spheres that we're going to build and that AI is going to build in a fully autonomous way. I care about what happens to humans and I want humans to be well off in this future.”

Claim: The speaker wants humans to fare well in a future where AI may build things autonomously.

“And I feel like that's where I can a lot more uniquely add value than like an incremental improvement in a frontier lab.”

Claim: The speaker believes they can contribute more uniquely to improving human outcomes than to incremental improvements at a frontier AI lab.

“And so I guess I'm most afraid of something maybe like depicted in movies like WALL-E or Idiocracy or something like that where humanity is sort of on the side of this stuff.”

Claim: The speaker fears a future resembling WALL-E or Idiocracy, where humanity is sidelined by technological developments.

“and I want humans to be much, much better in this future.”

Claim: The speaker wants humans to be substantially better off in the future.

“this is kind of like through education that you can actually achieve this.”

Claim: The speaker believes education can help achieve a future in which humans fare better.

“yeah. So Eureka is trying to build, I think maybe the easiest way I can describe it is we're trying to build the Starfleet Academy.”

Claim: Eureka is trying to build an institution modeled on the idea of Starfleet Academy.

“Okay, Starfleet Academy is this like elite institution for frontier technology, building spaceships and graduating cadets to be like the pilots of these spaceships and whatnot.”

Claim: In Star Trek, Starfleet Academy is an elite frontier-technology institution that graduates cadets to pilot spaceships.

“So I just imagine like an elite institution for technical knowledge and basically a kind of school that's very up-to-date and very like a premier institution.”

Claim: The speaker envisions Eureka as a premier, current institution for technical education.

“with respect to Eureka, I think like one thing that is very fascinating to me about the education is like, I do think educational pretty fundamentally changed with AI's on the side.”

Claim: Andrej Karpathy thinks education has fundamentally changed because AI is now available alongside learners.

“And I think it has to be rewired and changed to some extent.”

Claim: Andrej Karpathy thinks education needs to be rewired and changed to some extent in response to AI.

“I still think that we're pretty early.”

Claim: Andrej Karpathy thinks AI-enabled education is still at a very early stage.

“I think there's going to be a lot of people who are going to try to do to do the obvious things, which is like, oh, have an LLM and ask it questions and get, you know, do all the basic things that you would do via prompting right now.”

Claim: Andrej Karpathy expects many people to try using LLMs for basic educational tasks, such as asking questions through prompting.

“I think it's helpful, but it still feels to me a bit slop, like slop.”

Claim: Andrej Karpathy considers basic prompting-based educational use of LLMs helpful but somewhat low-quality or “slop-like.”

“And I think the capability is not there for what I would want. What I'd want is like an actual tutor experience.”

Claim: Andrej Karpathy thinks current AI capabilities are insufficient for the actual tutor experience he wants to build.

“I was recently learning Korean, so language learning. And I went through a phase where I was learning Korean by myself on the internet. I went through a phase where I was actually part of a small class in Korea, taking Korean with a bunch of other people, which was really funny. But we had a teacher and like 10 people or so taking Korean.”

Claim: Andrej Karpathy learned Korean independently online and later attended a small Korean class in Korea with a teacher and about ten students.

“And then I switched to a one-on-one tutor. And I guess what was fascinating to me is I think I had a really good tutor.”

Claim: Andrej Karpathy switched from a Korean class to one-on-one tutoring and considered his tutor very good.

“like what this tutor was doing for me and how incredible that experience was and how high the bar is for like what I actually want to build eventually um because uh I mean she was extremely so she instantly from a very short conversation understood like where I am as a student what I know and don't know and she was able to like probe exactly like the kinds of questions or things to understand my world model no LLM will do that for you 100% right now not even close right but a tutor will do that if they're good”

Claim: Andrej Karpathy says a good tutor can quickly assess a student's knowledge and probe their mental model, while no LLM can do this for him at a comparable level right now.

“really like really served me all the things that I needed at my current sliver of capability I need to be always appropriately challenged I can't be faced with something too hard or too trivial and a tutor is really good at serving you just the right stuff and so basically I felt like I was the only constraint to learning like my own I was the only constraint I was always given the perfect information I'm the only constraint and I felt good because I'm the only impediment that exists it's not that I can't find knowledge or that it's not properly explained or etc”

Claim: Andrej Karpathy says a good tutor adapts instruction to a learner's current ability, keeping the material appropriately challenging rather than too difficult or too trivial.

“or too trivial and a tutor is really good at serving you just the right stuff and so basically I felt like I was the only constraint to learning like my own I was the only constraint I was always given the perfect information I'm the only constraint and I felt good because I'm the only impediment that exists it's not that I can't find knowledge or that it's not properly explained or etc etc like it's just my ability to memorize and so on and this is what I want for”

Claim: Andrej Karpathy says that effective tutoring made his own ability to memorize the only meaningful constraint on his learning, rather than access to knowledge or adequate explanations.

“a very good question about the current capability you don't but I do think that with as and that's why I think it's not actually the right right time to actually build this kind of an AI tutor I still think it's a useful product and lots of people will build it but I still feel like the bar is so high and the capability is not there”

Claim: Andrej Karpathy considers AI tutoring a useful product and expects many people to build AI tutors, despite believing current capabilities are insufficient for his desired version.

“a very good question about the current capability you don't but I do think that with as and that's why I think it's not actually the right right time to actually build this kind of an AI tutor I still think it's a useful product and lots of people will build it but I still feel like the bar is so high and the capability is not there”

Claim: Andrej Karpathy thinks current AI capabilities fall short of the high bar for the kind of AI tutor he wants to build, so it is not yet the right time to build it.

“I would say Chachapati is an extremely valuable educational product.”

Claim: Andrej Karpathy considers ChatGPT an extremely valuable educational product at the time of the conversation.

“And when I was with her, I almost felt like there's no way I can build this.”

Claim: Andrej Karpathy says his experience with a highly effective tutor made him feel that he could not build an equivalent tutoring experience.

“So I guess I'm waiting for that capability.”

Claim: Andrej Karpathy is waiting for AI capabilities to advance enough to support the tutor experience he wants to build.

“for example, I did some AI consulting for computer vision. A lot of my times, the value that I brought to the company was telling them not to use AI.”

Claim: Andrej Karpathy says that in his computer-vision AI consulting, much of his value to companies came from advising them not to use AI.

“And I feel like it's in the same in education right now where I kind of feel like for what I have in mind, it's not yet the time, but the time will come. But for now, I'm building something that looks maybe a bit more conventional, that has a physical and digital component and so on.”

Claim: Andrej Karpathy thinks the time is not yet right for his envisioned AI tutor, though it will come; meanwhile, he is building a more conventional education product with both physical and digital components.

“But I think there's obvious there's obvious it's obvious how this should look like in the future.”

Claim: Andrej Karpathy thinks the future form of education is obvious.

“And I want to have a really, really good course. State of the art, obvious state of the art destination you go to learn AI in this case, because that's just what I'm familiar with. So I think it's a really good first product to get to be really good.”

Claim: Andrej Karpathy is building a first course intended to be a high-quality, state-of-the-art destination for learning AI.

“And so that's what I'm building. And NanoChat, which you briefly mentioned, is the capstone project of LLM 101N, which is a class that I'm building.”

Claim: NanoChat is the capstone project for LLM 101N, a class Andrej Karpathy is building.

“So that's a really big piece of it. But now I have to build out a lot of the intermediates and then I have to actually like hire a small team of, you know, TAs and so on and actually like build the entire course.”

Claim: Andrej Karpathy says completing his course requires building many intermediate components and hiring a small team of teaching assistants.

“And maybe one more thing I would say is like many times when people think about education, they think about sort of like the more what I would say is like kind of a softer component of like diffusing knowledge or like but I actually have something very hard and technical in mind. And so in my mind, education is kind of like the very difficult technical.”

Claim: Andrej Karpathy views education as a difficult, technical undertaking rather than primarily as the softer process of diffusing knowledge.

“like a process of building ramps to knowledge.”

Claim: Andrej Karpathy defines education as the process of building ramps that help learners access knowledge.

“So in my mind, Nanachat is a ramp to knowledge because it's a very simple, it's like the super simplified full stack thing. If you give this artifact to someone and they like look through it, they're learning a ton of stuff.”

Claim: Andrej Karpathy describes Nanachat as a simple, simplified full-stack learning artifact that helps people gain substantial knowledge by exploring it.

“And so it's giving you a lot of what I call Eureka's per second, which is like understanding per second. That's what I want, lots of Eureka's per second.”

Claim: Karpathy wants learning experiences to produce many moments of understanding per unit of time, which he calls “Eurekas per second.”

“And so to me, this is a technical problem of how do we build these ramps.”

Claim: Karpathy views designing effective ramps to knowledge as a technical problem.

“ramps to knowledge and uh so i also think of eureka as almost like a it's not like maybe that different maybe through to some of the frontier labs or some of the work that's going to be going on because i want to figure out how to build these frontier these ramps very efficiently so that people are never stuck and everything is always not too hard or not too not too trivial and uh you can't you have just the right material to actually progress”

Claim: Karpathy wants to build learning pathways efficiently, keeping learners unstuck and presenting material at an appropriate level so they can progress.

“I mean, I think you always have to be calibrated to what the capability exists in the industry.”

Claim: Karpathy believes people should calibrate their use of AI to the capabilities currently available in the industry.

“And I think a lot of people are going to pursue like, oh, just ask chat GPT, etc.”

Claim: Karpathy expects many people to try using ChatGPT by simply asking it for help.

“between you say oh teach me ai there's no way it's like i mean it's going to give you some slop right like when i ai is never going to write nanachat right now but nanachat is a really useful i think intermediate point so i still i'm collaborating with ai to create all this material so ai is still fundamentally very helpful um earlier on i built a cs231n at stanford which was one of the earlier actually sorry i think it was the first deep learning class at stanford which became very popular um um”

Claim: Karpathy says that, at the time of the conversation, asking ChatGPT to teach AI would produce low-quality material, and AI could not yet create Nanachat, which he considers a useful intermediate learning resource.

“between you say oh teach me ai there's no way it's like i mean it's going to give you some slop right like when i ai is never going to write nanachat right now but nanachat is a really useful i think intermediate point so i still i'm collaborating with ai to create all this material so ai is still fundamentally very helpful um earlier on i built a cs231n at stanford which was one of the earlier actually sorry i think it was the first deep learning class at stanford which became very popular um um”

Claim: Karpathy says AI is very helpful in his collaboration to create course materials.

“And the difference in building out to 31N and LN101N now is quite stark because I feel really empowered by the LMs as they exist right now, but I'm very much in the loop.”

Claim: Karpathy says current language models make developing his newer course materials substantially different from his earlier CS231N work, while he remains closely involved.

“So they're helping me build little materials. I go much faster. They're doing a lot of boring stuff, etc.”

Claim: Karpathy says language models accelerate his course development by helping create small materials and handling much of the routine work.

“So I feel like I'm developing the course much faster and those LLM infuse in it, but it's not yet at a place where I can creatively create the content. I'm still there to do that.”

Claim: Karpathy says LLMs currently speed up course development but cannot yet creatively produce the course content, which he continues to create himself.

“So I think the trickiness is always calibrating yourself. your software exists.”

Claim: Karpathy says a persistent challenge is calibrating one’s approach to the software capabilities that currently exist.

“So I think it would change over time. So I think right now it would be hiring faculty to help work hand in hand with AI and a team of people probably to build a state of the art courses.”

Claim: Karpathy thinks that, at present, creating state-of-the-art courses would require faculty working alongside AI and probably a team of other people.

“And then I think over time it can maybe some of the TAs can actually become AIs because some of the TAs like, okay, you just take all the course materials. And then I think you could serve a very good automated TA for the student when they have more basic questions or something like that, right?”

Claim: Karpathy thinks AI systems may eventually serve as automated teaching assistants for students’ basic questions by using course materials.

“But I think you'll need faculty for the overall architecture of a course and making sure that it fits.”

Claim: Karpathy believes faculty will remain necessary to design a course’s overall structure and ensure its coherence.

“And maybe at some future point, you know, I'm not even that useful and AI is doing most of the design much better than I could. But I still think that that's going to take some time to play out.”

Claim: Karpathy allows that AI may eventually design most courses better than he can, but expects this transition to take time.

“I will hire faculty, I think, because there are domains in which I'm not an expert. And I think that's the only way to offer the state of the art experience for the student ultimately.”

Claim: Karpathy expects to hire faculty in fields outside his expertise, believing this is necessary to provide students with a state-of-the-art educational experience.

“Yeah, I do expect that I would hire faculty, but I will probably stick around in AI for some time.”

Claim: Karpathy expects to hire faculty while probably continuing to work in AI for some time.

“But I do have something I think more conventional in mind for the current capability I think that people would probably anticipate.”

Claim: Karpathy says his plans for current AI capabilities are more conventional than people might expect.

“And when I'm building Starfleet Academy, I do probably imagine a physical institution and maybe a tier below that a digital offering that is not the state of the art experience you would get when someone comes in physically full time.”

Claim: Karpathy probably envisions Starfleet Academy as a physical institution offering a higher-quality experience than its lower-tier digital option.

“And we work through material from start to end and make sure you understand it. That's the physical offering.”

Claim: Karpathy describes the physical offering as guiding students through the full course material and ensuring they understand it.

“The digital offering is, yeah, a bunch of stuff on the internet and maybe some LLM assistant and it's a bit more gimmicky and a tier below, but at least it's accessible to like 8 billion people.”

Claim: Karpathy envisions a lower-tier digital education offering with online materials and possibly an LLM assistant that could be accessible to roughly 8 billion people.

“Yeah, and I think there's going to have to be a lot of not just education, but also re-education, and I would love to help out there because I think the jobs will probably change quite a bit.”

Claim: The speaker expects jobs to probably change substantially, creating a need for both education and re-education.

“really good course to teach in this in this respect um and yeah i think the motivation wise uh before agi uh motivation is very simple to solve because people want to make money and this is how you make money in the industry today i think post agi is a lot more interesting um possibly because yeah if everything is automated and there's nothing to do for anyone why would anyone go to a school etc um”

Claim: Before AGI, the speaker thinks people are motivated to pursue education because it helps them earn money in the industry.

“really good course to teach in this in this respect um and yeah i think the motivation wise uh before agi uh motivation is very simple to solve because people want to make money and this is how you make money in the industry today i think post agi is a lot more interesting um possibly because yeah if everything is automated and there's nothing to do for anyone why would anyone go to a school etc um”

Claim: The speaker suggests that after AGI, if everything is automated and people have nothing to do, it may be unclear why they would attend school.

“So I think, I guess like I often say that pre-AGI, education is useful. Post-AGI, education is fun.”

Claim: The speaker characterizes education as useful before AGI and as fun after AGI.

“for example, people go to gym today, but we don't need their physical strength to manipulate heavy objects because we have machines that do that. They still go to gym. Why do they go to gym? because it's fun, it's healthy, and you look hot when you have a six pack.”

Claim: The speaker says people continue going to the gym despite machines replacing the need for human strength to move heavy objects, because exercise is fun and healthy and a six-pack is attractive.

“And so I kind of think that education will kind of play out in the same way. Like you'll go to school, like you go to gym and you'll and I think that right now, I think not that many people learn because learning is hard.”

Claim: The speaker expects education to become like gym-going, with people attending school for reasons beyond practical necessity.

“And so I kind of think that education will kind of play out in the same way. Like you'll go to school, like you go to gym and you'll and I think that right now, I think not that many people learn because learning is hard. you bounce from material because and some people overcome that barrier but for most people it's”

Claim: The speaker believes that learning difficulty causes many people to disengage from learning, especially when material is too easy or too hard.

“but I do think that we should it's a technical problem to solve it's a technical problem to do what my tutor did for me when I was learning Korean I think it's tractable and buildable and someone should build it and I think it's going to make learning anything like trivial and desirable and people will do it for fun because it's trivial”

Claim: The speaker believes it is technically feasible to build a tutor that makes learning any subject easy and appealing, and says someone should build it.

“If I had a turtle like that for any arbitrary piece of like knowledge I think it's gonna be so much easier to learn anything”

Claim: The speaker expects a capable tutor to make learning arbitrary subjects much easier.

“that I think this, so I do definitely feel like people will be, I do think like eventually it's a bit of a losing game. if that makes sense i do think that it is in long term long term which i think is longer than i think maybe most people in the industry it's a losing game i i do think that people can go so far and that we barely scratch the surface of much person can can go and that's just because people are bouncing off of material that's too easy or too hard and they and and i i actually kind of feel that people will be able to go much further like anyone speaks five languages because why not because it's so trivial”

Claim: The speaker thinks that in the long term, human efforts to influence technological progress through individual labor or cognition will probably become a losing game.

“long term which i think is longer than i think maybe most people in the industry it's a losing game i i do think that people can go so far and that we barely scratch the surface of much person can can go and that's just because people are bouncing off of material that's too easy or too hard and they and and i i actually kind of feel that people will be able to go much further like anyone speaks five languages because why not because it's so trivial”

Claim: The speaker believes people could achieve much greater cognitive development if they were not held back by learning material that is too easy or too hard.

“long term which i think is longer than i think maybe most people in the industry it's a losing game i i do think that people can go so far and that we barely scratch the surface of much person can can go and that's just because people are bouncing off of material that's too easy or too hard and they and and i i actually kind of feel that people will be able to go much further like anyone speaks five languages because why not because it's so trivial anyone knows, you know, all the basic curriculum of undergrad, etc.”

Claim: The speaker imagines that if learning became trivial, people could learn five languages and the basic undergraduate curriculum.

“And I kind of feel like I am betting a little bit implicitly on some of the timelessness of human nature. And I think it will be desirable to be. to do all these things. And I think people will look up to it as they have for millennia because and I think this will continue to be true.”

Claim: The speaker is betting that aspects of human nature are timeless and that people will continue to value physical and cognitive pursuits.

“And actually also maybe there's some evidence of that historically, because if you look at, for example, aristocrats or you look at maybe ancient Greece or something like that, whenever you had little pocket environments that were post-AGI in certain sense, I do feel like people have spent a lot of their time flourishing in a certain way, either physically or cognitively. And so I think I feel OK about the prospects of that.”

Claim: The speaker suggests that people in historically privileged settings, such as aristocratic societies or ancient Greece, often spent substantial time developing physically or cognitively.

“And I think if this is false and I'm wrong and we end up in like, you know, WALL-E or idiocracy future, then I think it's very, I don't even care if there's like Dyson spheres. This is a terrible outcome.”

Claim: The speaker considers a future resembling WALL-E or Idiocracy terrible, regardless of whether it includes Dyson spheres.

“Like I actually really do care about humanity. Like everyone has to just be superhuman in a certain sense.”

Claim: The speaker says they care deeply about humanity and wants everyone to become superhuman in some sense.

“I don't actually think that, I think there will be transitionary period where we are going to be able to be in the loop and, you know, advance things if we actually understand a lot of stuff.”

Claim: The speaker expects a transitional period in which people can stay involved in advancing technology, provided they understand a great deal.

“I do think that long term. That probably goes away, right?”

Claim: The speaker expects human involvement in advancing technology to probably disappear in the long term.