⚡️ARC-AGI-3: The Interactive Reasoning Benchmark

“we are a nonprofit that wants to act as the North Star towards AGI.”

Claim: Arc Prize Foundation is a nonprofit that aims to guide progress toward AGI.

“And the way that we do that is we build benchmarks,”

Claim: Arc Prize Foundation builds benchmarks as a way to guide progress toward AGI.

“because the way that I think about a benchmark, it's kind of like a target that's out there in the sky. And it says, hey, this is where we need to go. And this is where we need to incentivize research to go for it.”

Claim: A benchmark can serve as a target that indicates a desired research direction and incentivizes research to pursue it.

“So the first benchmark Francois Chalet came up with in 2019,”

Claim: François Chollet created the first benchmark in 2019.

“and he came out with this paper on the measure of intelligence,”

Claim: François Chollet published a paper about measuring intelligence.

“which is kind of, it's an interesting way to come out with a benchmark because he first started. to try to define intelligence and then you define a benchmark that can actually go and measure that definition of intelligence so he did that in 2019 and then in 2024 ike knoop one of the co-founders of arc prize he went to francois and he said francois either i'm wrong or this benchmark is drastically underrated”

Claim: François Chollet first defined intelligence and then designed a benchmark to measure that definition.

“to try to define intelligence and then you define a benchmark that can actually go and measure that definition of intelligence so he did that in 2019 and then in 2024 ike knoop one of the co-founders of arc prize he went to francois and he said francois either i'm wrong or this benchmark is drastically underrated”

Claim: In 2024, Mike Knopp told François Chollet that either Knopp was mistaken or the benchmark was drastically underrated.

“So Mike actually put he put up a million dollars of his own money and said, hey, I'm going to put up bounty. So for anybody who can beat this benchmark, they're going to get a million dollars. And that's where ArcPrize, the competition,”

Claim: Mike offered a one-million-dollar bounty from his own money to anyone who could beat the benchmark.

“they're going to get a million dollars. And that's where ArcPrize, the competition, came out in 2024.”

Claim: The ArcPrize competition began in 2024.

“And then this year we expanded the mission and went into a full blown nonprofit.”

Claim: In the year of the conversation, Arc Prize expanded its mission and became a full-fledged nonprofit.

“So now not only is it the competition, we're also doing a lot of the leaderboard stuff.”

Claim: Arc Prize's work now includes leaderboard activities in addition to running the competition.

“We're building another benchmark and we're about to come out with our first platform or at least a preview of the first platform on interacting with that benchmark.”

Claim: Arc Prize is developing another benchmark and plans to release a preview of its first platform for interacting with that benchmark soon.

“I think nothing happens in a vacuum. Right time the space is growing.”

Claim: The growth of the field contributed to the timing of Arc Prize's increased attention.

“the way that we think about our company is if we were a company or a nonprofit, if we're going to equate it to a startup.”

Claim: Arc Prize views its benchmark as its product, analogous to the product of a startup.

“startup the product that we have is the benchmark and so you're not going to convince the community you can't bs your way through a properly inappropriately designed benchmark and so the great work that francois did like the foundational research that he did in the flag he put in the ground that is what the differentiator was for it and so in 2024 we were solving for awareness and we're very happy where that ended at the end of last year we left 2024 with open ai”

Claim: A community cannot be persuaded by bluffing when a benchmark is properly designed.

“startup the product that we have is the benchmark and so you're not going to convince the community you can't bs your way through a properly inappropriately designed benchmark and so the great work that francois did like the foundational research that he did in the flag he put in the ground that is what the differentiator was for it and so in 2024 we were solving for awareness and we're very happy where that ended at the end of last year we left 2024 with open ai”

Claim: Arc Prize was satisfied with the outcome of its awareness-building effort by the end of 2024.

“startup the product that we have is the benchmark and so you're not going to convince the community you can't bs your way through a properly inappropriately designed benchmark and so the great work that francois did like the foundational research that he did in the flag he put in the ground that is what the differentiator was for it and so in 2024 we were solving for awareness and we're very happy where that ended at the end of last year we left 2024 with open ai”

Claim: Arc Prize's differentiator was the foundational research François Chollet conducted and the benchmark he established.

“startup the product that we have is the benchmark and so you're not going to convince the community you can't bs your way through a properly inappropriately designed benchmark and so the great work that francois did like the foundational research that he did in the flag he put in the ground that is what the differentiator was for it and so in 2024 we were solving for awareness and we're very happy where that ended at the end of last year we left 2024 with open ai”

Claim: Arc Prize's goal in 2024 was to build awareness.

“But Francois's definition of intelligence is not necessarily how well you do at any one test.”

Claim: Francois's definition of intelligence is not necessarily based on performance on a single test.

“Francois's definition of intelligence is what is your ability to learn new things?”

Claim: Francois defines intelligence as the ability to learn new things.

“So can you go and learn new things? So we already know that AI can beat chess and we know that can beat go.”

Claim: AI systems can beat humans at chess and Go.

“We know that it can do self-driving,”

Claim: AI systems can perform self-driving tasks.

“but those systems, they cannot generalize to other domains outside of their training data.”

Claim: The AI systems discussed cannot generalize to domains outside their training data.

“So the way that Francois calls it is skill acquisition,”

Claim: Francois calls the ability to generalize and learn new things skill acquisition.

“but then the kicker that comes with this, which is really important is skill acquisition efficiency.”

Claim: Skill acquisition efficiency is an important factor in evaluating intelligence.

“there's two things. Number one is the amount of energy required to actually go and learn new things.”

Claim: One denominator for measuring intelligence is the amount of energy required to learn new things.

“So the reason why this is so cool and important is because ArcPrice Foundation, we use humans as the benchmark for what general intelligence is because that's our only proof point of general intelligence out there.”

Claim: The ArcPrice Foundation uses humans as its benchmark for general intelligence because humans are its only available proof point of general intelligence.

“and we know how much energy the human brain takes you can literally measure like the number of calories and like wattage that the human brain actually does what's really cool is with that you have an intelligence output and you have an energy input so you know how much that is and you can directly compare that to the energy required for current ai like how much energy do they take for now the second denominator is how much training data do you need in order to output that intelligence that you see from there so”

Claim: The energy input required for current AI can be directly compared with the human brain's energy input in relation to intelligence output.

“and we know how much energy the human brain takes you can literally measure like the number of calories and like wattage that the human brain actually does what's really cool is with that you have an intelligence output and you have an energy input so you know how much that is and you can directly compare that to the energy required for current ai like how much energy do they take for now the second denominator is how much training data do you need in order to output that intelligence that you see from there so”

Claim: A second denominator for intelligence is the amount of training data needed to produce the observed intelligence.

“and we know how much energy the human brain takes you can literally measure like the number of calories and like wattage that the human brain actually does what's really cool is with that you have an intelligence output and you have an energy input so you know how much that is and you can directly compare that to the energy required for current ai like how much energy do they take for now the second denominator is how much training data do you need in order to output that intelligence that you see from there so”

Claim: The energy consumption of the human brain can be measured in calories and wattage.

“So humans, they do not have an internet's worth of training data in their training data, but yet they can still do generally intelligent things,”

Claim: Humans can perform generally intelligent tasks without having an internet's worth of training data.

“whereas we're not seeing that with AI right now.”

Claim: Current AI is not demonstrating the general ability to perform diverse intelligent tasks that humans demonstrate despite limited training data.

“So energy and training data are the two denominations where we use for intelligence.”

Claim: The speaker uses energy consumption and training-data requirements as two denominators for measuring intelligence.

“And I think one could argue that if you had a proper environment, then yes, you could RL on a whole bunch and humans are RRailing on the ultimate eval engine that there possibly is,”

Claim: With a proper environment, reinforcement learning could be applied across many domains.

“then yes, you could RL on a whole bunch and humans are RRailing on the ultimate eval engine that there possibly is, which is physics reality, That is the ultimate eval engine that we're all going for.”

Claim: Physical reality is the ultimate evaluation engine on which humans learn through reinforcement learning.

“Now, if you're not using physics or an eval engine, then you have to have a manufactured simulation RL environment.”

Claim: When physical reality is not used as an evaluation engine, a manufactured simulated reinforcement-learning environment is needed.

“And often what happens is the human or developer intelligence is often injected into that environment itself.”

Claim: Manufactured simulation environments often incorporate intelligence supplied by humans or developers.

“And so the model isn't actually intelligent. You're just almost like taking the intelligence from the developer, injecting it into the environment, and then you're transferring it over into the AI from there.”

Claim: The speaker argues that a model is not genuinely intelligent when its apparent intelligence comes from developer intelligence embedded in the environment and transferred to it.

“So the important part of the intelligence definition is your ability to learn new things, but that's an unseen task. tasks.”

Claim: A key part of intelligence is the ability to learn new things, including tasks that have not been seen before.

“So it's really difficult to create environments for things you can't anticipate, but yet humans are good at that, right?”

Claim: Creating environments for unanticipated tasks is difficult, whereas humans are good at learning to handle them.

“Like we come out the womb and all of a sudden we can, not all of a sudden, but throughout our lifetime, we can learn to drive, we can learn to play chess, we can learn English, we can learn all these different things.”

Claim: Over their lifetimes, humans can learn diverse skills such as driving, playing chess, and speaking English.

“And so generalization is the key piece of that.”

Claim: Generalization is a key component of intelligence as the ability to learn diverse new tasks.

“is what we're announcing either today or however maybe for this podcast or recording is we're coming out with Arc AGI 3. And what this is going to be is it's going to be a series of 100 different novel environments or you could simply call them 100 different novel games that we're making ourselves.”

Claim: Arc AGI 3 will consist of 100 novel environments or games created by its developers.

“So these are simple games, 2D games that humans can do that are really easy for humans to be able to play and intuit about them. But AI is still going to have a really, really hard time with it.”

Claim: Arc AGI 3's simple 2D games are easy for humans to play and intuit, but are expected to be very difficult for AI.

“I'm with this. Now, the reason why we're moving to actual games is because of this movement that goes from static benchmarks to interactive benchmarks.”

Claim: AI evaluation is moving from static benchmarks toward interactive benchmarks.

“My hypothesis is that when AGI is declared, it will happen via an interactive benchmark.”

Claim: The speaker hypothesizes that AGI will be declared through an interactive benchmark.

“We're not going to know that AGI is here just via a static benchmark.”

Claim: The speaker believes a static benchmark alone will not establish that AGI has arrived.

“And the reason why is because humans are very good at intuiting their environment, understanding what the goals are, making a plan, making a long horizon plan.”

Claim: Humans are good at intuiting their environment, identifying goals, and making both plans and long-horizon plans.

“And you're not going to get any of that from a static benchmark and you're required to have an interactive benchmark.”

Claim: Assessing environmental intuition, goal understanding, and long-horizon planning requires an interactive benchmark rather than a static benchmark.

“So that's what RKGI3 is going to be.”

Claim: Arc AGI 3 is an interactive benchmark.

“So here we have an example of one of our games and each one of the games has a has a fun name that goes along with it. So as a part of the preview, we're launching five games. Now, three of them are going to be public on day one and two of them are going to be private. Now,”

Claim: The preview will launch five games: three will be public on day one, and two will initially be private.

“the two are private is because we're having an agent competition so we want to see how good can the community be against trying to beat these things and so we'll release the two extra ones at the end of the competition it's only 30 days long so you'll get these immediately now”

Claim: The two initially private games are being withheld for a 30-day agent competition and will be released at the end of the competition.

“here we are looking at locksmith now you just look at this one screenshot and it doesn't make a ton of sense but that's on purpose because you need to explore in order for it to like into it in order to intuit what's going on so we'll show the same thing to ai except the ai is going to get a json grid of a list of lists so they'll just get a bunch of numbers 64 by 64 and they can choose to turn that into an image if they want to we're agnostic could do whatever you want with it if they want to do multi-model”

Claim: The games require players to explore in order to understand the environment's rules.

“is each new level should introduce a new game mechanic.”

Claim: Each new level in the game is intended to introduce a new game mechanic.

“Now, the reason why we introduce a new game mechanic is because we're testing your ability to learn on the fly.”

Claim: The game introduces new mechanics to test players' ability to learn on the fly.

“This is what humans are very good at is sample efficient learning,”

Claim: Humans are very good at learning efficiently from a small number of samples.

“what's interesting here though is even though i know the game that doesn't omit the fact that i need to pre-plan what i need to go do because like”

Claim: Knowing the game does not eliminate the need to plan actions in advance.

“we need to actually do some planning on this one and that's what humans are very good at so as you start thinking about COTs and reasoning here reasoning and AI is going to have to plan their actions and what they're going to have to go do for it so now that we've matched up the symbols okay great looks good now we go to this last one and this is where I'll stop right here let me make sure this matches up let me get the right color all right let me get some more life let me match the shape okay so I match the shape but notice that it's out of rotation right now”

Claim: Humans are very good at planning their actions.

“we need to actually do some planning on this one and that's what humans are very good at so as you start thinking about COTs and reasoning here reasoning and AI is going to have to plan their actions and what they're going to have to go do for it so now that we've matched up the symbols okay great looks good now we go to this last one and this is where I'll stop right here let me make sure this matches up let me get the right color all right let me get some more life let me match the shape okay so I match the shape but notice that it's out of rotation right now”

Claim: AI systems that reason will need to plan their actions and what they need to do.

“So we're forcing exploration. You're forcing long-term planning.”

Claim: The game is designed to force players to explore and plan over the long term.

“You're forcing understanding what the different actions do and how the actions map to the action space.”

Claim: The game requires players to learn the effects of different actions and how those actions map to the action space.

“And then you're getting just minimal sparse rewards if you actually did something.”

Claim: The game provides minimal, sparse rewards for player actions.

“jump out of here what agents will get is agents will get a series of frames and those frames will be 64 by 64 now generally it's just going to be one frame but you might be able to get like maybe two in a row or three in a row and that would show an animation and so beginning state middle state end state and so you can kind of get a perception of over time there what we ask for back is just one through six a series of one through six actions and it's really just integers and so you pass back a one two three four five or six”

Claim: The API gives agents 64-by-64 frames, usually one at a time, but may provide two or three consecutive frames to convey an animation over time.

“jump out of here what agents will get is agents will get a series of frames and those frames will be 64 by 64 now generally it's just going to be one frame but you might be able to get like maybe two in a row or three in a row and that would show an animation and so beginning state middle state end state and so you can kind of get a perception of over time there what we ask for back is just one through six a series of one through six actions and it's really just integers and so you pass back a one two three four five or six”

Claim: The API provides agents with 64-by-64 frames, usually one at a time, but it may provide two or three consecutive frames to represent an animation over time.

“jump out of here what agents will get is agents will get a series of frames and those frames will be 64 by 64 now generally it's just going to be one frame but you might be able to get like maybe two in a row or three in a row and that would show an animation and so beginning state middle state end state and so you can kind of get a perception of over time there what we ask for back is just one through six a series of one through six actions and it's really just integers and so you pass back a one two three four five or six”

Claim: Agents submit actions as integers from one through six.

“jump out of here what agents will get is agents will get a series of frames and those frames will be 64 by 64 now generally it's just going to be one frame but you might be able to get like maybe two in a row or three in a row and that would show an animation and so beginning state middle state end state and so you can kind of get a perception of over time there what we ask for back is just one through six a series of one through six actions and it's really just integers and so you pass back a one two three four five or six next one two three four five those are just like we call them basic actions and then six could represent a click so like six you submit a six action and a coordinate somewhere on the screen and that represents that you actually clicked on something on there now we we like a really scoped environment because that removes a ton of we'll call it it's really easy to analyze the learning efficiency for humans and ai if we scope down the environment really well so to your point yes it will just be two by two”

Claim: Agents submit actions as integers from one to six; action six can encode a click at specified screen coordinates.

“next one two three four five those are just like we call them basic actions and then six could represent a click so like six you submit a six action and a coordinate somewhere on the screen and that represents that you actually clicked on something on there now we we like a really scoped environment because that removes a ton of we'll call it it's really easy to analyze the learning efficiency for humans and ai if we scope down the environment really well so to your point yes it will just be two by two”

Claim: Tightly scoping an environment makes learning efficiency easier to analyze for both humans and AI.

“next one two three four five those are just like we call them basic actions and then six could represent a click so like six you submit a six action and a coordinate somewhere on the screen and that represents that you actually clicked on something on there now we we like a really scoped environment because that removes a ton of we'll call it it's really easy to analyze the learning efficiency for humans and ai if we scope down the environment really well so to your point yes it will just be two by two”

Claim: Tightly scoping the environment makes it easier to analyze learning efficiency in humans and AI.

“next one two three four five those are just like we call them basic actions and then six could represent a click so like six you submit a six action and a coordinate somewhere on the screen and that represents that you actually clicked on something on there now we we like a really scoped environment because that removes a ton of we'll call it it's really easy to analyze the learning efficiency for humans and ai if we scope down the environment really well so to your point yes it will just be two by two”

Claim: In the action interface, integers one through five represent basic actions, and integer six represents a click at a specified screen coordinate.

“Two in the beginning.”

Claim: The environment will initially be two-by-two.

“You know, we get that pushback all the time. We are agnostic as to how the model represents the data to themselves or anybody represents it. So we've had people turn the like we give a JSON grid.”

Claim: The team does not prescribe how a model or its users represent the data.

“You know, we get that pushback all the time. We are agnostic as to how the model represents the data to themselves or anybody represents it. So we've had people turn the like we give a JSON grid. We've had people turn that into emojis. We've had different text representations. We've had people turn it into pictures.”

Claim: Participants have transformed the provided JSON grid into emojis, other text representations, and pictures.

“we communicate these puzzles to humans via pictures because it's so much easier to do it via visual.”

Claim: The team presents puzzles to humans as pictures because it considers visual presentation much easier for people.

“It's a valid pushback we hear all the time. We're not concerned about it. about it because we're agnostic as to how you represent the data to the model yeah”

Claim: The team is not concerned by objections to visual representation because it does not prescribe how data is represented to the model.

“yeah yeah it's it's still true we have we have yet to have an AI successfully beat any level on any of these games so it hasn't happened yet”

Claim: At the time of the conversation, no AI had successfully beaten a level in any of the games.

“Yeah. And that's one of the reasons why we're having our agent competition too, because we want to know, we want to use the collective intelligence of the community and say, hey, what is, what are the types of, like, what are the types of scaffolds that are needed in order to be competitive?”

Claim: The agent competition is intended to draw on the community's collective intelligence to identify which scaffolds are needed for competitive performance.

“be competitive on this and what are the types of scores you can get for it so the way that we're going to judge the agent competition is we will have a general like a generalization score which is like how well do you do on the private test set we know people are going to overfit to the public stuff but we'll place a higher weighting on the private side but that's a good point and one of the things i'm actually really curious about is what role does scaffolding play in future agi”

Claim: The agent competition will score generalization using performance on a private test set and weight that private-set performance more heavily because participants are expected to overfit to public material.

“Now, I know that sounds kind of like a weird question, but my current hypothesis that AGI will be heavily scaffolded. And why do you have that?”

Claim: The speaker's current hypothesis is that AGI will rely heavily on scaffolding.

“my hypothesis is that you're going to need different components that are working together in order to get the effect that you want.”

Claim: The speaker hypothesizes that achieving desired AI capabilities will require multiple components working together.

“you need to have a concept of memory because certain actions that you took back then are going to inform your future actions.”

Claim: In the game being discussed, an agent needs memory because its past actions inform its future actions.

“probably not you're probably going to want to do some compression okay well then all of a sudden you have the beginning of a harness that's sitting right there so one of the questions i have going into the agent competition is what is the minimal viable harness that you need in order just to swap out the model and see how they do on this and then if you're going to have the porsha of harnesses like if you're going to go crazy what are you going to go do for it and so that's why we're going to throw up a little bit of prize money incentivize people and see what they do for it”

Claim: The agent competition will investigate the smallest viable harness that allows models to be swapped out and evaluated on the games.

“I think that the jury is still out for that, but I think that, well, yeah, the jury's still out. I could argue both ways for it.”

Claim: The speaker considers it unresolved whether AI systems will rely on scaffolding and sees arguments on both sides.

“if a provider drops a price, yes, we will update our pricing. We just want to reflect reality.”

Claim: ARC-AGI will update its reported pricing when a model provider lowers the model's price, so the reporting reflects the current price.

“So we use cost as a proxy for that because in theory, cost is market efficient. Now it's not. That's not perfect because it's not a complete commodity yet, but in theory,”

Claim: ARC-AGI uses cost as a proxy for the energy and training-data inputs of closed models; the proxy is imperfect because the market is not yet a complete commodity, despite the theoretical expectation that cost is market-efficient.

“What's cool about interactivity is we get a new efficiency metric. We actually get action efficiency.”

Claim: ARC-AGI 3 can measure action efficiency as a new efficiency metric enabled by interactive tasks.

“In fact, that's one of the templates we'll launch with is just like a random agent. It's going to be extremely inefficient.”

Claim: ARC-AGI 3 is planned to launch with a random-agent template, which the speaker expects to be extremely inefficient.

“And one of the quality checks that we do is we run a random agent at a million steps to see if it beats it or not. And no, it doesn't beat lockstep at all or locksmith. And so what we'll do is when we come out with Arc AGI.”

Claim: ARC-AGI's quality check runs a random agent for one million steps, and the agent does not beat the Locksmith benchmark.

“AGI 3 in Q1 of next year 2026 we're going to go and test hundreds of people on how they do on locksmith right here and what's nice is that you can get on the x-axis you're going to get number of actions and on the y-axis you're going to get levels completed until game over and you're going to get a little line that goes up and up and up and up and that slope you can kind of think of that as you're learning efficiency right if you take a lot of actions to go a little bit of levels well you're not really learning the rules of the game and we're going to know who's the best we're going to know where it lands”

Claim: In ARC-AGI's Locksmith evaluation, the slope relating actions taken to levels completed represents learning efficiency; many actions for few completed levels indicate that the player is not learning the game rules effectively.

“AGI 3 in Q1 of next year 2026 we're going to go and test hundreds of people on how they do on locksmith right here and what's nice is that you can get on the x-axis you're going to get number of actions and on the y-axis you're going to get levels completed until game over and you're going to get a little line that goes up and up and up and up and that slope you can kind of think of that as you're learning efficiency right if you take a lot of actions to go a little bit of levels well you're not really learning the rules of the game and we're going to know who's the best we're going to know where it lands”

Claim: ARC-AGI plans to test hundreds of people on Locksmith in Q1 2026, plotting the number of actions taken against the number of levels completed before game over.

“Now the brute force agent, the random agent that I was just talking about, that's going to be a flatline all the way out. It's not even going to be even close.”

Claim: The speaker expects the random agent's Locksmith performance to remain flat as it takes more actions and to fall far short of human performance.

“So when we report learning efficiency for this, especially with AI versus humans, it's all going to be around how many actions do you take in order to complete the goal of the environment,”

Claim: ARC-AGI will measure learning efficiency, particularly when comparing AI with humans, by counting the actions required to complete an environment's goal.

“which not only does that encompass learning what the environment entails, but also executing what you perceive to be the goal of the environment as well. So it's almost learning and execution there.”

Claim: ARC-AGI's action-based learning-efficiency measure captures both understanding an environment and executing the perceived goal within it.

“stuff yeah so what's actually really cool is so the one game i showed you is agent based like you have a little thing that's moving around we have a requirement that games must be novel from each other we have a large percentage of games that are non-agent based so think of it as like like solitaire or connect four or like simon or memory or something like that those are non-agent based games and so game mechanics will be completely different across those now to your other point we are also designing some games”

Claim: ARC-AGI requires its games to be novel from one another, and many games will be non-agent-based, including examples such as solitaire, Connect Four, Simon, and memory.

“that require cooperation with other things in the environment to be able to complete the goal.”

Claim: ARC-AGI is designing games in which completing the goal requires cooperation with other entities in the environment.

“And so you can think of it as if we have 100 different games, we can tag different levels in different games with different skills required in order to complete that thing. One of those skills may be cooperation or alignment with other things in order to complete the common goal.”

Claim: ARC-AGI can label game levels by the skills needed to complete them, including cooperation or alignment with other entities toward a shared goal.

“So just making this up, for example, kind of like Prisoner's Dilemma, if you were to just go right for the exit or something, you're going to get a lower score. than if you were to cooperate, or maybe you can't even complete the game unless you go and cooperate,”

Claim: In a hypothetical Prisoner's Dilemma-like ARC-AGI game, choosing the exit instead of cooperating would yield a lower score, and completing the game might require cooperation.

“And so we don't have plans yet to have multi-agent in the fact of like two LLMs battling against each other because we're really kind of focused on the one, but yet we may have deterministic agents out there in the environment that are kind of more code and program based.”

Claim: ARC-AGI has no current plan for multi-agent games in which two LLMs compete, because its focus is on a single agent; it may nevertheless include deterministic agents implemented through code or programs in the environment.

“The other thing I wanted to show you really quick is going back to that learning efficiency piece. We're actually taking inspiration from a Josh Tenenbaum paper that came out. um in 2000 i think it was 2021 and they actually this is this is a little bit of the same thing they they're not doing the exact same test that we have but you can see that there's number of steps for these games number of steps taken by agents and the number of levels that they were able to do and then they started to differentiate between the um like human players and ai players now our games are going to be different it's going to look completely different but this is the chart that i have in mind for when we go and start to report the efficiencies here”

Claim: ARC-AGI's learning-efficiency approach draws inspiration from a Josh Tenenbaum paper that the speaker believes was published in 2021.

“um in 2000 i think it was 2021 and they actually this is this is a little bit of the same thing they they're not doing the exact same test that we have but you can see that there's number of steps for these games number of steps taken by agents and the number of levels that they were able to do and then they started to differentiate between the um like human players and ai players now our games are going to be different it's going to look completely different but this is the chart that i have in mind for when we go and start to report the efficiencies here”

Claim: The referenced paper charts agent steps against completed game levels and separates results for human and AI players.

“um in 2000 i think it was 2021 and they actually this is this is a little bit of the same thing they they're not doing the exact same test that we have but you can see that there's number of steps for these games number of steps taken by agents and the number of levels that they were able to do and then they started to differentiate between the um like human players and ai players now our games are going to be different it's going to look completely different but this is the chart that i have in mind for when we go and start to report the efficiencies here”

Claim: Although ARC-AGI's games will be different from those in the referenced paper, the speaker intends to use a similar chart as a model for reporting efficiency.

“It's kind of a balance of what do we need for statistical significance versus what is realistic from an operations perspective in order to actually go and execute these.”

Claim: The choice of 120 games balances statistical-significance needs with operational feasibility.

“we actually have a team of seven game developer contractors right now.”

Claim: ArcPrize currently has seven game-developer contractors.

“Francois made an architecture document that's basically this is what makes a good game. This is why. why it's going to measure intelligence.”

Claim: Francois created an architecture document that explains the qualities of a good game and why such a game can measure intelligence.

“Then we take that and then we turn it into a game idea and game spec and we go and hand that to one of our game developer contractors.”

Claim: The team converts the architecture document into a game idea and specification and assigns the work to a game-developer contractor.

“They make it for a week. We go and test it. We do some light testing, some QA. Is it a good game, you know, et cetera, et cetera.”

Claim: A game-development contractor spends about a week building a game, and the team then tests it and conducts light quality assurance.

“Our goal is to come out with 120 by Q1 of next year.”

Claim: The team aims to produce 120 games by Q1 of the next year.

“And so that means that we need to make 20 per month.”

Claim: Producing 120 games by the stated deadline requires a pace of 20 games per month.

“We need to make five per week because there's four weeks in a month.”

Claim: The stated production pace requires five games per week, assuming four weeks per month.

“So we need five or six game devs that are helping us making these. And so we got the assembly line.”

Claim: The team needs five or six game developers to support game production.

“And the problem with that is that we don't want to incentivize AI to derive the program that made the game.”

Claim: The team does not want its benchmark setup to incentivize AI systems to infer the program that generated a game.

“And so if we continue to have humans make the game, then the AI is incentivized to try to reverse engineer the G inside of humans. And that's kind of the whole point of what we're trying to do here.”

Claim: Having humans make the games incentivizes AI to reverse-engineer the G in humans, which the speaker says is the project's purpose.

“it's an operational limit mixed with what do we need for data significance?”

Claim: The 120-game target is constrained by operational capacity and the data needed for statistical significance.

“So like we have three FTEs on ArcPrize right now. I'm one of them.”

Claim: ArcPrize currently has three full-time employees, one of whom is the speaker.

“and it's kind of like gna and like like everything else so i run archives right now and then we have a very awesome generalist engineer he's an absolute rock star his name is david he's doing everything from serving the api to user off to management to game serving to qa to all that other good stuff and then we have a lead game dev his name is hunter and he is the one who looks over and kind of does the quality check and idea inspiration for all the different game devs from there now mike and francois are still very involved but they're not full time they're on the board they”

Claim: Mike and Francois remain involved with ArcPrize as board members but do not work on it full-time.

“and it's kind of like gna and like like everything else so i run archives right now and then we have a very awesome generalist engineer he's an absolute rock star his name is david he's doing everything from serving the api to user off to management to game serving to qa to all that other good stuff and then we have a lead game dev his name is hunter and he is the one who looks over and kind of does the quality check and idea inspiration for all the different game devs from there now mike and francois are still very involved but they're not full time they're on the board they”

Claim: ArcPrize's lead game developer, Hunter, oversees quality checks and provides idea inspiration for the game developers.

“and it's kind of like gna and like like everything else so i run archives right now and then we have a very awesome generalist engineer he's an absolute rock star his name is david he's doing everything from serving the api to user off to management to game serving to qa to all that other good stuff and then we have a lead game dev his name is hunter and he is the one who looks over and kind of does the quality check and idea inspiration for all the different game devs from there now mike and francois are still very involved but they're not full time they're on the board they”

Claim: ArcPrize's generalist engineer David handles API serving, user-facing work, management, game serving, and quality assurance.

“or that comes with it so our press and our engineering team is one person”

Claim: The speaker says ArcPrize's press and engineering team consists of one person.

“yeah you know we're opportunistic with that as a non-profit we are very budget conscious and mindful for what we can we're not we don't um we can like raise the whole bunch of cash for it so i would say it's also a bit of financial driven as well and that's one of the things you know it's on my plate as well as going in fundraising”

Claim: ArcPrize approaches hiring opportunistically and is budget-conscious as a nonprofit, with limited ability to raise substantial funds for the work.

“yeah you know we're opportunistic with that as a non-profit we are very budget conscious and mindful for what we can we're not we don't um we can like raise the whole bunch of cash for it so i would say it's also a bit of financial driven as well and that's one of the things you know it's on my plate as well as going in fundraising raising for our price with philanthropic donors.”

Claim: The speaker is responsible for fundraising for ArcPrize from philanthropic donors.

“So I would say have folks reach out. We may not have a spot. We may have a spot. We'll see how it goes.”

Claim: People interested in working with ArcPrize should reach out, although an opening may or may not be available.

“I tell you what, my view on it is at the Grok4 live stream last night, Elon actually said something along these lines, which I agree with, which is reality is the ultimate eval engine.”

Claim: The speaker agrees with Elon Musk’s view that reality is the ultimate evaluation engine for AI.

“And if I could have any benchmark possible, I would want a perfect simulator of reality. and of course we wouldn't but like a perfect simulator of reality and then that's when you go and that's what you go and simulate on that's your environment that you can just manufacture different tests for that in lieu of that we have approximations that happen every now and then so the way i think about it is if you had like a linear spectrum rkg i1 and 2 it's a static list of like three or four json grids static doesn't move rkg i3 i guess on the other side would be reality like pure representation of reality rkg i3 is a step more towards reality but it's still going to be We're going to be a scoped environment.”

Claim: The speaker would prefer a perfect simulator of reality as an AI benchmark, while acknowledging that one would not be available.

“and of course we wouldn't but like a perfect simulator of reality and then that's when you go and that's what you go and simulate on that's your environment that you can just manufacture different tests for that in lieu of that we have approximations that happen every now and then so the way i think about it is if you had like a linear spectrum rkg i1 and 2 it's a static list of like three or four json grids static doesn't move rkg i3 i guess on the other side would be reality like pure representation of reality rkg i3 is a step more towards reality but it's still going to be We're going to be a scoped environment.”

Claim: The speaker characterizes ARC-AGI 1 and 2 as static lists of a few JSON grids and ARC-AGI 3 as a more reality-like but still scoped environment.

“I do know that ArcGIS 4 or 5 or whatever it may be will need to allow us to have more axes of freedom that are above a 2D 64 by 64 type of grid that comes from there.”

Claim: The speaker expects ARC-AGI 4 or 5 to provide more degrees of freedom than the two-dimensional 64-by-64 grid format.

“But one thing I do want to emphasize here is one of the things that sets us apart from other benchmark creators is we don't aim for PhD plus plus problems like the hardest possible thing that nobody understands. Our anchor and our constraint, which is actually really freeing, is can humans do this, right?”

Claim: The speaker says their benchmark team prioritizes problems humans can solve rather than maximally difficult problems that nobody understands.

“Because our hypothesis and our definition of AGI is as long as we can come up with problems that humans can do and AI cannot, then we do not have AGI.”

Claim: Under the speaker’s definition, AGI has not been achieved as long as humans can solve problems that AI cannot.

“And then the flip side of that is also true, which is when us as ArcPrize, we're like, we consider ourselves like our job is to come up with problems that humans can do and AI cannot. When we can no longer do that. For all intents and purposes, that's practically AGI.”

Claim: The speaker considers it practically AGI when ArcPrize can no longer identify problems humans can solve that AI cannot.

“And the fact that ARC2 is still out there and humans can do them and the fact that ARC3 is going to be out there and humans can do them proves that we do not yet have AGI.”

Claim: The speaker argues that because humans can solve ARC-AGI 2 and ARC-AGI 3, AGI has not yet been achieved.

“So the other thing I do know about ARC4, and this is where I'll wrap it up, is that ARC4 will still be doable for humans, but it's still going to be hard for AI,”

Claim: The speaker expects ARC-AGI 4 to remain solvable by humans while still being difficult for AI.

“but it's going to need to have more access and freedom for us to test.”

Claim: The speaker says ARC-AGI 4 will need to allow greater access and freedom for testing.

“yeah so i will say that the v3 timeline also depends on how v2 goes so like right now like you said if we end the end of the year and it's 16”

Claim: The speaker says the timing of ARC-AGI 3 depends on the performance and trajectory of ARC-AGI 2.

“We don't necessarily have a timeline on V2.”

Claim: The speaker says they do not have a definite timeline for ARC-AGI 2.

“My hypothesis is it's not going to get beat in 2025.”

Claim: The speaker hypothesizes that ARC-AGI 2 will not be surpassed during 2025.

“i'm not sure i'm very i'm more confident to say that if you have a base llm like a gpt5 if gpt5 is a base llm without all the routing and cot and all that crazy stuff uh it's not gonna it's i'm pretty confident it's not gonna be um arc2”

Claim: The speaker is fairly confident that a base GPT-5 model, without routing, chain-of-thought, or similar additions, will not beat ARC-AGI 2.

“would an o5 be needed for it i i don't know but o5 is not coming out this year is my guess you know it's going to be coming out next year so”

Claim: The speaker guesses that O5 will not be released in the current year and will be released the following year.

“I guess long-winded saying is I don't know, but my guess is it's not going to be beat for the next 12 months.”

Claim: The speaker guesses that ARC-AGI 2 will not be surpassed during the next 12 months.

“And then v3, our durability estimate for that is three years. That's what we're aiming for is 36 months for v3. That's kind of wild in this time frame. Who knows? That's our hypothesis. That's what we're aiming for.”

Claim: The speaker says the team’s durability estimate and target for ARC-AGI 3 is 36 months without being beaten.

“We told the big lab that our timeline for v3 was 36 months for it not to get beaten. And I won't say which lab it was, but we were basically laughed out of the room when we told them that.”

Claim: The speaker says a major AI lab laughed when the team said it expected ARC-AGI 3 to remain unbeaten for 36 months.

“that's that's exactly how i think about it and like i'm even hesitant to say the words like beat arc agi because it's like we're not putting up this test as as like a like look at us go and try to beat this thing it's like it is a tool to incentivize research like that is why we put up a million dollars to see who can actually beat it because our hypothesis is the thing that thing that actually does beat ArcAGI 2 has something worthy in it that can be applied to other types of domains.”

Claim: The speaker hypothesizes that a method capable of beating ARC-AGI 2 will contain an approach applicable to other domains.

“that's that's exactly how i think about it and like i'm even hesitant to say the words like beat arc agi because it's like we're not putting up this test as as like a like look at us go and try to beat this thing it's like it is a tool to incentivize research like that is why we put up a million dollars to see who can actually beat it because our hypothesis is the thing that”

Claim: The speaker says ArcPrize created ARC-AGI as a research incentive and offered a $1 million prize to encourage attempts to beat it.

“And so in ArcPrize 2024, we had over 40 papers submitted for our best paper award to try to beat ArcAGI.”

Claim: ArcPrize received over 40 submissions for its 2024 best-paper award from researchers attempting to beat ARC-AGI.

“And that's one of the most rewarding things for us because that's open research that's out there.”

Claim: The speaker considers the open research generated through ArcPrize’s award one of the team’s most rewarding outcomes.

“And so one of the things that came out of that was the whole test time fine-tuning adaptation for that. And so as test time compute was coming around, a form of that is test time fine-tuning.”

Claim: The speaker says research around ARC-AGI produced work on test-time fine-tuning and adaptation.

“And one of the, and that was a huge proponent with an ARC prize of last year's as we think about what, as I think about it, it's almost like I don't want ARC prize beat per se. Yeah, sure. That's what's going to happen, but I want cool research to happen because of ARC.”

Claim: The speaker’s priority is for ARC to stimulate valuable research, rather than for ARCPrize simply to be beaten.

“like tactically people ask me like what what am i trying to do or what is like what's the goal it's like well once agi is found i think arc prize is setting itself up for a very good place to declare to be the one to declare that agi is actually here”

Claim: The speaker thinks Arc Prize is positioning itself to declare when AGI has arrived.

“Because someone's going to have to do it. And like, maybe it might be like the Turing test where it kind of just happens and goes. I doubt that that's actually going to be it. I think there's going to be a before and after moment.”

Claim: The speaker doubts that AGI will be recognized through a gradual process like the Turing test and expects a distinct before-and-after moment.

“And I would love ArcPrize to be in the place to validate that.”

Claim: The speaker wants Arc Prize to be positioned to validate that AGI has arrived.

“I would want to look back at the learning path and research path towards that and say ArcPrize had a role in accelerating that progress.”

Claim: The speaker wants Arc Prize to have helped accelerate the learning and research progress leading to AGI.

“Any AGI definition that involves money has ulterior motives.”

Claim: The speaker believes that any definition of AGI based on money has ulterior motives.

“Money has nothing to do with intelligence.”

Claim: The speaker believes that money is unrelated to intelligence.

“I would argue that AGI is here once an artificial machine can match the learning efficiency and generalization of an actual human.”

Claim: The speaker defines AGI as the point at which an artificial machine matches a human's learning efficiency and generalization.

“And to me, that's a pretty binary place to go for it. I agree that there's going to be spectrums and levels of that. We're already seeing it with the progress on RKG AI too.”

Claim: The speaker considers AGI's arrival a fairly binary threshold, while acknowledging that capabilities leading to it vary across spectrums and levels.

“They're showing non-zero levels of fluid intelligence.”

Claim: The speaker says RKG AI demonstrates non-zero fluid intelligence.

“But we will be able to look back at a point in which machines were. We're basically able to learn faster or quicker or at the same level as humans.”

Claim: The speaker predicts that people will be able to identify a point when machines learned faster than or at the same rate as humans.

“yeah totally so i now i've had the luxury of attending two labs of live streams here and so for the open ai live stream in december and then grox here they called us up today's thursday they called us up on tuesday and they said hey we got a new model we want to test it we say yep sure this sounds good we have a kind of a standard testing procedure with them so like no data retention”

Claim: The speaker says Arc Prize's standard testing procedure for xAI's new model included no data retention.

“attention and the model that you that we test should be the one that's going to go out to the public so that people can actually reproduce these results and they said yep cool no problem they gave us a little bit of credits to go for it which is always very happily accepted and then tested it and one of the things that we do is we say what is the score that you're claiming because they do their own self-testing and we want to see what what's their score so that we can go and validate it and they said their public eval scores yep we we ran it on semi-private it worked out looked great”

Claim: The speaker says xAI's tested model was intended to be the public release model so that the results could be reproduced.

“attention and the model that you that we test should be the one that's going to go out to the public so that people can actually reproduce these results and they said yep cool no problem they gave us a little bit of credits to go for it which is always very happily accepted and then tested it and one of the things that we do is we say what is the score that you're claiming because they do their own self-testing and we want to see what what's their score so that we can go and validate it and they said their public eval scores yep we we ran it on semi-private it worked out looked great”

Claim: The speaker says Arc Prize checked xAI's claimed public evaluation scores by testing the model on a semi-private evaluation, where the results were consistent with the claims.

“going from there and then they announce the results it's great i was sitting there in the audience with my laptop like getting the arc price tweet ready and then click enter right right when i saw the slide come up and click merge on the website push when we go from there yeah i mean it was great so we love when labs pull us in early to help them validate scores um because that really gives kind of a public boost and a public like vote of confidence that arcagi like has the merit to like they're saying that yes we trust arcagi to help describe the intelligence of these models”

Claim: The speaker says early involvement by AI labs in score validation gives Arc AGI public credibility as an evaluator of model intelligence.

“tour and so i quickly realized that and then pivoted over to the v3 conversation and so i said hey we're coming out v3 next year it's going to be a series of 100 video games and he immediately did his like elon like i can't do it but he did his elon like eyebrow raise like right”

Claim: The speaker said Arc Prize V3 was planned for the following year as a series of 100 video games.

“grok really needs to do is grok needs to play the video game and then iterate on the game and then play it and then iterate and then play it and so i knew that that would probably hit with him and i said hey man we're i mean we're making 100 video games here that are going to be easy for humans hard for ai”

Claim: The speaker's team is making 100 video games designed to be easy for humans and difficult for AI.

“grok really needs to do is grok needs to play the video game and then iterate on the game and then play it and then iterate and then play it and so i knew that that would probably hit with him and i said hey man we're i mean we're making 100 video games here that are going to be easy for humans hard for ai”

Claim: Grok needs to improve a video game through repeated cycles of playing it and iterating on it.

“oh we're doing the developer preview next week and then he raised his second eyebrow right at the right at that point and so he was interested in i think you know he's a gamer himself and so we're hoping to talk talk to him more about it he's excited and so we'll see how it goes”

Claim: The speaker hopes to discuss the project further with Elon, believes he is excited, and remains uncertain about how it will go.

“oh we're doing the developer preview next week and then he raised his second eyebrow right at the right at that point and so he was interested in i think you know he's a gamer himself and so we're hoping to talk talk to him more about it he's excited and so we'll see how it goes”

Claim: The speaker believes Elon was interested in the project and says Elon is a gamer.

“oh we're doing the developer preview next week and then he raised his second eyebrow right at the right at that point and so he was interested in i think you know he's a gamer himself and so we're hoping to talk talk to him more about it he's excited and so we'll see how it goes”

Claim: The speaker's team plans to hold the developer preview the following week.

“it was cool to have that experience with Elon just because of his prominence in the field and in industry in general, shake his hand,”

Claim: The speaker valued meeting Elon because of Elon's prominence in the field and in industry generally.

“But like if we want to get XAI. I, to come jump on board with us and like come along with this journey for us. Elon like wouldn't necessarily be the person necessary who's going to be doing the operational execution of getting that to happen, right?”

Claim: If the speaker's team seeks xAI's participation, Elon would not necessarily handle the operational execution of making it happen.

“Like it's going to be Jimmy and team and all those other folks. And so we're excited if we can have Elon support either way, we're excited to have Jimmy support and the rest of the team.”

Claim: The speaker expects Jimmy, his team, and others to handle operational execution, and welcomes support from Elon, Jimmy, and the wider team.

“i mean that's what i mean that's what it looks like right now i mean i tell you i'm with everybody else when you look at the benchmarks it's like okay great let me get this thing inside a cursor and what's the deal let me get this thing inside of let me go in the chat let me see like actually how it does right so the vibes seem great right now i'm excited to see more adoption that comes from it you know i think it still needs to be investigated and researched as to the reason the rl paradigm like you saw that they 10x the rl that they put on top of grok4 for that and so”

Claim: The speaker says Grok 4's current reception seems very positive and is eager to see whether adoption increases.

“i mean that's what i mean that's what it looks like right now i mean i tell you i'm with everybody else when you look at the benchmarks it's like okay great let me get this thing inside a cursor and what's the deal let me get this thing inside of let me go in the chat let me see like actually how it does right so the vibes seem great right now i'm excited to see more adoption that comes from it you know i think it still needs to be investigated and researched as to the reason the rl paradigm like you saw that they 10x the rl that they put on top of grok4 for that and so”

Claim: The speaker says the rationale for Grok 4's reinforcement-learning approach needs further investigation and notes that the amount of RL applied on top of Grok 4 was increased 10-fold.

“So how does that bleed down in actual performance and where are the gotchas and where are the nooks and crannies in the corners and where's the spiky intelligence actually spike?”

Claim: The speaker says it remains unclear how Grok 4's benchmark performance translates to practical use, what its gotchas are, and where its uneven or spiky capabilities appear.

“I think there's still a lot of open questions for it, but either way right now we're seeing freaking awesome performance.”

Claim: The speaker says Grok 4 is currently performing extremely well, despite unresolved questions about its capabilities.

“I heard rumors that Grok doesn't want to release the coding model until it's better than one specific other lab out there.”

Claim: According to rumors the speaker has heard, Grok plans to delay releasing its coding model until it outperforms a specific competitor's model.

“So they're going to wait and see when it's actually better for them so they can have that marketing point.”

Claim: The speaker says Grok would wait to release its coding model until it is better than a competitor's, enabling a marketing claim based on that advantage.

“yeah i mean sure that makes sense you know but also like the fact that they can credibly do it you know kind of come from a standing start it's really it's really i think commendable i think it's there's there's a lot of stories to be told from inside of x like how they did this because you know there's just a lot of data they have to get and you can't all get it from twitter because like god knows twitter is not the best source of information”

Claim: The speaker says X needed to acquire substantial data to build its AI capabilities, that Twitter alone could not provide all of it, and that Twitter is not the best information source.

“Well, I think it's really if you think about the ablation study about what does it take to get a good model out there? I mean, obviously, there's people, there's money, there's compute, there's there's data, there's all these other things.”

Claim: The speaker identifies people, money, compute, data, and other resources as inputs needed to produce a good AI model.

“And it's super interesting to have AGI labs out there that are frustrated with their external performance. And there's other labs out there who are very happy with their external performance.”

Claim: The speaker observes that some AGI labs are dissatisfied with their externally visible performance, whereas other labs are satisfied with theirs.

“And what's the difference between the two? And they all took different routes together.”

Claim: The speaker says AI labs followed different approaches and asks what accounts for the difference between labs satisfied with external performance and those dissatisfied with it.

“I mean, more or less, I mean, they've been at this for not for not too long. They're literally building the ship as they're doing it. And so it's very impressive for what they have.”

Claim: The speaker says xAI has been working on its AI effort for a relatively short time and has been building its organization as it goes, making its results impressive in that context.

“And I think other labs may look at this and be like, man, what is going on there that we want that type of success too.”

Claim: The speaker thinks other AI labs may study xAI's success to identify practices they could adopt to achieve similar results.

“My only call to action for folks is we want agent builders.”

Claim: The speaker invites people to participate as agent builders.

“And so we're going to come out with three different games. I would love if you could build agents however the heck you want using whatever tools you want, RL based, LLM based or whatever it is.”

Claim: The speaker says the team plans to release three games and invites people to build agents for them using any tools, including reinforcement-learning-based or LLM-based approaches.

“We're going to have about a $10,000 prize pool and put up some money for it. It actually may be more valuable. We're going to like like push all like on our every single social channel that we have push every single top performing agent so i would love if you guys would participate with that call out to the audience for that beautiful”

Claim: The speaker says the agent-building competition will have an approximately $10,000 prize pool, possibly more, and that every top-performing agent will be promoted on the team's social channels.