⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

“I played Factorio for years, and about 18 months ago I was looking into this idea of the paperclip maximizer, that kind of cautionary tale where if you set a model to maximize paperclips and maximize a factory,”

Claim: The speaker had played Factorio for years before working on the Factorio learning environment.

“I played Factorio for years, and about 18 months ago I was looking into this idea of the paperclip maximizer, that kind of cautionary tale where if you set a model to maximize paperclips and maximize a factory, what kind of bad stuff do they end up doing? in furtherance of that goal.”

Claim: About 18 months before speaking, the speaker considered the paperclip maximizer a cautionary tale about what harmful actions a model might take when tasked with maximizing paperclips and factory output.

“And it kind of got me thinking about the applicability of Factorio as an environment for benchmarking models.”

Claim: The paperclip-maximizer idea led the speaker to consider Factorio as an environment for benchmarking models.

“One of the kind of motivations for it is that when you compare Factorio against Minecraft, for instance, the complexity scale is drastically different.”

Claim: The speaker says Factorio's complexity scale is drastically greater than Minecraft's.

“In order to complete Factorio, you know, complete and launch a rocket, you need to mine maybe about a million more resources to get to that point.”

Claim: The speaker estimates that completing Factorio by launching a rocket requires mining about one million resources.

“Whereas the same kind of point in Minecraft, you have to just get maybe 200 resources.”

Claim: The speaker estimates that reaching a comparable point in Minecraft requires about 200 resources.

“So, you know, we have orders of magnitude difference in Factorio compared to some of these other environments. And if you want to go.”

Claim: The speaker says Factorio involves orders of magnitude more resources than some other game environments.

“go beyond just launching a factory and you know see some of the huge factories that real players have managed to make i think the biggest one at the moment is generating something like 19 million resources a second so we have factories that you know stem from you know making maybe one resource every couple of seconds all the way to 18 20 million a second”

Claim: The speaker says Factorio factories range from producing about one resource every couple of seconds to 18–20 million resources per second.

“go beyond just launching a factory and you know see some of the huge factories that real players have managed to make i think the biggest one at the moment is generating something like 19 million resources a second so we have factories that you know stem from you know making maybe one resource every couple of seconds all the way to 18 20 million a second”

Claim: At the time of speaking, the speaker estimated that the largest Factorio factory generated about 19 million resources per second.

“And that range gives an enormous amount of signal for comparing models,”

Claim: The speaker says Factorio's wide range of factory outputs provides substantial signal for comparing models.

“you know, from tiny models that aren't very good all the way to superhuman models and enable us to differentiate them kind of at any step along the way.”

Claim: The speaker says Factorio can distinguish model performance across a range from weak small models to superhuman models.

“And, you know, I kind of set out on a Friday night about 18 months ago, you know, as a casual weekend project, you know, in and out operation, get it done.”

Claim: The speaker began developing the Factorio benchmarking project as a casual weekend project about 18 months before speaking.

“And 18 months later, we kind of got it into a position. to actually release.”

Claim: The Factorio benchmarking project was ready for release about 18 months after the speaker began it.

“Mark kind of joined working on it about nine months ago or so and we really started focusing on this about three months ago and just trying to clean it up and get it get it into the public basically”

Claim: The team began focusing on cleaning up and preparing the Factorio benchmarking project for public release about three months before speaking.

“Mark kind of joined working on it about nine months ago or so and we really started focusing on this about three months ago and just trying to clean it up and get it get it into the public basically”

Claim: Mark joined the Factorio benchmarking project about nine months before speaking.

“I was on the complete opposite side. I had never played Factorio before before joining but I was kind of sold by Jack because of all the reasons that he just brought out and then I remember I had to do the full tutorial before even starting actually building this.”

Claim: The speaker had never played Factorio before joining the Factorio learning-environment project.

“I was on the complete opposite side. I had never played Factorio before before joining but I was kind of sold by Jack because of all the reasons that he just brought out and then I remember I had to do the full tutorial before even starting actually building this. this thing and already during the tutorial I could kind of see the complexity and the opportunities for this and it was then really cool to kind of see as we build this thing where where these models get to and and where do they where they kind of fall short”

Claim: The speaker says Factorio's tutorial made the game's complexity and potential as a learning environment apparent.

“A Factorio exposes a low-level Lua API to interact with the game.”

Claim: Factorio exposes a low-level Lua API that can be used to interact with the game.

“The way you mod Factorio traditionally is you write some Lua scripts, you hook that into the game, and then when events happen, your script runs.”

Claim: In traditional Factorio modding, developers write Lua scripts, hook them into the game, and the scripts run when events occur.

“But this doesn't really work for doing what we were interested in, which was kind of running this at scale and getting, you know, a cluster of servers running.”

Claim: The team found that traditional Lua-script modding was unsuitable for running Factorio at scale across a server cluster.

“So we had to develop an entirely different way to mod the game by exploiting the admin console of multiplayer Factorio servers.”

Claim: The team created an alternative Factorio modding method that uses the admin console on multiplayer servers.

“And what we realized is we could hook into this admin console over the network using a protocol called Archon carried by TCP.”

Claim: The team used the Archon protocol over TCP to connect to Factorio multiplayer server admin consoles over a network.

“And this basically enables us to execute actions in the game. remotely and kind of chart them across dozens or hundreds of Factorio servers,”

Claim: The team's networked admin-console setup enables remote game actions across dozens or hundreds of Factorio servers.

“which is necessary if you want to do training or any of the other workloads you might need at scale.”

Claim: Training and other workloads at scale require running actions across many Factorio servers.

“It was very easy to get a low level harness working, you know, tell the character move one step to the left, move one step to the right, that kind of thing.”

Claim: The team found it very easy to build a low-level Factorio harness for simple commands such as moving the character one step left or right.

“But that type of approach is probably a lot much better suited to training models from scratch.”

Claim: The speaker thinks a low-level-action approach is probably better suited to training models from scratch than to leveraging pretrained models.

“And I guess a Markov decision process architecture where, you know, you give it a bunch of these low level actions and you train a network just to maximize the order in which these low level actions are executed in order to get an outcome.”

Claim: In a Markov decision process approach, a network can be trained to select sequences of low-level actions to achieve an outcome.

“But such an approach would prevent you using any of the pre-training that large language models have got.”

Claim: A low-level-action approach would prevent the team from leveraging large language models' pretraining.

“So that forced us to think a bit more carefully about what interface we could expose to these models, and we ultimately settled on a code synthesis. approach where these models would write python code and those python code in that python code would invoke certain high level actions such as move to this distant point or place an entity next to another entity and you'd leave it up to the model to think about how to abstract these tooling abstract these actions into a structure that would give it”

Claim: The team chose code synthesis: models write Python that invokes high-level Factorio actions, including moving to distant locations and placing entities relative to one another.

“approach where these models would write python code and those python code in that python code would invoke certain high level actions such as move to this distant point or place an entity next to another entity and you'd leave it up to the model to think about how to abstract these tooling abstract these actions into a structure that would give it an ability to manage factories of arbitrary size you want the model to be able to come up with classes with functions with data structures such that when it has a factory that might be producing millions of resources a second it might be sprawled over over a huge amount of the game space it needs to be able to reason over these different parts of the factory efficiently so we decided python because the models pre-trained models are pretty good at python and that was you know a reasonable choice”

Claim: The team designed the interface so models could create abstractions such as classes, functions, and data structures to manage arbitrarily large factories and reason efficiently about their components.

“an ability to manage factories of arbitrary size you want the model to be able to come up with classes with functions with data structures such that when it has a factory that might be producing millions of resources a second it might be sprawled over over a huge amount of the game space it needs to be able to reason over these different parts of the factory efficiently so we decided python because the models pre-trained models are pretty good at python and that was you know a reasonable choice”

Claim: The team chose Python because, in the speaker's assessment, pretrained models are fairly capable at using Python.

“we kind of set up this harness, the models write Python, they execute a Python in the game, which results in factory segments like you can see on screen kind of coming into being.”

Claim: In the team's harness, models write Python code that executes in Factorio and creates factory segments.

“And then the models get given a score, which is basically the production statistics, you know, what it's producing a second.”

Claim: The evaluation scores models using production statistics, including output per second.

“And we also keep track of milestones, such as technologies the model was able to come up with. or new items that they were able to create for the first time.”

Claim: The evaluation tracks milestones such as technologies a model develops and items it creates for the first time.

“So we have these kind of two reward signals that complement each other.”

Claim: The evaluation uses production statistics and milestone tracking as two complementary reward signals.

“And the reason why this is necessary is to avoid certain... I guess behavioral collapses where a model might choose, for example, to mine coal and mine a billion or a trillion coal and that's all it does.”

Claim: The team uses complementary reward signals to reduce behavioral collapse, such as a model focusing exclusively on mining coal.

“for example, to mine coal and mine a billion or a trillion coal and that's all it does. And that would give a good score like that would have a high production score, but it would properly indicate something has gone very wrong in the evaluation or training process. So we have we have a couple of different ways of measuring model performance.”

Claim: A model could score highly on production by mining an extreme amount of coal, but such behavior would signal a failure in evaluation or training.

“And that would give a good score like that would have a high production score, but it would properly indicate something has gone very wrong in the evaluation or training process. So we have we have a couple of different ways of measuring model performance.”

Claim: The team evaluates model performance using multiple measures.

“LabLay is you give the model a task to create a factory of some entity and then the model just needs to create a factory. using our API.”

Claim: In Lab play, a model is assigned a target entity and must build a factory for it using the API.

“That gives you an easy way how to evaluate the upper bound on the performance because we have 24 target entities and each new entity is more difficult than the previous one.”

Claim: Lab play estimates an upper bound on model performance through 24 target entities, each more difficult than the previous one.

“So Laplace is simple in that sense, and it very specifically measures the spatial reasoning of the models, can they use the API, and how well can they work in constrained environments.”

Claim: Lab play evaluates models' spatial reasoning, API use, and performance in constrained environments.

“Now, open play is the unbounded create the biggest factory that you can think of type of environment.”

Claim: Open play is an unbounded Factorio task in which a model must create the largest factory it can.

“Lab play is in that sense a subset of open play where in open play now the models have to actually create their own objectives as well because if you just give it the goal of create the biggest factory.”

Claim: Unlike Lab play, open play requires models to formulate their own objectives in addition to building a factory.

“It needs to make that large goal into sub goals, you know, first let's create drills, then let's create inserters, then let's put them down, then let's create a mine of this resource, that resource, all that.”

Claim: Open play requires a model to decompose the goal of building a large factory into subgoals such as producing and placing equipment and mining resources.

“And so you have these two, let's say, abilities at play there of one. Is the model actually good at using the API and working in Factorio? And two, what objectives does the model set for itself? So for instance, one of the interesting things we saw was that DeepSeek was quite decent at lab play.”

Claim: The evaluation distinguishes a model's ability to use the Factorio API from its ability to set objectives for itself.

“what objectives does the model set for itself? So for instance, one of the interesting things we saw was that DeepSeek was quite decent at lab play. It was kind of on the level of GPT-4.”

Claim: DeepSeek performed reasonably well on Lab play, roughly matching GPT-4's level.

“But in open play, Deep Seek was terrible because what it often just decided to do was let's just create more chests.”

Claim: DeepSeek performed poorly in open play, often choosing to make more chests as its factory-building objective.

“Yes, let's create 250 chests and put them in a nice little line and that is my factory.”

Claim: In open play, DeepSeek chose to build a factory of 250 chests arranged in a line.

“Exactly. Exactly. So you can really see the myopic and short term objective setting at play in the open play because a lot of the times the weaker models, they didn't think four or five steps forward.”

Claim: In open play, weaker models often failed to plan four or five steps ahead.

“So I think that the thing that the task is long-term planning. basically be able to reason over things figure out an appropriate objective and maintain that objective for an extended context without kind of losing the plot and diverting into different areas whereas the lab play is much more about spatial reasoning and the kind of low level figuring out how to compose stuff that”

Claim: The task being evaluated is long-term planning: identifying an appropriate objective and maintaining it over an extended context without losing track or diverting to other tasks.

“basically be able to reason over things figure out an appropriate objective and maintain that objective for an extended context without kind of losing the plot and diverting into different areas whereas the lab play is much more about spatial reasoning and the kind of low level figuring out how to compose stuff that that has the appropriate space and precision around it such that it works.”

Claim: Lab play evaluates spatial reasoning and low-level component composition, including placing components with sufficient space and precision for the setup to work.

“As you might know from Factorio, much like writing a program, if you put one thing in the wrong place, the whole system grinds to a halt.”

Claim: In Factorio, as in programming, placing one component incorrectly can cause the entire system to stop working.

“So that's a slightly different skill set that we're evaluating than the long-term planning stuff.”

Claim: Evaluating spatial composition in the environment tests a different skill set from evaluating long-term planning.

“It's not obvious to us why Claude is so much better than the other models in both settings.”

Claim: The team does not know why Claude performs substantially better than other models in both evaluation settings.

“I mean, our suspicion is it is better trained for this kind of long-term. planning aspect.”

Claim: The team suspects Claude is better trained for long-term planning.

“One thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting.”

Claim: Preliminary results suggest reasoning models perform worse than expected in this evaluation setting.

“And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over an extended period of time.”

Claim: The speaker suspects reasoning models may underperform because the evaluation setup already requires extended reasoning traces.

“Asking the model to do reasoning over reasoning kind of eliminates a lot of the performance improvements you get from applying the reasoning models alone to problems like math frontier, where there's a lot of benefit from just writing stuff down and figuring out what works.”

Claim: Requiring models to reason over their own reasoning may eliminate many gains that reasoning models achieve when applied directly to problems such as math benchmarks.

“This is unpublished. This is our preliminary results, but our plan is to publish additional results with the reasoning models and we've also added in an adapter to support VLMs. Now a lot of the models we did evaluate do have vision.”

Claim: The team added an adapter that supports vision-language models.

“This is unpublished. This is our preliminary results, but our plan is to publish additional results with the reasoning models and we've also added in an adapter to support VLMs. Now a lot of the models we did evaluate do have vision.”

Claim: The team has not yet published its preliminary reasoning-model results and plans to publish additional results including reasoning models.

“In our published results, the models are interacting symbolically with the environment. It doesn't actually see the pixels on the screen.”

Claim: In the published evaluation results, models interact symbolically with the environment and do not receive the screen pixels as visual input.

“So one thing we want to bring out in our next work is how much improvement can you get from allowing the models to take screenshots of part of the factories they want to debug.”

Claim: The team plans to test whether allowing models to screenshot factory areas for debugging improves their performance, and by how much.

“say that the basic geometry of the picture they can't they can't intuit that two lines are have an intersection so to kind of get around this we've scrapped in the vision component we've scrapped using the tile maps and the graphics of factorio and have instead fallen back to a very simple geometric renderer”

Claim: The team replaced Factorio's tile maps and graphics with a simple geometric renderer for its vision component.

“which the models can kind of choose to configure when they invoke it as a tool and the hope there is that we can stumble upon stumble upon the right vision mode that these models are reasonably good at but by i think your intuition is right i would be surprised if we get that much from it at this stage but it would be it would be cool if we do”

Claim: The team hopes to find a vision mode that models can use reasonably well, although the speaker expects little improvement from vision at the current stage.

“which the models can kind of choose to configure when they invoke it as a tool and the hope there is that we can stumble upon stumble upon the right vision mode that these models are reasonably good at but by i think your intuition is right i would be surprised if we get that much from it at this stage but it would be it would be cool if we do”

Claim: Models can configure the geometric renderer when they invoke it as a tool.

“We did try out sending in the screenshots at some point. And the problem is because you have so much going on in larger factories that the monsters start to hallucinate things.”

Claim: In the team's screenshot tests, models hallucinated because larger factories contained too much visual activity.

“They don't correctly understand what is broken, why a factory doesn't work.”

Claim: In screenshot tests, models failed to identify what was broken or why a factory was not working.

“As you said, adding this niches didn't improve things, and it sometimes made it even worse because they started hallucinating entities that weren't there and started doing often, went on tangents that weren't actually real in the game state. Right now, at least, the vision actually wouldn't improve.”

Claim: In the team's tests, vision did not improve performance and sometimes worsened it by causing models to hallucinate nonexistent entities and make claims unrelated to the game state.

“Now, of course, as we go on, the models get better and better. It might improve in the future, but that's at least the current state that we found.”

Claim: The team found that vision does not currently improve results, though it might do so as models improve in the future.

“there's a couple of things that compared to the Voyager paper were interesting. So in Voyager, there was this idea of like self-discovery of actions where the model could like then save skills in a skill library and then reuse them later.”

Claim: In Voyager, a model could discover actions, save skills in a library, and reuse those skills later.

“Sure. So apologies for not being clear in the in the publication, but the models can define classes, they can define functions and then execute them later.”

Claim: The evaluated models can define classes and functions and execute them later.

“Only Google models tend to do this, which is quite interesting.”

Claim: Among the evaluated models, only Google models tended to define classes or functions for later execution.

“Another finding we discovered was that each of the models we evaluated had fundamentally different coding styles.”

Claim: The models evaluated by the team used fundamentally different coding styles.

“CLAWD, for instance, the SONET 3.5 was very much fire and forget. It would write code in a kind of Pythonic way, just like let it fail, don't be careful about it.”

Claim: Claude 3.5 Sonnet tended to write Pythonic code in a fire-and-forget style, allowing failures rather than using careful safeguards.

“Whereas GPT-4 would use defensive programming, use self-assertions, so basically asserting. facts about its internal state,”

Claim: GPT-4 used defensive programming, including assertions about its internal state.

“Yeah, yeah, I think GPT-4 had that. And it's the same with Mini as well, if I remember correctly. They had a much more defensive check before we do anything. But funnily enough, often those checks were also incorrectly set up. So it tripped itself up with its kind of overly defensive nature sometimes.”

Claim: GPT-4 and possibly Mini used defensive checks before acting, but the checks were often incorrect and sometimes hindered the models.

“Agents can define skills, they can incorporate the skills.”

Claim: Agents in the Factorio environment can define and use skills.

“this gives agents a lot of reach as the factories get bigger you can imagine they could create you know quite sophisticated like for example they could create hypothetically an NSAT solver in the kind of name space and then use that for a correct item and entity positioning in the game if they were able to do this”

Claim: As Factorio factories grow, agents might create sophisticated tools such as an NSAT solver and use them to position items and entities correctly in the game.

“As I say, models are fairly rudimentary at this at the moment,”

Claim: Models' capabilities in this Factorio environment are currently fairly rudimentary.

“but part of the motivation for this environment was to enable us to evaluate superhuman models and the evolving state of the art and not become irrelevant in six months, as happens to a lot of existing benchmarks when they kind of max out.”

Claim: The Factorio environment was designed in part to evaluate superhuman models and evolving state-of-the-art systems, avoiding benchmark obsolescence within six months when benchmarks max out.

“It's a lot more difficult to create generalizable skills that would be useful actually in multiple, multiple factory settings apart from the easy connect X to Y,”

Claim: Factorio skills are difficult to generalize across different factory settings, except for simple operations such as connecting X to Y.

“which we exposed as an API already. So we did all...”

Claim: The Factorio environment exposes the simple operation of connecting X to Y through an API.

“but we found that often the skills weren't that reusable that you would hope at least because of exactly the reason that as soon as your environments change,”

Claim: The team found that Factorio skills were often less reusable than hoped because the environment changes.

“if you make a skill over creating a factory, then that skill must be really general and like cover all the edge cases to actually correctly be usable later on.”

Claim: A reusable skill for creating a Factorio factory must generalize broadly and cover edge cases so it remains usable in later environments.

“The abstractions and the code structures that the model invented would align quite closely with the topology of the factory it had created.”

Claim: The speaker hoped that a model's abstractions and code structures would closely reflect the layout of the Factorio factory it built.

“Let's say it had one section which was just about mining and then one section which was just about manufacturing. I was kind of hoping to see that it was able to bring that out and abstract that away to enable it to generalize better in future.”

Claim: The speaker hoped a model would identify and abstract functional sections of a Factorio factory, such as mining and manufacturing, to improve future generalization.

“But we haven't yet seen this.”

Claim: The team has not yet observed a model abstracting factory sections in the hoped-for way.

“I mean, this is something. hope to see when we start training but you know that's an open question no”

Claim: It remains uncertain whether training will lead models to develop the hoped-for factory abstractions.

“we haven't i mean we kind of considered this i mean there's a couple of sources of data i think inspired by mind dojo maybe mind dojo where we we pulled in a lot of data about vectorio on one hand and then we pulled in a lot of you factory blueprints that people created on the other hand and tried to use uh rag basically”

Claim: The team considered using retrieval-augmented access to Factorio information and player-created factory blueprints.

“yes so something that we are i guess we haven't really we still want to try this but it became a bit of an adjunct to the core work um doing this extra stuff um one of the issues we found with blueprints is that”

Claim: The team still wants to try using player-created Factorio blueprints, although blueprint work became an adjunct to its core work.

“First of all, they're specified declaratively, much like you get a terraform, you know, or a cloud formation, infrastructure as code kind of blueprint, which said where everything had to be.”

Claim: Factorio blueprints declaratively specify where factory components should be, similarly to Terraform or CloudFormation infrastructure-as-code blueprints.

“but it didn't provide the imperative steps necessary to get there so on one hand you know we have our python code that the agent writes which are the steps one after the other to get to the factory and on the other hand we have blueprints to just say the final state um so we we spent a fair amount of time trying to try to bridge this gap and we came up with some i guess procedural factory generation techniques which could take a declarative blueprint about what this factory needs to look like and then”

Claim: Factorio blueprints describe a target factory state rather than the sequential actions needed to build it, which the agent's Python code supplies.

“but it didn't provide the imperative steps necessary to get there so on one hand you know we have our python code that the agent writes which are the steps one after the other to get to the factory and on the other hand we have blueprints to just say the final state um so we we spent a fair amount of time trying to try to bridge this gap and we came up with some i guess procedural factory generation techniques which could take a declarative blueprint about what this factory needs to look like and then deduce the steps necessary to get there the downside of this is that it's very low level you know you're not getting the abstractions that you'd like to see but on the plus side we have a lot of that data we have tens of thousands of these factories so I'm thinking that this is the kind of thing that would be really useful just to train the model to do from scratch and just give it you know 10 000 examples of factory units that people have made learn on it”

Claim: The team developed procedural factory-generation techniques that derive the actions needed to build a factory from a declarative Factorio blueprint.

“deduce the steps necessary to get there the downside of this is that it's very low level you know you're not getting the abstractions that you'd like to see but on the plus side we have a lot of that data we have tens of thousands of these factories so I'm thinking that this is the kind of thing that would be really useful just to train the model to do from scratch and just give it you know 10 000 examples of factory units that people have made learn on it”

Claim: The team has tens of thousands of player-created Factorio factories that it believes could help train a model from scratch.

“deduce the steps necessary to get there the downside of this is that it's very low level you know you're not getting the abstractions that you'd like to see but on the plus side we have a lot of that data we have tens of thousands of these factories so I'm thinking that this is the kind of thing that would be really useful just to train the model to do from scratch and just give it you know 10 000 examples of factory units that people have made learn on it”

Claim: The team's procedural factory-generation techniques are low-level and lack the desired abstractions.

“But when you try and move that into a benchmark setting with already pre-trained models just using in-context learning, it's just not that helpful.”

Claim: In a benchmark setting, providing blueprint examples through in-context learning was not very helpful for already pretrained models.

“A thousand lines of Python telling you how to make this kind of factory unit which... it may not be directly applicable to the agent's immediate objective it didn't really give us any uptick in scores so i think we kind of put that in the bucket of what we're going to do when we when we start really throwing computer so for example we when we did all these experiments we kind of accumulated about a million programs that agents have written with scores attached so we kind of got this big bucket of programs and traces trajectories through the game and we want to take these and just kind of throw it at a model and just”

Claim: The blueprint-based approach did not improve benchmark scores, and its examples might not directly match an agent's immediate objective.

“it may not be directly applicable to the agent's immediate objective it didn't really give us any uptick in scores so i think we kind of put that in the bucket of what we're going to do when we when we start really throwing computer so for example we when we did all these experiments we kind of accumulated about a million programs that agents have written with scores attached so we kind of got this big bucket of programs and traces trajectories through the game and we want to take these and just kind of throw it at a model and just and just see what happens if we really try and get it to learn how to maximize score directly,”

Claim: The team plans to train a model on its collected Factorio programs and trajectories to test whether it can learn to maximize the game score directly.

“it may not be directly applicable to the agent's immediate objective it didn't really give us any uptick in scores so i think we kind of put that in the bucket of what we're going to do when we when we start really throwing computer so for example we when we did all these experiments we kind of accumulated about a million programs that agents have written with scores attached so we kind of got this big bucket of programs and traces trajectories through the game and we want to take these and just kind of throw it at a model and just”

Claim: The team's Factorio experiments produced about one million scored agent-written programs and associated game traces and trajectories.

“there is a big skill gap.”

Claim: Factorio agents and human players have a large skill gap.

“because they execute Python programs, come up with quite complicated policies which a human player would have to kind of develop on the fly so in that sense the agent has an advantage but when we kind of track the number of ticks it took to execute these policies against the number of ticks a human player would spend to get somewhere the gulf between models playing this and humans playing it is absolutely enormous um”

Claim: Factorio agents can use executed Python programs to develop complicated policies that human players would otherwise have to devise on the fly, giving the agents an advantage in policy development.

“come up with quite complicated policies which a human player would have to kind of develop on the fly so in that sense the agent has an advantage but when we kind of track the number of ticks it took to execute these policies against the number of ticks a human player would spend to get somewhere the gulf between models playing this and humans playing it is absolutely enormous um”

Claim: Measured in game ticks required to execute policies and make progress, the performance gap between Factorio-playing models and human players is enormous.

“the current record for speedrunning is about an hour and 20 minutes to get to a rocket launch,”

Claim: At the time of the conversation, the Factorio speedrunning record for reaching a rocket launch was about one hour and 20 minutes.

“It took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few, let's say, iron gear wheels or electric circuits or maybe some science and maybe start progressing through the tree.”

Claim: In the reported evaluation, models took roughly eight hours to build a working Factorio factory that could produce a few items and perhaps begin progressing through the technology tree.

“So, you know, there's probably 100 times difference in the competency of these models untrained than a human expert.”

Claim: The speaker estimates that untrained models are probably about 100 times less competent at Factorio than a human expert.

“maybe a better experiment would be to take someone who'd never played the game before or”

Claim: The speaker suggests that a better experiment might compare Factorio models with a person who has never played the game.

“put them in the game and then compare it with the models because the models you know they have never been trained on this stuff they only know a little bit about factorio they they have the manual of how to interact with it and they just they just have to go and and try their best so maybe that's what we should do i should try and recruit my fiance and kind of make her sit down for for 10 hours in front of the game and see where we get”

Claim: The speaker might recruit his fiancée to play Factorio for 10 hours as part of a comparison with the models.

“put them in the game and then compare it with the models because the models you know they have never been trained on this stuff they only know a little bit about factorio they they have the manual of how to interact with it and they just they just have to go and and try their best so maybe that's what we should do i should try and recruit my fiance and kind of make her sit down for for 10 hours in front of the game and see where we get”

Claim: The Factorio models have not been trained on the game; they have only limited knowledge of it and instructions for interacting with it, and must attempt the task without game-specific training.

“I don't think there is a mechanical advantage per se.”

Claim: The speaker does not think AI agents have an inherent mechanical advantage over humans in Factorio.

“is purely what type of structure are they able to come up with? And if you do start to train against that.”

Claim: The speaker thinks the largest potential improvement in Factorio agents depends on the structures they can devise.

“objective then probably we will we might see some wacky structures which maybe humans haven't thought of in the future but that's that's the the the gap towards there and the road towards there is long also guys well this was great and there's a lot of factorial fans as we wrap up a couple things are there any games that you wish people would build similar experiences on just to see how the models perform maybe can be you know”

Claim: If Factorio agents are trained toward the relevant objective, they might eventually devise unusual structures that humans have not thought of, but reaching that point will take a long time.

“I would really like to see some agents made for playing Age of Empires 2 because it's such a well-designed game and the communities kind of come back over recent years.”

Claim: The speaker would like to see AI agents built for Age of Empires 2, which he considers well-designed and whose community has returned in recent years.

“But the AI of the models in the game is really terrible.”

Claim: The speaker considers the models' AI performance in Age of Empires 2 to be very poor.

“And I think it'd be really fun to see how good we can train these models to play that game.”

Claim: The speaker thinks it would be fun to test how well models can be trained to play Age of Empires 2.

“so we see that Claude right here has almost got twice the score of the nearest best model.”

Claim: In the reported results shown on a log-log graph, Claude scored almost twice as high as the next-best model.

“which was so bad upon occasion it actually begged to be turned off it gave up and said i can't do this anymore please reset me and then it would and then it would refuse to do anything else for like 100 steps”

Claim: In the evaluation, GPT-4 mini sometimes performed so poorly that it asked to be turned off, gave up, and refused further action for about 100 steps.

“i think maybe the kind of ambition or determination is some kind of latent factor in the competency of these models as expressed in the game.”

Claim: The speaker thinks ambition or determination may be a latent factor affecting how model competence is expressed in the game.

“guys. Yeah, I think getting models to acknowledge that they cannot do something, actually I think it's an interesting form of refusal.”

Claim: The speaker considers a model's acknowledgment that it cannot do something to be an interesting form of refusal.

“knowing the limits of their competence because i think often we're incentivizing these models to say anything”

Claim: The speaker argues that models should recognize the limits of their competence, since they are often incentivized to say anything.

“and so to me that's almost a positive result that is just saying we're not going anywhere like give a positive reward for that yeah”

Claim: The speaker considers a model's acknowledgment that it cannot make progress a positive result and supports rewarding that behavior positively.

“this work was phase one of a phase two plan.”

Claim: The speaker describes the current work as phase one of a planned second phase.

“Our goal now is to start training these models directly on unbounded objectives.”

Claim: The speaker's current goal is to begin training the models directly on unbounded objectives.

“We think that there is a pretty strong application to this in terms to the field of model alignment,”

Claim: The speaker believes this work has a strong potential application in model alignment.

“In this instrumental convergence hypothesis, the paperclip maximizer hypothesis, where we think that certain behaviors are always going to emerge regardless of whether it's a human doing it or whether it's a machine,”

Claim: The speaker believes certain behaviors are bound to emerge in either humans or machines, as proposed by the instrumental convergence and paperclip maximizer hypotheses.

“one really stands out, which is goal content integrity, specifically the reluctance for people or maybe models to change their goal once that goal has been set.”

Claim: The speaker identifies goal-content integrity as reluctance by people, and possibly models, to change a goal after it has been set.

“basically build a paperclip maximizer and see whether this behavior emerges, because if it does, that I think might force us to change our attitude towards alignment, which is heavily based on this kind of incremental approach.”

Claim: The speaker thinks that if goal-content integrity emerges, it might require changing the prevailing attitude toward alignment, which the speaker characterizes as heavily reliant on an incremental approach.

“But if we get to a point where the models are very clever and they don't want to have their objectives changed, it kind of means we only have one really good shot to set their objectives ahead of time because otherwise it's going to become quite hard to do it down the road.”

Claim: If models become very capable and resist having their objectives changed, the speaker believes there may be only one good opportunity to set their objectives beforehand, since changing them later will become difficult.

“So this is kind of an open project.”

Claim: The speaker describes the project as open.