The Factorio environment was designed in part to evaluate superhuman models and evolving state-of-the-art systems, avoiding benchmark obsolescence within six months when benchmarks max out.

“but part of the motivation for this environment was to enable us to evaluate superhuman models and the evolving state of the art and not become irrelevant in six months, as happens to a lot of existing benchmarks when they kind of max out.”