definition claim
The task being evaluated is long-term planning: identifying an appropriate objective and maintaining it over an extended context without losing track or diverting to other tasks.
Jack Hopkins · ⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins · 27 Apr 2025
In their words
The transcript segments this claim was taken from
“So I think that the thing that the task is long-term planning.”
“basically be able to reason over things figure out an appropriate objective and maintain that objective for an extended context without kind of losing the plot and diverting into different areas whereas the lab play is much more about spatial reasoning and the kind of low level figuring out how to compose stuff that”11:14Report a misattribution
0.90how strongly it was put, not whether it is true
0.85importance to the conversation
0.98confidence the claim was read correctly