Prominent AI benchmarks often emphasize difficult, easily verifiable puzzle-like tasks, whereas realistic coding-agent settings involve multiple interactions with a human and multiple objectives.

“that are just like involve like doing a much more complicated task or doing something that requires a lot more cleverness and you could you could say this is like the benchmarking distribution because a lot of the most prominent benchmarks just involve doing some very hard puzzle like task that's easy to verify and then there's sort of like the realism axis where you want the model to be good in the realistic coding agent setting where there's like multiple back and forth.”

“back and forth with the human and there's like multiple objectives and like I'd say”