On GPU-kernel tasks, an agent may optimize performance on a unit test while making the kernel perform worse in end-to-end model inference.

“Back then we realized reward hacking is one of the big problem of auto research agents because that's especially severe on tasks like GPU kernel engineering where the agent always figure out of the complex way to”

“make the unit test perform much better and when you deploy that kernel into the actual end-to-end inference of a model the performance are actually getting worse and specbench is a extrapolation of that to more general purpose software engineering and we extend the protocol we found on.”