SpecBench evaluates general-purpose software-engineering agents with a public unit-test-like set and a holdout set that tests generated software on combinations of tests intended to simulate real-world use.

“make the unit test perform much better and when you deploy that kernel into the actual end-to-end inference of a model the performance are actually getting worse and specbench is a extrapolation of that to more general purpose software engineering and we extend the protocol we found on.”

“GPU kernels like we maintain a public set which is a more like a unit test for the software system and a holdout set where we test the capability of the agent on capability of the generated software on the composition of those unit tests like simulating real-world usage”