Factory benchmarking can involve curating coding-agent runs for a task category, such as front-end coding, into a benchmark suite.

“And so benchmarking in this context, in the factory context, means essentially the ability to like take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means like, you know, you might curate a set of coding agent runs that were related to like doing front-end coding or something like that.”