The benchmark provides a target for desired model capabilities, but training models to perform well on it requires substantially larger private datasets.

“And I think maybe what's useful to add is that the benchmark gives a North Star on where some of these things need to be. But to actually train the models to do really well at this benchmark, the private data sets need to be much larger to actually, like, for the model to investigate many different situations and see how it's reasoning through those situations and how it acts in those situation.”