The team tests its system on hold-out benchmarks, including a weather-forecasting task far outside the optimization distribution, and reports that it generalizes well on that task.

“Of course, we have a set of benchmarks we're trying to hear time on.”

“After that, we will test it on hold-out benchmarks.”

“Those are public benchmarks like MLEBenchLite,”

“ALEBenchLite. ALEBench is an algorithmic discovery benchmark from Salkana.”

“MLEBench is the...”

“OpenAI's machine learning engineering benchmark and we also tested on a very far out of distribution task called weather bench 2 essentially you're trying to build a physics-based forecasting engine to predict weather”

“And the the weather bench is very far out of distribution from the training or optimization set of but it is still generalized really well.”