The speaker says they evaluate the meta-learning system by testing the discovered inner-loop agent on held-out benchmarks unseen by the outer loop, each with public and private splits.

“but in order to evaluate the whole”

“Meta learning system.”

“We need to test it on a hold out set of benchmarks so that the out those benchmarks are tasks that even auto loop never sees and for all of those benchmarks they are still private and a public split so we tested the discovered inner loop agent on those hold out tasks that's the protocol”