Open Problems required participants to submit their own metrics alongside the provided metrics so that subtle aspects of model performance could be quantified.

“That sound a bit cooler, right? We picked up very well and but of-- with metrics that are robust were maybe, but maybe not, not that easy. So for ac-ac-actually our solution in, in this, this, this consortium where a lot of labs contributed called Open Problems, was actually that people need to submit their own metrics in addition, so in order to really quantify maybe subtle things that you might not have thought before.”