A model could score highly on production by mining an extreme amount of coal, but such behavior would signal a failure in evaluation or training.

“for example, to mine coal and mine a billion or a trillion coal and that's all it does.”

“And that would give a good score like that would have a high production score, but it would properly indicate something has gone very wrong in the evaluation or training process. So we have we have a couple of different ways of measuring model performance.”