A 79% benchmark score does not by itself establish that a model is useful or ready for use; consistently delivering work only 79% of the way to completion would be inadequate.

“Fourteen sixty, right? So, like, you can still give assignment accordingly for RL, um, and obviously that, that helps. But I think on the evaluation side, what's, what's really important is I thought it was kind of, kind of find this funny, um, labs will often be like, "Oh, like our model score is 79% on this-”

“... on this benchmark."”

“But yeah, but not interesting anymore. It's like, you know, imagine you just had an employee who just kept getting everything 79% over the line, and it was like, "Cool, I'm done.”

“Never gonna look at it again." It's like this is unusable.”