The speaker's team believes its evaluation scenarios are sufficiently challenging that the model does not succeed on them consistently.

“We have reached a certain stage where we feel confident that yes, there is taken hard scenarios and the model is not able to accomplish all the time. So this is going to like reset, this is going to calibrate the expectation of the users of this LLMs because our last leader was saying that models are really great at tool calling.”