The speaker expects the evaluation to recalibrate users' expectations of LLMs, which make mistakes even though they are often characterized as very capable at tool calling.

“We have reached a certain stage where we feel confident that yes, there is taken hard scenarios and the model is not able to accomplish all the time. So this is going to like reset, this is going to calibrate the expectation of the users of this LLMs because our last leader was saying that models are really great at tool calling.”

“So it's like safe, but they're actually not right. They're making mistakes and this is going to recalibrate the expectation.”