On SPEAKER_02's benchmark, reasoning models such as OpenAI's O-series scored poorly in part because they often returned only one tool call for requests requiring multiple calls, while the evaluation required all calls in one output.

“uh that worked surprisingly well then another surprise for me was that the reasoning models were not performing well enough they had certain kind of limitation when we probed into it like why are they scoring less overall they were like the o1 the o4 o3 they”

“When not performing well specifically for multiple tool call outputs that you have this request and it requires either the same tool being called multiple time or different tools being called all at once and they were outputting only single tool call and because our evaluation required that okay all the tools should be outputted at once that's the instruction as well in the prompt it was not falling and it was getting low score so because”

“Because of certain inherent characteristic of specifically O-series model,”

“we did not find them well to be performing on this kind of data set. We had like multiple data set on the multiple tool output and that brought down the score for this model.”