Pratik said the benchmark included single-turn and multi-turn datasets and edge cases involving missing parameter information, missing tools, long context, and composite scenarios.

“Targeting certain domains for evaluation but what our goal was that let's take all of these benchmarks because it's possible that certain models are good at certain data sets there might be issues of leakage also so we wanted to create a as much possible holistic benchmark by taking on data sets slices on single turn multi turn”

“and also on like different kind of edge cases like there is some information missing for the parameter the tool itself is missing and long context and all those problems all at once called composite scenario all those things we wanted to take from this different data sets we did filtering on it for as hard as possible and then we created 14 small data sets and then evaluated LM specifically for”