⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

“Which quickly gets saturated.”

Claim: The host said that model evaluations can become saturated quickly.

“I think LLM Arena also had some controversies.”

Claim: The host said that LLM Arena had some controversies.

“And I think more generally people want agent evals anyway, where they are evaluating the ability of, you know, models to do real tasks.”

Claim: The host said that people increasingly want agent evaluations that assess models' ability to perform real tasks.

“And instead of like testing knowledge, they're actually, you know, doing tool calling and utilizing context very well that kind of stuff so yeah I think one of those things where there's one of those things where I think there's a lot of fresh area and potential for progress if you can do a credible eval and the other thing I like to see is an eval that has a cost awareness component because I think you know people don't really”

Claim: The host said agent evaluations assess tool calling and effective context use rather than only testing knowledge.

“utilizing context very well that kind of stuff so yeah I think one of those things where there's one of those things where I think there's a lot of fresh area and potential for progress if you can do a credible eval and the other thing I like to see is an eval that has a cost awareness component because I think you know people don't really”

Claim: The host favors agent evaluations that account for cost.

“utilizing context very well that kind of stuff so yeah I think one of those things where there's one of those things where I think there's a lot of fresh area and potential for progress if you can do a credible eval and the other thing I like to see is an eval that has a cost awareness component because I think you know people don't really”

Claim: The host thinks there is substantial room for progress in agent evaluation if evaluations are credible.

“stay very honest with like oh yeah like we spent a lot more tokens or we spent a lot more money on this agents and we did better but it's not really clear like if the other comparison could also just do the same thing like just think harder and also have the same performance”

Claim: The host said agent comparisons can be misleading when one system spends more tokens or money for better performance, because another system might match that performance with the same resources by thinking harder.

“100% yeah so we have been hearing from our customers that they are now moving towards agents and it become very clear to us that last year we did rag evaluation because that's what we are discussing more with the customers but we quickly understood that this is going to be a year of agents and what we have understood from our previous experiments is that it's not necessary that what you see as top”

Claim: Pratik said Galileo's customers were moving toward agents, leading Galileo to see the current year as a year of agents, whereas customer discussions the previous year had focused more on RAG evaluation.

“100% yeah so we have been hearing from our customers that they are now moving towards agents and it become very clear to us that last year we did rag evaluation because that's what we are discussing more with the customers but we quickly understood that this is going to be a year of agents and what we have understood from our previous experiments is that it's not necessary that what you see as top stop models in let's say Alemarina or other specific evaluation or general evaluations they might not be also the same ranking for other task like agentic task so that was our goal very fundamentally speaking that we want to get very specific insights apart from just performance and cost was the other thing that comes up a lot with enterprise customers and”

Claim: Pratik said models that rank highly on general or other specific evaluations may rank differently on agentic tasks.

“stop models in let's say Alemarina or other specific evaluation or general evaluations they might not be also the same ranking for other task like agentic task so that was our goal very fundamentally speaking that we want to get very specific insights apart from just performance and cost was the other thing that comes up a lot with enterprise customers and That was the second focus essentially and then there is ASLM versus LLM kind of debate are the small models also catching up and there is the big one open source versus private. So those were the dimensions we were interested in.”

Claim: Pratik said the leaderboard aimed to provide insights beyond performance, including cost, small-versus-large model comparisons, and open-source-versus-private model comparisons.

“stop models in let's say Alemarina or other specific evaluation or general evaluations they might not be also the same ranking for other task like agentic task so that was our goal very fundamentally speaking that we want to get very specific insights apart from just performance and cost was the other thing that comes up a lot with enterprise customers and”

Claim: Pratik said enterprise customers frequently raise performance and cost as important considerations.

“Neural leaderboard is kind of eval harness specifically created to test LLMs on hundreds of domains.”

Claim: Pratik described the leaderboard as an evaluation harness for testing LLMs across hundreds of domains.

“Targeting certain domains for evaluation but what our goal was that let's take all of these benchmarks because it's possible that certain models are good at certain data sets there might be issues of leakage also so we wanted to create a as much possible holistic benchmark by taking on data sets slices on single turn multi turn”

Claim: Pratik said the leaderboard combined benchmarks and datasets to create a holistic evaluation, since models may excel on particular datasets and data leakage may occur.

“Targeting certain domains for evaluation but what our goal was that let's take all of these benchmarks because it's possible that certain models are good at certain data sets there might be issues of leakage also so we wanted to create a as much possible holistic benchmark by taking on data sets slices on single turn multi turn and also on like different kind of edge cases like there is some information missing for the parameter the tool itself is missing and long context and all those problems all at once called composite scenario all those things we wanted to take from this different data sets we did filtering on it for as hard as possible and then we created 14 small data sets and then evaluated LM specifically for”

Claim: Pratik said the benchmark included single-turn and multi-turn datasets and edge cases involving missing parameter information, missing tools, long context, and composite scenarios.

“and also on like different kind of edge cases like there is some information missing for the parameter the tool itself is missing and long context and all those problems all at once called composite scenario all those things we wanted to take from this different data sets we did filtering on it for as hard as possible and then we created 14 small data sets and then evaluated LM specifically for”

Claim: Pratik said Galileo filtered source datasets for difficulty and created 14 smaller datasets to evaluate tool calling.

“for tool calling and we use a used a gallows metric called tool selection quality which is an LLM judge based metric powered by GPT 4.0 and it has multiple judges so that we get high performance and that's how we kind of get the score for each sample and we aggregate for a data set and then we do average over the data set and that's how we get the overall score for the ranking but yes customers sorry”

Claim: Pratik said Galileo scored samples with its Tool Selection Quality metric, an LLM-judge-based metric powered by GPT-4.0 and using multiple judges.

“for tool calling and we use a used a gallows metric called tool selection quality which is an LLM judge based metric powered by GPT 4.0 and it has multiple judges so that we get high performance and that's how we kind of get the score for each sample and we aggregate for a data set and then we do average over the data set and that's how we get the overall score for the ranking but yes customers sorry”

Claim: Pratik said Galileo aggregated sample scores by dataset and averaged the dataset scores to calculate the overall ranking score.

“Readers can also go through particular performances or certain data sets as well.”

Claim: Pratik said readers could inspect model performance on individual datasets.

“Apart from that, we were pragmatic about selection in a way that we were very interested in how the small models perform as well as open source model performs because we know that there can be interesting insights historically what we have seen from our previous initiatives is that, okay, maybe the best GPT or best cloud is the top model, but there might be very small gap with.”

Claim: Pratik said Galileo deliberately included small and open-source models in its evaluation because they could reveal useful insights.

“Apart from that, we were pragmatic about selection in a way that we were very interested in how the small models perform as well as open source model performs because we know that there can be interesting insights historically what we have seen from our previous initiatives is that, okay, maybe the best GPT or best cloud is the top model, but there might be very small gap with.”

Claim: Pratik said the performance gap between the top GPT or Claude model and the next model could be small.

“the model just below it and then it becomes a cost performance trade-off so that users can kind of select a model which is a slightly worse but also gives them a similar performance and there can be some hidden models that nobody's talking about because it's all about the latest top releases and not about like the mid-tier or slightly lower tier models from other providers and sometimes it's just they haven't got enough traction on social so they might be just missing from the discussion so we try to find this kind of models and get insights as much as possible so that that was the whole initiative we launched in February this year previously we had done RAC specific evaluations now”

Claim: Pratik said mid-tier and lower-tier models may be overlooked because discussion focuses on the latest top releases, while some models may also lack social traction.

“the model just below it and then it becomes a cost performance trade-off so that users can kind of select a model which is a slightly worse but also gives them a similar performance and there can be some hidden models that nobody's talking about because it's all about the latest top releases and not about like the mid-tier or slightly lower tier models from other providers and sometimes it's just they haven't got enough”

Claim: Pratik said users may find a cost-performance trade-off in choosing a slightly lower-performing model that provides similar performance at lower cost.

“traction on social so they might be just missing from the discussion so we try to find this kind of models and get insights as much as possible so that that was the whole initiative we launched in February this year previously we had done RAC specific evaluations now”

Claim: Pratik said Galileo had conducted RAG-specific evaluations before launching the agent leaderboard initiative.

“traction on social so they might be just missing from the discussion so we try to find this kind of models and get insights as much as possible so that that was the whole initiative we launched in February this year previously we had done RAC specific evaluations now”

Claim: Pratik said Galileo launched the agent leaderboard initiative in February of that year.

“A small update on the agent leaderboard also like we are going to do a v2 of it because we find certain issues with this evaluation and let's talk about this more at the end of the podcast like what's next coming we can dive more into that.”

Claim: Pratik said Galileo planned a second version of its agent leaderboard because it had found issues with the evaluation.

“I was expecting GPTs to perform the best because they kind of pioneered the tool calling, function calling support and they have been working on that for so long.”

Claim: SPEAKER_02 expected GPT models to perform best on tool-calling evaluations because they pioneered tool and function calling and had worked on it for a long time.

“There was some criticism around Gemini models, although they were catching up and I was very surprised to see when we released this leaderboard. Of course, there has been some shaking of the leaderboard also later on, but when we released, we found the Gemini models. are performing really good and they were extremely cost efficient at the time compared to the second and second model like GPT-4 at the time and so it became kind of a no-brainer that if anybody wants top performance they can just directly go and go with the Gemini models and they just not only release like the pro model they also release the flash version later on and flash light if the flash light is a very interesting model in in the blog we have the models when we release this little bit but in the”

Claim: At the time SPEAKER_02's leaderboard was released, Gemini models performed very well and were extremely cost-efficient compared with GPT-4.

“the and the original leaderboard we also have flash light which is nobody talks about i think even now but i think it's got a very decent score which is like in the top five models it's extremely cheap model like it's so dirt cheap that it makes a lot of sense only model that is cheaper than it is and performant is from amazon which also nobody talks about so we found this more interesting models”

Claim: SPEAKER_02 said that the only model cheaper than Gemini Flash Lite while also performing well came from Amazon.

“the and the original leaderboard we also have flash light which is nobody talks about i think even now but i think it's got a very decent score which is like in the top five models it's extremely cheap model like it's so dirt cheap that it makes a lot of sense only model that is cheaper than it is and performant is from amazon which also nobody talks about so we found this more interesting models”

Claim: On the original leaderboard, Gemini Flash Lite scored in the top five and was extremely inexpensive.

“uh that worked surprisingly well then another surprise for me was that the reasoning models were not performing well enough they had certain kind of limitation when we probed into it like why are they scoring less overall they were like the o1 the o4 o3 they When not performing well specifically for multiple tool call outputs that you have this request and it requires either the same tool being called multiple time or different tools being called all at once and they were outputting only single tool call and because our evaluation required that okay all the tools should be outputted at once that's the instruction as well in the prompt it was not falling and it was getting low score so because Because of certain inherent characteristic of specifically O-series model, we did not find them well to be performing on this kind of data set. We had like multiple data set on the multiple tool output and that brought down the score for this model.”

Claim: On SPEAKER_02's benchmark, reasoning models such as OpenAI's O-series scored poorly in part because they often returned only one tool call for requests requiring multiple calls, while the evaluation required all calls in one output.

“And of course they are extremely costly,”

Claim: SPEAKER_02 described the O-series models as extremely costly.

“So then it became straightforward that you should not be using this for tool calling and instead go for the regular models program. Probably.”

Claim: SPEAKER_02 concluded that users should generally choose regular models rather than reasoning models for tool calling.

“Apart from that on the open source side I would say that I was very impressed by Mistral's entry into the ecosystem.”

Claim: SPEAKER_02 was very impressed by Mistral's entry into the open-source model ecosystem.

“3.1 just straight away went up and it also very cost efficient so I think not people might have known at the time and in the launch blog also there was not much discussion on tool calling but the model performed very well on our benchmark.”

Claim: Mistral 3.1 was cost-efficient and performed very well on SPEAKER_02's benchmark, although its launch blog gave little attention to tool calling.

“And of course, the other surprise was that all the llama models were not performing well on our benchmark, 3.3 and even the llama 4 all were really performing extremely poor. So”

Claim: All Llama models, including Llama 3.3 and Llama 4, performed very poorly on SPEAKER_02's benchmark.

“and that was unbelievable like it felt like they were not tuned for full calling at all in some way so I mean historically it's all historical right there were certain biases and we are like meta one”

Claim: SPEAKER_02 felt that the Llama models were not tuned for function calling.

“The Lama 1 was great, Lama 2 was great, 3 also was great and then suddenly something happened and it went downhill.”

Claim: SPEAKER_02 said Llama 1, Llama 2, and Llama 3 were great, after which the Llama models' performance declined.

“When we started the leaderboard, Cloud 3.5 was somewhere in the middle.”

Claim: Claude 3.5 ranked around the middle of SPEAKER_02's leaderboard when it began.

“It was really good at certain other things but for tool calling it was not awesome.”

Claim: Claude 3.5 performed well on some tasks but not especially well at tool calling.

“But then when we released the leaderboard just in a week they launched 3.7 and that went straight up.”

Claim: Claude 3.7 rose to the top of SPEAKER_02's leaderboard after its release, which occurred about a week after the leaderboard was published.

“And nobody has beaten it so far. We have not evaluated the latest one like the trend models. Also the latest mistral models and not the O3 pro models which just came out in last few days.”

Claim: As of the conversation, no evaluated model had beaten Claude 3.7 on SPEAKER_02's leaderboard; the team had not yet evaluated the latest models, including recent Mistral models and O3 Pro.

“We have not evaluated but it held the fort for many many weeks.”

Claim: Claude 3.7 remained at the top of SPEAKER_02's leaderboard for many weeks.

“No, we actually took the data slices from the original data set and then ran it ourselves. We got the outputs and then we evaluated with our metrics. So it's fresh.”

Claim: SPEAKER_02's team reran evaluations on slices of the original datasets, generated outputs themselves, and scored those outputs with their own metrics, producing fresh results.

“I feel like they are very similar in nature. It's just that the data set is from different domains and certain data sets have multi-turn conversations available while certain do not have it.”

Claim: SPEAKER_02 considered the benchmarks similar in nature, while noting that their datasets cover different domains and vary in whether they include multi-turn conversations.

“I do not remember exactly which one has it and which one doesn't, but there were these discrepancies and we really enjoyed, like I really loved Tau benchmark also because it had vertical.”

Claim: SPEAKER_02 did not recall exactly which datasets included multi-turn conversations, and said the datasets had discrepancies.

“I do not remember exactly which one has it and which one doesn't, but there were these discrepancies and we really enjoyed, like I really loved Tau benchmark also because it had vertical. hard evaluation data sets and a lot of models had some struggle over there and”

Claim: SPEAKER_02 valued the Tau benchmark for its vertical, difficult evaluation datasets, on which many models struggled.

“I mean yes, the Tau Bench is for two domains, test agentic capabilities for two domains which is airline and retail and the setup is that they have very long prompts in the beginning with lot of requests and then you have these tools defined and the tools can interact.”

Claim: Tau-Bench evaluates agentic capabilities in the airline and retail domains.

“test agentic capabilities for two domains which is airline and retail and the setup is that they have very long prompts in the beginning with lot of requests and then you have these tools defined and the tools can interact. interact with certain databases for read and write operations and it is predefined that once the whole agent task is done then they will validate on the final edits made on and they will check for the expected value in the data sets in the database which are basically CSV style of files and in that fashion they will validate whether the task was done or not”

Claim: Tau-Bench tasks use long, multi-request prompts and tools that can read from and write to databases.

“interact with certain databases for read and write operations and it is predefined that once the whole agent task is done then they will validate on the final edits made on and they will check for the expected value in the data sets in the database which are basically CSV style of files and in that fashion they will validate whether the task was done or not”

Claim: Tau-Bench checks an agent's final database edits against expected values in CSV-style datasets to determine whether the task succeeded.

“You get a deterministic evaluation all the times that's the beauty of this and it's it's domain specific Which I think is missing from other evaluation data sets and that is also one of the themes for our v2”

Claim: The speaker believes other evaluation datasets lack domain specificity and identifies domain specificity as a theme for their version 2.

“You get a deterministic evaluation all the times that's the beauty of this and it's it's domain specific Which I think is missing from other evaluation data sets and that is also one of the themes for our v2”

Claim: Tau-Bench provides deterministic evaluations, which the speaker considers a strength of the benchmark.

“Because I think people don't just come from a general mindset, people in the industry come from let's say I'm from healthcare I want to see if this models work for healthcare or not People are coming from investment finance insurance industry industry and they want to know”

Claim: Industry users want to assess whether models are suitable for their particular domains, including healthcare, investment, finance, and insurance.

“That is all the models ready for my domain or not because we all know that things work on the general domain side and when you just put it on the internet data things start to blow up with internal terminologies and the cases that happen.”

Claim: The speaker says models that work in general domains can encounter problems when applied to internet data because of internal terminology and domain-specific cases.

“So I think that was the beauty that they specifically targeted domains.”

Claim: The speaker considers Tau-Bench's domain-specific focus a strength.

“I don't know if they might have expanded domains in this because two was I felt like less but it was a decent start.”

Claim: The speaker thought Tau-Bench's initial coverage of two domains was limited but a decent start.

“It was more about the methodology rather than the domains I felt.”

Claim: The speaker viewed Tau-Bench as being more significant for its evaluation methodology than for its domain coverage.

“We have seen historically they have issues with long contacts and they have issues with certain kind of outputs. They prefer, let's say, lengthy outputs and stuff.”

Claim: Historically, LLMs have struggled with long contexts and certain output types, and have tended to favor lengthy outputs.

“our philosophy in the company is that any metric that comes out which is a Galileo metric we do extensive research on it.”

Claim: Galileo conducts extensive research on every metric it releases as a Galileo metric.

“We create very hard data sets internally and we evaluate them, we annotate them and check for correlations and only once we feel comfortable then only we release it and which is usually”

Claim: Before releasing a metric, Galileo creates challenging internal datasets, evaluates and annotates them, and checks for correlations.

“based on the correlation scores like AUR ROC scores that we find if they are high enough only then we release it otherwise it goes through a rigorous process of prompt optimization so we allow”

Claim: Galileo releases a metric only if its correlation scores, including AUR ROC scores, are sufficiently high; otherwise, it subjects the metric to rigorous prompt optimization.

“people to use any LLM with the metric and so because LLMs keep on improving the metric also keeps on improving and then the prompts is what we kind of”

Claim: Galileo allows users to pair its metric with any LLM, and says the metric improves as LLMs improve.

“The prompt is where all the research goes through and we have seen tremendous value in that and yes people would definitely have skepticism a lot of people a lot of customers also have that skepticism some are still stuck on blue and red scores so which is surprising because they think that LLMs are not there yet.”

Claim: Galileo concentrates its research on prompts and reports seeing substantial value from that work.

“The prompt is where all the research goes through and we have seen tremendous value in that and yes people would definitely have skepticism a lot of people a lot of customers also have that skepticism some are still stuck on blue and red scores so which is surprising because they think that LLMs are not there yet.”

Claim: Many Galileo customers remain skeptical of LLM-based evaluation; some still rely on blue-and-red scores because they believe LLMs are not yet capable enough.

“But we what we try to do from our side is essentially we try to follow the best practice of prompting, we try to exploit the reasoning as much as possible from this LLMs and then we use basically self-consistency by using multiple outputs and then taking a majority voting over there.”

Claim: Galileo's evaluation approach applies prompting best practices, elicits as much reasoning as possible from LLMs, and uses majority voting across multiple outputs for self-consistency.

“People have flexibility on these things. They can select the LLMs they will they want to use if they want to use much more powerful LLM. go ahead you can customize it if you want to change the prompts go ahead customize it if you want to increase the number of judges go ahead and customize it so we allow full flexibility to people to modify this llm as judges so that they can get the best performance possible of course”

Claim: Galileo gives users flexibility to select LLMs, customize prompts, and change the number of LLM judges to pursue the best performance.

“People should always validate with some small annotated data set before they go ahead and just make it the default evaluation.”

Claim: Users should validate an LLM-based evaluation against a small annotated dataset before adopting it as their default evaluation.

“So that is always a part of our POCs and they get clarity and confidence only then they will be they are using it.”

Claim: Galileo includes validation in its proofs of concept to help users gain clarity and confidence before adopting an evaluation.

“So I feel I have basically I've been a full stack ML engineer previously and I've worked a lot with annotators and I have seen a lot of biases and issues.”

Claim: The speaker says that his prior work as a full-stack machine-learning engineer involved extensive work with annotators, through which he observed many biases and problems.

“issues with annotators as well it's just so time consuming that you can't afford to kind of wait for annotation to get completed so it's just much much better to add more instructions do some initial validation and get that clarity that okay this is where it works if it doesn't”

Claim: The speaker says human annotation can take so long that teams cannot afford to wait for completion.

“issues with annotators as well it's just so time consuming that you can't afford to kind of wait for annotation to get completed so it's just much much better to add more instructions do some initial validation and get that clarity that okay this is where it works if it doesn't just go back to annotation but I feel like it has started working a lot one year back because we I was running this experiments like we were working with hallucination index two years back where we were using our own similar LM as judge methodology and we had certain issues and there was some correlation issues at the time but last year it got pretty strong and this year I feel it's so strong I have LM as judges”

Claim: The speaker recommends adding instructions and conducting initial validation to determine where an evaluation works, and returning to annotation if it does not.

“just go back to annotation but I feel like it has started working a lot one year back because we I was running this experiments like we were working with hallucination index two years back where we were using our own similar LM as judge methodology and we had certain issues and there was some correlation issues at the time but last year it got pretty strong and this year I feel it's so strong I have LM as judges”

Claim: The speaker reports that his LLM-as-judge experiments for a hallucination index had correlation problems two years ago, improved substantially last year, and are very strong this year.

“This is so strong that you can use it. I've gone through like hundreds of explanations like our LLMs also generate explanations that users can read to validate whether it actually went to this correct reasoning or not.”

Claim: The speaker says Galileo's LLMs generate explanations for users to inspect and verify whether the reasoning was correct, and that he has reviewed hundreds of these explanations.

“And I have gone through many explanations in the during this V2 experiment that I'm doing right now and also the V1 and I did not find any issues so”

Claim: The speaker says he reviewed many explanations in the current V2 experiment and the earlier V1 experiment and found no issues.

“I feel like when we so that's why we released all the data set people can go and run the experiment with the v2 also we are going to release the data sets people can just take them and run it on our cluster it's free our product is free so anybody can just sign up and start using it they can just put the open AI key and they will get a clarity okay it's working or not we want to be as open as possible while still being a closed source company so I feel like it's working and people should definitely pay attention to it”

Claim: Galileo aims to be as open as possible while remaining closed source.

“I feel like when we so that's why we released all the data set people can go and run the experiment with the v2 also we are going to release the data sets people can just take them and run it on our cluster it's free our product is free so anybody can just sign up and start using it they can just put the open AI key and they will get a clarity okay it's working or not we want to be as open as possible”

Claim: Galileo says it has released a dataset and plans to release V2 datasets for public experimentation; its product is free, and users can sign up and use an OpenAI key to test the evaluation.

“while still being a closed source company so I feel like it's working and people should definitely pay attention to it”

Claim: The speaker believes LLM-as-judge evaluation is working and recommends that people pay attention to it.

“So I was not working on fine tuning the LLM as a judge, it is done by a separate team member and once the matrix is ready only then I start using it.”

Claim: The speaker says a separate team member handled fine-tuning the LLM judge, and he started using it only after the matrix was ready.

“That was they went through like a months of rigorous process of prompt tuning and it's a very structured prompt it's like an art in itself as you know where we have certain sections for certain things on certain instruction and what we do is like it's very simple philosophy we start with a base prompt with what we think is correct and then we see where it's failing and then we try to add those instructions again. Again, defining the failure cases in a general way to the prompt of the LLM is just and see if it's following it or not and that iteration goes on for multiple times till we are sure that we are getting the performance that we need.”

Claim: The team iteratively improves a base prompt by identifying failure cases and adding instructions to address them.

“That was they went through like a months of rigorous process of prompt tuning and it's a very structured prompt it's like an art in itself as you know where we have certain sections for certain things on certain instruction and what we do is like it's very simple philosophy we start with a base prompt with what we think is correct and then we see where it's failing and then we try to add those instructions again.”

Claim: The LLM-judge prompts underwent months of rigorous tuning and were organized into sections for different instructions.

“Again, defining the failure cases in a general way to the prompt of the LLM is just and see if it's following it or not and that iteration goes on for multiple times till we are sure that we are getting the performance that we need.”

Claim: The team describes failure cases in general terms in the prompt and iterates until it is confident the LLM judge meets the required performance.

“So it's very common sense oriented approach I would say and it's just plain English writing with just following the right way of writing prompts.”

Claim: The speaker characterizes the prompt-tuning approach as common-sense-oriented and involving plain-English prompt writing according to good prompting practices.

“And of course, the leaderboard, if you are using different judges with different models, there can be heavy shakeup of the leaderboard also.”

Claim: A leaderboard can change substantially when different judge models are used.

“So that's why we use the best LLM available at that when we are starting or near completion of the experiments.”

Claim: Galileo tries to use the best available LLM at the start of an experiment or near its completion.

“We try to use the best LLM available so that it should not be a situation where we are trying to evaluate 3.7 with GPT-4.0 meaning, right? uh risky proposition because gpt4o mini can't find uh cannot evaluate the hard outputs that 3.7 might be doing correctly right um so and so for this v2 i am using the 3.7 because it is one of the top reigning models right now and i've seen really good performance from it so we essentially swap these models as much as”

Claim: The speaker considers it risky to use a weaker judge, such as GPT-4o mini, to evaluate difficult outputs from a stronger model such as 3.7, because the judge may fail to recognize correct outputs.

“uh risky proposition because gpt4o mini can't find uh cannot evaluate the hard outputs that 3.7 might be doing correctly right um so and so for this v2 i am using the 3.7 because it is one of the top reigning models right now and i've seen really good performance from it so we essentially swap these models as much as just possible when we are running the experiment otherwise it looks way we have seen situation previously where we were using probably gpt-4 turbo or something or even gpt-3.5 long back i'm talking like two years back um and we had to use gpt-4 turbo at the time because with the 3.5 it was something else entirely because it can't find errors”

Claim: The team changes judge models as much as possible during experiments.

“uh risky proposition because gpt4o mini can't find uh cannot evaluate the hard outputs that 3.7 might be doing correctly right um so and so for this v2 i am using the 3.7 because it is one of the top reigning models right now and i've seen really good performance from it so we essentially swap these models as much as”

Claim: The speaker says he is using 3.7 for the V2 experiment because it is one of the leading models at the time and has performed well in his experience.

“just possible when we are running the experiment otherwise it looks way we have seen situation previously where we were using probably gpt-4 turbo or something or even gpt-3.5 long back i'm talking like two years back um and we had to use gpt-4 turbo at the time because with the 3.5 it was something else entirely because it can't find errors”

Claim: The speaker says that about two years earlier his team used GPT-4 Turbo rather than GPT-3.5 because GPT-3.5 could not identify errors in the outputs.

“What we found issues with V1 is It has a lot of domains, but it does not have a domain vertical in itself that, okay, if somebody is interested on how this model perform in, let's say, telecommunication, they can”

Claim: V1 covers many domains but lacks a dedicated vertical for evaluating model performance in a specific domain, such as telecommunications.

“we are doing domain specific. which means that these are like new data sets from scratch and why are we doing this is of course the first problem is that they are not available easily secondly it's because which is I would say the main reason is that we don't want leakage at all to be possible in any of the LLM so far so we want that these are new data sets so we guarantee that it's going to be performing”

Claim: The evaluation uses newly created, domain-specific datasets because existing suitable datasets are not readily available and to prevent data leakage into the evaluated LLMs.

“which means that these are like new data sets from scratch and why are we doing this is of course the first problem is that they are not available easily secondly it's because which is I would say the main reason is that we don't want leakage at all to be possible in any of the LLM so far so we want that these are new data sets so we guarantee that it's going to be performing”

Claim: The newly created datasets are intended to make the evaluation as unbiased as possible.

“as unbiased as possible then we saw with v1 that the scores are saturating we see that the best score has already reached 0.95 and so if we are going to evaluate even better models that are coming out they'll be definitely in the range of 0.9 to 1 which i feel is not very informative for anybody to take a call because there can be some noise also in this evaluation so we want the benchmarks”

Claim: The speaker expects that better models evaluated on the saturated benchmark would score between 0.9 and 1, making results less informative for decisions because evaluation noise could affect them.

“as unbiased as possible then we saw with v1 that the scores are saturating we see that the best score has already reached 0.95 and so if we are going to evaluate even better models that are coming out they'll be definitely in the range of 0.9 to 1 which i feel is not very informative for anybody to take a call because there can be some noise also in this evaluation so we want the benchmarks”

Claim: In version 1 of the benchmark, scores were saturating and the best score had reached 0.95.

“to be harder and this tool calling to be more complicated and realistic at the same time this bfcl if you just open and check it you would be like okay this is okay but you don't want to use it for realistic evaluation because it's like some math problem statement is given some simple tool is given just like let's say some exponential to logarithmic to multiplication stuff and you are just like evaluating on that so I was not particularly happy with that but that's the best we could got get at the time and that's why we mixed up all the data sets available at the time to kind of average out the things because very specific right if some model is good at math then it might perform really good”

Claim: The speaker considered BFCL's simple math-style tool-calling tasks unrealistic for practical evaluation, while acknowledging that BFCL was the best available option at the time.

“to be harder and this tool calling to be more complicated and realistic at the same time this bfcl if you just open and check it you would be like okay this is okay but you don't want to use it for realistic evaluation because it's like some math”

Claim: The speaker wants the benchmarks to use harder tasks and more complicated, realistic tool calling.

“problem statement is given some simple tool is given just like let's say some exponential to logarithmic to multiplication stuff and you are just like evaluating on that so I was not particularly happy with that but that's the best we could got get at the time and that's why we mixed up all the data sets available at the time to kind of average out the things because very specific right if some model is good at math then it might perform really good Great for BFCL but if some model doesn't understand the terminologies of industry then it will not perform well for Tau. So we kind of mixed up the data set during that time.”

Claim: The speaker's team combined the datasets available at the time to average out differences in what they measured.

“problem statement is given some simple tool is given just like let's say some exponential to logarithmic to multiplication stuff and you are just like evaluating on that so I was not particularly happy with that but that's the best we could got get at the time and that's why we mixed up all the data sets available at the time to kind of average out the things because very specific right if some model is good at math then it might perform really good”

Claim: A model with strong math ability might perform very well on BFCL.

“Great for BFCL but if some model doesn't understand the terminologies of industry then it will not perform well for Tau.”

Claim: A model that does not understand industry terminology will perform poorly on Tau.

“So two things domain and one is increasing the hardness. Third thing is multi-turn evaluation.”

Claim: The evaluation is being made domain-specific and harder, and is being expanded to include multi-turn evaluation.

“coordinate the tools and then kind of adversarial scenarios customers manipulating the bot to kind of do certain things and”

Claim: The evaluation includes adversarial customer scenarios that attempt to manipulate the support bot into taking particular actions.

“Yeah multi-turn coming to multi-turn the in the multi-turn what we are doing is where this time last time it was more like there is an input with one message or multiple messages and then give one output that was how we are evaluating this time it's like a realistic scenario that we start with a scenario scenario as a first message and there is like goals to be accomplished this scenario is given to a user simulator”

Claim: The earlier evaluation used one or more input messages and a single output, whereas the new evaluation begins with a scenario and goals to accomplish.

“Now user simulator is basically a prompt which gets the whole history of the chat and it's supposed to generate a new message and that simulation is ran by GPT 4.1 mini model.”

Claim: In the evaluation, a user simulator receives the full conversation history and generates the next user message, using GPT-4.1 mini to run the simulation.

“Why we selected tool simulator essentially because of ease of running the simulation.”

Claim: The speaker selected a tool simulator primarily to make the evaluation simulation easier to run.

“As we saw with Tau, Tau has like tool definitions and databases with CSV file kind of thing with read-write operations going on.”

Claim: Tau provides tool definitions and CSV-file databases that support read-write operations.

“Now I wanted to have like many tools created. We have like 20 tool per domain and we have like Five domains, so it's like 100 tools, I do not have the time to kind of write this tool with all these databases,”

Claim: The evaluation setup called for 20 tools in each of five domains, totaling 100 tools.

“Five domains, so it's like 100 tools, I do not have the time to kind of write this tool with all these databases, so I kind of created a tool simulator, again with the GPT 4.1 mini model,”

Claim: Because the speaker lacked time to implement 100 tools and their databases, they created a tool simulator with GPT-4.1 mini.

“which takes into the input as a tool call. So it gets the tool call, it gets the response of the tool call schema and then it is supposed to give an output. So rather than like interacting with an API or calling certain databases, it is going to generate an output based on the history of the conversation and the tool call information that is given.”

Claim: The tool simulator generates tool outputs from the conversation history and tool-call information instead of calling APIs or accessing databases.

“And the main one that we are evaluating is basically the agent. The agent we are swapping all the LLMs to evaluate.”

Claim: The evaluation tests different LLMs by swapping them into the agent.

“And this agent basically is accomplishing the task given by the user. and then user gives a task, the agent either gives a response or gives the tool call.”

Claim: In the evaluation, the agent attempts the user's task by either responding directly or making a tool call.

“If it's a tool call, it's given to the tool simulator and the tool simulator will give an output and then that output will be used by the agent to give a response to the user.”

Claim: When the agent calls a tool, the tool simulator generates an output that the agent uses to formulate its response to the user.

“for every scenario we have like at least like five to seven goals to be accomplished.”

Claim: Each evaluation scenario contains at least five to seven goals that the agent must accomplish.

“They are pretty complex like tau and they all need to be accomplished otherwise it's the task is not accomplished so this time you have to get done many things at once.”

Claim: The evaluation scenarios are complex, and an agent fails to complete a task if it does not accomplish every goal in that scenario.

“which makes the thing hard and it's possible that you might miss out on certain things and we see that's the best models are good at it but certain models are really struggling on it and they can go into iterations for no reason and so the scores are coming out to be lower much lower like the main problem point nine five”

Claim: Complex multi-goal tasks make the evaluation harder: some models miss requirements or enter unnecessary iterations, while the best models perform well and overall scores are much lower.

“is like salt that okay the scores are lower we are getting domain specific insights and then it's a very realistic scenario where you are having multiple turn and then only finally when everything is done then the task is accomplished so what how do we evaluate the performance in this case and we use two metrics we use the older one the tool selection quality which is just telling us whether the tool selected was correct or not”

Claim: The tool selection quality metric measures whether the agent selected the correct tool.

“but other thing which is super important from a business point of view and practical point of view is that whether the agent completed all the task or not right so for that we have the action completion metric which will check that whether all the requests that were given by the user were all of them accomplished and were they accomplished correctly or not so”

Claim: The action completion metric measures whether every user request was completed correctly.

“Kind of focusing more on action completion metric I would say which is telling us that the user asked for these five things and whether the agent completed all the five things or not.”

Claim: The evaluation places greater emphasis on action completion because it shows whether the agent completed every item requested by the user.

“So that's how we are going to evaluate it. We'll also have the tool selection quality metric but that doesn't tell us the whole story. It just tells a very granular story. It's a debugging metric but this is like an overall performance metric for the user.”

Claim: Tool selection quality is a granular debugging metric rather than a complete account of performance; action completion provides an overall user-oriented performance measure.

“And then so essentially a user workflow will be like in the realistic production workflow is that they have this agent working and in certain scenarios the action completion is lower, so they will go there and then check what went wrong and also they might see some tool selection issues and they will debug them.”

Claim: In production, users can investigate scenarios with low action completion and debug problems such as incorrect tool selection.

“It's difficult to tune prompts to get the right direction of what you want to achieve because you want to be realistic and the hard part for me.”

Claim: Tuning prompts to produce the desired direction while keeping scenarios realistic is difficult.

“that I have spent a lot of time was generating hard scenarios because you take a hard more you take the best model to generate hard scenarios and you then see if the best model can accomplish those tasks or not like let's say I generate by 3.7 and then check on 4.1 and then it's able to accomplish so I had to kind of do a lot of iterations on like finding where the model fails and how to trigger this kind of scenarios”

Claim: The speaker iterated to identify model failure points and determine how to elicit those failures with challenging scenarios.

“that I have spent a lot of time was generating hard scenarios because you take a hard more you take the best model to generate hard scenarios and you then see if the best model can accomplish those tasks or not like let's say I generate by 3.7 and then check on 4.1 and then it's able to accomplish so I had to kind of do a lot of iterations on like finding where the model fails and how to trigger this kind of scenarios”

Claim: Generating challenging evaluation scenarios took substantial time; the speaker used a strong model to create scenarios and tested whether that model could complete the tasks.

“We have reached a certain stage where we feel confident that yes, there is taken hard scenarios and the model is not able to accomplish all the time. So this is going to like reset, this is going to calibrate the expectation of the users of this LLMs because our last leader was saying that models are really great at tool calling. So it's like safe, but they're actually not right. They're making mistakes and this is going to recalibrate the expectation.”

Claim: The speaker expects the evaluation to recalibrate users' expectations of LLMs, which make mistakes even though they are often characterized as very capable at tool calling.

“We have reached a certain stage where we feel confident that yes, there is taken hard scenarios and the model is not able to accomplish all the time. So this is going to like reset, this is going to calibrate the expectation of the users of this LLMs because our last leader was saying that models are really great at tool calling.”

Claim: The speaker's team believes its evaluation scenarios are sufficiently challenging that the model does not succeed on them consistently.

“Expectation of the users that be careful because they are still not perfect in real scenarios.”

Claim: Users should be cautious when relying on LLMs in real-world scenarios because the models are not yet perfect.