Pratik said the leaderboard aimed to provide insights beyond performance, including cost, small-versus-large model comparisons, and open-source-versus-private model comparisons.

“stop models in let's say Alemarina or other specific evaluation or general evaluations they might not be also the same ranking for other task like agentic task so that was our goal very fundamentally speaking that we want to get very specific insights apart from just performance and cost was the other thing that comes up a lot with enterprise customers and”

“That was the second focus essentially and then there is ASLM versus LLM kind of debate are the small models also catching up and there is the big one open source versus private.”

“So those were the dimensions we were interested in.”