⚡️Composio: 10,000+ tools that evolve for Agents — Karan Vaidya and Soham Ganatra

“Yeah, we are kind of self-evolving skills now for agents. So people building or using agents want to like.”

Claim: Composio provides skills for AI agents to interact with users’ applications, and those skills evolve over time.

“like basically make agents interact with their apps they can use composio we manage all the authentication and user account related stuff for them so that they don't have to take care of all that and provide agent friendly skills which can be anything from like a direct api call to a somewhat more complex like directly agent giving natural language tasks to an app and the idea is that kind of like the skills evolve over time so we analyze the”

Claim: Composio manages authentication and user-account-related tasks for agents, relieving agent builders and users of that work.

“like basically make agents interact with their apps they can use composio we manage all the authentication and user account related stuff for them so that they don't have to take care of all that and provide agent friendly skills which can be anything from like a direct api call to a somewhat more complex like directly agent giving natural language tasks to an app and the idea is that kind of like the skills evolve over time so we analyze the”

Claim: Composio’s agent-friendly skills range from direct API calls to skills that let an agent give an app a task in natural language.

“like basically make agents interact with their apps they can use composio we manage all the authentication and user account related stuff for them so that they don't have to take care of all that and provide agent friendly skills which can be anything from like a direct api call to a somewhat more complex like directly agent giving natural language tasks to an app and the idea is that kind of like the skills evolve over time so we analyze the is the patterns of how the agents are using these skills and kind of like like the specific usage patterns of a particular agent and improve these skills over time so that the agent take reliability is on a higher side over time.”

Claim: Composio analyzes agents’ skill-usage patterns and improves the skills over time to increase agent reliability.

“So, yeah, I think firstly MCB is really amazing and but it has its own quirks of how like it is good for some over like like universal clients and not that great for like people building on top of.”

Claim: MCP works well for universal clients connecting to different applications, but is less well suited to developers building products on top of MCP.

“So like MCB is great, for example, if you want to connect your chat, GPT, cloud cursor sort of a universal client with different apps.”

Claim: MCP can connect universal clients such as ChatGPT and Cursor to different applications.

“It provides a ton of like servers that people have already built. So like. You can connect with all the tail end of applications but even in the AI engineering conference I was kind of like looking at the MCP stream and the MCP creator mentioned that they are optimizing the protocol so that like the ease of server creation is on the higher side and not so much for the client so what that means is like kind of people building on top of MCP can't customize a lot of things and like it's kind of like”

Claim: MCP offers many already-built servers that can connect clients to a wide range of applications.

“You can connect with all the tail end of applications but even in the AI engineering conference I was kind of like looking at the MCP stream and the MCP creator mentioned that they are optimizing the protocol so that like the ease of server creation is on the higher side and not so much for the client so what that means is like kind of people building on top of MCP can't customize a lot of things and like it's kind of like”

Claim: MCP’s emphasis on making server creation easy rather than optimizing for clients limits how much developers building on MCP can customize.

“The DX of creating on top of MCP is somewhat difficult while we are optimizing ComposeU for the like server side of like the client side of things so that like people building has have a very easy experience of connecting with tools.”

Claim: Building products on MCP currently offers a somewhat difficult developer experience.

“The DX of creating on top of MCP is somewhat difficult while we are optimizing ComposeU for the like server side of like the client side of things so that like people building has have a very easy experience of connecting with tools.”

Claim: Composio is optimized for client-side use to make it easy for developers to connect tools.

“In addition I think MCP is one of the protocols there might be a lot more protocols that might come over time but there needs to be an overarching learning infrastructure that needs to be that needs to be developed be it MCP or.”

Claim: MCP is one of several protocols, and additional protocols may emerge over time.

“In addition I think MCP is one of the protocols there might be a lot more protocols that might come over time but there needs to be an overarching learning infrastructure that needs to be that needs to be developed be it MCP or. or function calling so that when one of the agent discovers an edge case, let's say while connecting to Salesforce like other agents also benefit from that. So that's what Composio is. We over like we connect via,”

Claim: A shared learning infrastructure across protocols is needed so that solutions to edge cases encountered by one agent can benefit other agents.

“we will connect to Composio tools via MCP, via function calling, via codec, multiple formats, and like all of them have the same learning infrastructure.”

Claim: Composio supports access to its tools through MCP, function calling, and other formats, all backed by the same learning infrastructure.

“So I think MCB, as I mentioned, is great for like the server side of things, but they can do a lot of improvements on the client to make it easier to connect to different servers.”

Claim: MCP works well on the server side, but client-side improvements could make it easier to connect to different servers.

“So like Kodak is becoming somewhat more real. We have seen like recently a surge in how people use composure tools via like like directly in code generation so I think that still is somewhat new so like there might be some changes that needs to that need to be done to improve on that”

Claim: The speaker has recently observed increased use of Composio tools directly in code generation; this use case is still new and may require changes to support it.

“The first answer like how MCB could be but so some of the limitations that we've been observing in production is like MCB, let's just say you have a tool that has really long context, really long set of response. Currently if you use MCB for those kind of tools, it ends up sort of messing up, right? Because let's say I'm trying to fetch like 100 emails on Gmail. and my sort of context over blockers and I'm not able to sort of post process the output that is coming out from MCP, I'm not able to sort of play around with it before it reaches my sort of final lab and a lot of like potential failure scenarios like that exist. Some of the other things that we have observed is like hey maybe I want to sort of change the tool descriptions but like in MCP the tool descriptions are controlled by the server itself and so essentially for me as a developer who's sort of”

Claim: In production, MCP can struggle with tools that return long responses: fetching 100 Gmail emails may overwhelm context and leave developers unable to post-process the output before it reaches the final application.

“and my sort of context over blockers and I'm not able to sort of post process the output that is coming out from MCP, I'm not able to sort of play around with it before it reaches my sort of final lab and a lot of like potential failure scenarios like that exist. Some of the other things that we have observed is like hey maybe I want to sort of change the tool descriptions but like in MCP the tool descriptions are controlled by the server itself and so essentially for me as a developer who's sort of using an mcb to build on top like messing around with tone descriptions or messing around between schemas becomes a very let's just say painful experience uh to actually do that so a lot of like this client side thinking just like how do you optimize the whole thing so that for a developer who's sort of building on top is kind of missing today and truthfully like when you talk about like new product like so many thoughts merged in my brain but that but the way i would put it is”

Claim: Because MCP servers control tool descriptions, developers building on MCP find it difficult to modify tool descriptions or schemas.

“using an mcb to build on top like messing around with tone descriptions or messing around between schemas becomes a very let's just say painful experience uh to actually do that so a lot of like this client side thinking just like how do you optimize the whole thing so that for a developer who's sort of building on top is kind of missing today and truthfully like when you talk about like new product like so many thoughts merged in my brain but that but the way i would put it is”

Claim: MCP currently lacks sufficient client-side features and considerations for developers building on top of it.

“I think we are like really sort of in the emerging phase, few different lines of thinking I could get into on like how new protocols could look like, but I don't really have a concrete answer to like where it should be.”

Claim: The field of agent protocols is still emerging, and the speaker does not have a concrete view on what new protocols should look like.

“In Alisa we have an agent essentially which builds a lot of these skills and kind of like does a lot of like testing as well as eval so it's like you can say like agent as an eval where our internal agent uses these tools and kind of like we figure out where it is messing up and improve our time even asking agent what we can do better etc so that like that's one then obviously we have like how the agent is using it where it's getting stuck that data to improve to overtime.”

Claim: Composio uses an internal agent to build, test, and evaluate skills, then uses observed failures and feedback to improve them over time.

“So I think like MCP, by the way, so one of the really cool things that it ended up doing was it ended up sort of creating a very simple way to sort of integrate with universal flags. So for composer tools to be used in cursor today, it's super easy.”

Claim: MCP provides a simple way to integrate tools with Cursor.

“Anyone can go start using it and that creates a very... sort of interesting way for us to actually create event on top because then I can allow anyone who's using compositional tools via cursor to essentially give me feedback right inside the cursor chat window saying hey will we able to sort of execute on what you wanted to execute and and so we are getting a lot of like human feedback through this and we are able to sort of somehow accumulate some part of that feedback into like a internal sort of eval”

Claim: Composio collects user feedback through Cursor’s chat window and incorporates some of it into an internal evaluation.

“uh so again tool calling has a lot of different evals you have seen the function tool calling evals you have multifunction like like multi multi-time tool calling ones but like none of them are actual like production quality like like none of them are actually something that we are able to rely on truthfully because like they don't really represent the situations that occur like the orchestration issues that occur uh in a complicated sort of scenarios that that you often see in production and so this like getting the feedback through mcp clients and essentially using that in some way is something that we are actually finding it way better for at least our internal sort of enterprises.”

Claim: Composio has found feedback gathered through MCP clients more useful than existing tool-calling evaluations for its internal enterprise use.

“uh so again tool calling has a lot of different evals you have seen the function tool calling evals you have multifunction like like multi multi-time tool calling ones but like none of them are actual like production quality like like none of them are actually something that we are able to rely on truthfully because like they don't really represent the situations that occur like the orchestration issues that occur uh in a complicated sort of scenarios that that you often see in production and so this like getting the feedback through mcp”

Claim: The speaker considers existing function-calling and multi-tool-calling evaluations unreliable for production because they do not represent orchestration problems in complex real-world scenarios.

“So 95% of all our integrations are completed using agents maintained using agents.”

Claim: Agents complete and maintain 95% of Composio’s integrations.

“I think at this point we have a repository with prevent more than three million lines of code completely by doing maintenance all the integrations out there that you could think of and”

Claim: Composio’s integration-maintenance repository contains more than three million lines of code.

“the enterprise tools that have really shitty documentation and you basically just wish that if they are taking like maybe 10% more time documenting the edge cases I would have solved it and like more than that the problem is that you need a dev environment to sort of run these agents on like a lot of this enterprise tools don't really have that dev environment like a very simple example would be okay I want to sort of integrate with Rippling but like my dev instance of Rippling has no extra data and so what am I going to do answer like you do your best case analysis of like what the API is like looking like and how to go about it and then you try that and like real sort of applications that I'm talking about are like work day and like that really tight right in terms of the documentation sort of put out right so yeah those kind of applications.”

Claim: Composio’s integrations that are not agent-maintained tend to involve enterprise tools with poor edge-case documentation or without a usable development environment and test data.

“So we have at this point more than 500 different applications on top of Composer. We are adding close to at this point,”

Claim: Composio has more than 500 applications on its platform.

“like we were adding like close to so during the a week like a month back now that like a lot of sort of improvements have been done on the agent side we added like last month has been like basically 35 week uh we're planning to set out times that up to like close to 100 a week in next month”

Claim: Composio added approximately 35 applications per week in the previous month.

“so during the a week like a month back now that like a lot of sort of improvements have been done on the agent side we added like last month has been like basically 35 week uh we're planning to set out times that up to like close to 100 a week in next month”

Claim: Composio plans to add nearly 100 applications per week in the next month.

“planning to reach like a target of like 5,000 applications by end of the sort of year which would like put us in the similar lines of like all the sort of like which would essentially encompass all the major applications you could think of in terms of developers we have like more than 100,000 developers who have been building on top of us and like trying our product out yeah I guess what would the numbers seem kind of.”

Claim: Composio aims to have 5,000 applications by the end of the year.

“planning to reach like a target of like 5,000 applications by end of the sort of year which would like put us in the similar lines of like all the sort of like which would essentially encompass all the major applications you could think of in terms of developers we have like more than 100,000 developers who have been building on top of us and like trying our product out yeah I guess what would the numbers seem kind of.”

Claim: More than 100,000 developers have built on or tried Composio’s product.

“Sure. So I think like we have seen over like a lot of practice that like more than like 20-25 actions like just confuses the server like it's not able to kind of like figure out which tool to use and same goes with like if the schema of the tools are really complex that also confuses the agent so we try to keep our kind of like schemas simple like some of the things are like flattening the schemas etc.”

Claim: The Composio team has found that exposing more than roughly 20–25 actions can confuse an agent about which tool to use.

“Sure. So I think like we have seen over like a lot of practice that like more than like 20-25 actions like just confuses the server like it's not able to kind of like figure out which tool to use and same goes with like if the schema of the tools are really complex that also confuses the agent so we try to keep our kind of like schemas simple like some of the things are like flattening the schemas etc.”

Claim: Complex tool schemas can confuse agents, so the Composio team tries to simplify schemas, including by flattening them.

“Like basically I think it's like agents like because they are LLMs inherently so they like to think in natural language so some of the thing that we are planning on that front is like a single natural language execution of a tool so that like the agent just gives like like agent has just given a single tool and can call any of the other kind of skills that we have like inside Composeo with just natural language so like create a notion doc and send a meeting invite just in natural language and we'll do that.”

Claim: Agents, being LLMs, inherently tend to think in natural language.

“Like basically I think it's like agents like because they are LLMs inherently so they like to think in natural language so some of the thing that we are planning on that front is like a single natural language execution of a tool so that like the agent just gives like like agent has just given a single tool and can call any of the other kind of skills that we have like inside Composeo with just natural language so like create a notion doc and send a meeting invite just in natural language and we'll do that.”

Claim: Composio is planning a single natural-language tool interface that lets an agent invoke its other skills, such as creating a Notion document and sending a meeting invitation.

“We do have a composure search tool, which kind of essentially does exactly what you said, like searches across all the, like we have around like 15,000 plus actions that we support across composure tools, so it can search all across depending on the. think on the natural language use case that you provide and kind of like then execute that specific action yeah”

Claim: Composio's search tool searches its roughly 15,000-plus supported actions according to a supplied natural-language use case and can execute the selected action.

“So they're like few, right? Like Gorilla was one, but then like a bunch of others emerged, And they were all sort of more on the face of like, is an LLM sort of good with tool calling,”

Claim: The speaker characterizes existing benchmarks such as Gorilla as primarily evaluating whether an LLM can perform tool calling well.

“We are trying to create a different kind of benchmarks. It's like, how good is this tool for LLM to be sort of like calling it like in the correct fashion,”

Claim: The team is developing benchmarks that assess whether a tool is designed so an LLM can call it correctly.

“And this is more like in the lines of DSPY. if you look at like how they're like like they sort of engineered the prompt and stuff you're trying to engineer the sort of tool in a way that like it becomes easier for LLM to sort of call it like internally we have essentially the way I would go about it is like you can break it down into terms like we have figured out the right heuristics that we have seen essentially while better for LLMs and then we have also sort of given like a A-B testing kind of infrastructure for LLMs to go ahead and like test out with.”

Claim: The speaker compares tool design to prompt engineering, arguing that tools can be engineered to make them easier for LLMs to call.

“if you look at like how they're like like they sort of engineered the prompt and stuff you're trying to engineer the sort of tool in a way that like it becomes easier for LLM to sort of call it like internally we have essentially the way I would go about it is like you can break it down into terms like we have figured out the right heuristics that we have seen essentially while better for LLMs and then we have also sort of given like a A-B testing kind of infrastructure for LLMs to go ahead and like test out with.”

Claim: The team uses identified heuristics and A/B-testing infrastructure to test ways of improving tools for LLM use.

“So there are like two kind of optimizations we do, right? One is like, is the tool through the API transformation, It is like, is this Salesforce tool getting transformed into the right request payload so that it is hitting the right endpoint in Salesforce, And that is like the major source of error today because a lot of documentations are not great, a lot of like catch cases are not mentioned and you kind of keep seeing that issue. And then the second feedback that we kind of optimize for, which is like, is the Salesforce tool really sort of simplified so that an MLM can call it nicely?”

Claim: The team optimizes tools by evaluating both the correctness of their API transformations and how easily an LLM can call them.

“It is like, is this Salesforce tool getting transformed into the right request payload so that it is hitting the right endpoint in Salesforce, And that is like the major source of error today because a lot of documentations are not great, a lot of like catch cases are not mentioned and you kind of keep seeing that issue.”

Claim: The speaker says incorrect API request transformations are a major current source of errors, in part because API documentation is often poor and edge cases are often omitted.

“And that optimization essentially is built on a bunch of sort of internal like benchmarks that we have, which is like which are around. Here's like a query that we saw. Here's a query that LLM couldn't really transform into the right set of Salesforce call. That is the reason Salesforce API endpoint doesn't get into failure. How do we sort of like make sure that we are able to get those kind of queries working nicely, right? And like we keep seeing the meta patterns.”

Claim: The team uses internal benchmarks built from queries that LLMs failed to translate into correct Salesforce calls, then works to improve tool behavior on those queries.

“say a module and then we do a bit testing here which is like if we would have built the tool in this way and like all of this is through agents of course we can't do it manually for all the tools but we start doing a bit testing that way so like if we had sort of built the tool in this way maybe this query would have been easier resolved right and the the reason sort of mcp helps here is because if you know mcp allows you to do dynamic to it sort of like exposure which means essentially like based on the query i can decide what what tools to expose and how to go about it and so we can do that so we can essentially decide that for the same Salesforce for the same query we can expose two different kinds of Salesforce tools to the MCP client and just see like what tool is being better sort of used here so that's the infrastructure we're building.”

Claim: MCP's dynamic tool exposure lets a system select which tools to expose according to the user's query.

“say a module and then we do a bit testing here which is like if we would have built the tool in this way and like all of this is through agents of course we can't do it manually for all the tools but we start doing a bit testing that way so like if we had sort of built the tool in this way maybe this query would have been easier resolved right and the the reason sort of mcp helps here is because if you know mcp allows you to do dynamic to it sort of like exposure which means essentially like based on the query i can decide what”

Claim: Because it cannot manually test every tool, the team uses agents for some testing of whether alternative tool designs make queries easier to resolve.

“what tools to expose and how to go about it and so we can do that so we can essentially decide that for the same Salesforce for the same query we can expose two different kinds of Salesforce tools to the MCP client and just see like what tool is being better sort of used here so that's the infrastructure we're building.”

Claim: The team is building infrastructure to expose different versions of Salesforce tools to an MCP client for the same query and compare which version is used more effectively.

“what you can do is like you can decide what tools to expose, right? So since the Salesforce server is with us, like the MCP server, we can decide like in 20% cases, we want to expose these tools and in the other 80% cases, we want to expose these tools.”

Claim: Composio can configure its Salesforce MCP server to expose different tool sets in different cases, including one set in 20% of cases and another set in 80% of cases.

“And then we can expose like different tools for different people.”

Claim: Composio can configure different users to receive different exposed tools.

“And so we can know like what tools are essentially getting better results.”

Claim: Composio can identify which tools produce better results.

“I mean, that's also one more layer of dynamic CPT that is coming, which is like in MCP, you can actually change tools and you can notify the client that you have changed tools while MCP is like I can share that with you.”

Claim: MCP supports changing available tools dynamically and notifying the client about those changes.

“The idea here is that eventually we want to sort of create a layer where like we can like keep like changing tools and like recommending better and better tools.”

Claim: Composio aims to build a layer that dynamically changes available tools and recommends better tools over time.

“Yeah, I think as I was mentioning kind of like one of the tools like or kind of like servers that we support which we internally name as like agent API is essentially that where kind of like the whole end-to-end use case to. Tool formation like kind of like tool calling formation is done by us so the client just gives like for example that it wants to schedule meeting with like five people probably just names and we figure out all their email addresses beneath the kind of like the server and kind of creates create their meeting invite with that.”

Claim: Composio's agent API handles tool-call formation on the server, including resolving invitees' email addresses from their names and creating a meeting invitation when a client requests a meeting with five people.

“it's not just like the agent can take action but they can get notified over time so like essentially when you like when you receive an email or when you receive a slack message or like anything that you can use to spawn an agent that it can work on top of the other thing that I'm like really excited about and we are optimizing our tools for is like Kodak where a lot of like a lot of tool calling is happening via generated code so you can use Composeos”

Claim: Composio triggers can notify and start an agent in response to events such as receiving an email or Slack message.

“it's not just like the agent can take action but they can get notified over time so like essentially when you like when you receive an email or when you receive a slack message or like anything that you can use to spawn an agent that it can work on top of the other thing that I'm like really excited about and we are optimizing our tools for is like Kodak where a lot of like a lot of tool calling is happening via generated code so you can use Composeos actions not directly in like mcp or tool calling but like expose them as functions which the llm can use while generating code so that's very interesting and we are doing a lot of optimizations on that front yeah”

Claim: Composio is optimizing its actions for code-generation workflows by exposing them as functions that an LLM can call while generating code, rather than only through MCP or conventional tool calling.

“We built it like a year back. nobody used it for a very long time today it's probably one of the most used tools the idea here was that we just wanted anyone to just install composure and inside composure and that's pretty much it you don't need to think about anything else like that Yeah, yeah”

Claim: Composio built its self-integration tool about a year ago; after going unused for a long time, it is now probably one of its most-used tools.

“turn up this talking about it but essentially why do you need like a layer of tools because you want to sort of execute stuff on the client right but let's say if a server is smart enough and like basically if i build something very interesting and i say hey you don't have to worry about anything just send me the query and let me do all of the stuff then like you don't need a layer of tools then you just need one tool like hey here's the query go do it right and like that's the idea which is like compositing eventually gets to that that point but like you just give us the query and we just do it that's”

Claim: Composio aims eventually to let users submit a query for the server to execute, replacing the need for users to manage a layer of tools with a single tool.

“I think it's anyway hard to kind of hire really good engineers and SF, you have like a lot of competition all around.”

Claim: Hiring highly skilled engineers in San Francisco is difficult because employers face substantial competition.

“So kind of like that was the idea where we want some of the best engineers to work at Composer.”

Claim: Composio intended its hiring incentive to help attract highly skilled engineers to the company.

“so yeah i mean like when grok 4 came out we were like experimenting with it and like just developed the cli because there was none like for clod you had clod code you can use probably codex”

Claim: When Grok 4 was released, the speaker's team experimented with it and built a command-line interface because one was not available.

“with Grok4 as well but I think the idea was I think Grok4 actually has some of the really cool agentic capabilities so kind of like I think I really love Claude Sonet for example because of the same and Grok4 had like some of a mix of Opus and Sonet which I really like so kind of like wanted to use a CLI to use that on my code base so kind of just developed it overnight not”

Claim: The speaker particularly likes Claude Sonnet because of its agentic capabilities, which the speaker sees as similar to Grok 4's.

“with Grok4 as well but I think the idea was I think Grok4 actually has some of the really cool agentic capabilities so kind of like I think I really love Claude Sonet for example because of the same and Grok4 had like some of a mix of Opus and Sonet which I really like so kind of like wanted to use a CLI to use that on my code base so kind of just developed it overnight not”

Claim: The speaker liked Grok 4 because it combined capabilities resembling those of Opus and Sonnet.

“with Grok4 as well but I think the idea was I think Grok4 actually has some of the really cool agentic capabilities so kind of like I think I really love Claude Sonet for example because of the same and Grok4 had like some of a mix of Opus and Sonet which I really like so kind of like wanted to use a CLI to use that on my code base so kind of just developed it overnight not”

Claim: The speaker built the CLI overnight to use Grok 4 on their codebase.

“with Grok4 as well but I think the idea was I think Grok4 actually has some of the really cool agentic capabilities so kind of like I think I really love Claude Sonet for example because of the same and Grok4 had like some of a mix of Opus and Sonet which I really like so kind of like wanted to use a CLI to use that on my code base so kind of just developed it overnight not”

Claim: The speaker considered Grok 4 to have some particularly appealing agentic capabilities.

“that much I'm a cursor fan so I think”

Claim: The speaker is a fan of Cursor.

“I mean, we are hiring across like a research engineer to work on some really cool RL stuff that we just kind of like to build skills on top of tools that we just discussed.”

Claim: The company is hiring a research engineer to work on reinforcement-learning projects and develop skills built on the tools discussed in the conversation.

“We are hiring for product engineers,”

Claim: The company is hiring product engineers.

“we are hiring for DevRel,”

Claim: The company is hiring for developer relations roles.

“might be interested to work in one of the coolest AI companies as it is and growth person as well yeah any kind of of these roles would love yep”

Claim: The company is also hiring for a growth role.