Latent Space: The AI Engineer Podcast
⏭️ Forward Deployed: Voice AI on what works in 2026
Watch the full show on YouTube: https://www.youtube.com/watch?v=MwNvowwcZOo
25 Aug 2026 · 36:30 · transcript: transcribed · original
177claims extracted
0speakers named
0%of speech attributed
406transcript segments
Claims
In the order they were said
“Yeah, so I started as a product manager at Intuit.”
Claim: The speaker began their career as a product manager at Intuit.
“I was specifically working on credit karma for a couple years.”
Claim: The speaker worked on Credit Karma for a couple of years.
“I worked at a small venture studio after that.”
Claim: After working on Credit Karma, the speaker worked at a small venture studio.
“And then I ended up starting a consulting business called Exoflop Labs,”
Claim: The speaker started a consulting business named Exoflop Labs after working at a venture studio.
“where we were working with like retailers, insurance companies, that sort of thing,”
Claim: Exoflop Labs worked with retailers, insurance companies, and similar clients.
“just building agents essentially.”
Claim: Exoflop Labs built AI agents for its clients.
“This was like two years ago.”
“So it just felt like a natural extension of a lot of the product work that I was doing earlier.”
“It was like, hey, I'm working with customers, they have a specific type of of project that they want built for some use case and I'm going to understand what they want I'm going to make trade-offs and I'm going to help them build that so yeah I just felt like a natural extension of what I was doing and so back in January we started working with a couple private equity firms and a lot of people were just like hey like I'm following on what's going on on Twitter what's going on LinkedIn I don't know what's marketing BS and what's not so I was like why not just bring on people who are working on the cool stuff at cool companies we will”
“of project that they want built for some use case and I'm going to understand what they want I'm going to make trade-offs and I'm going to help them build that so yeah I just felt like a natural extension of what I was doing and so back in January we started working with a couple private equity firms and a lot of people were just like hey like I'm following on what's going on on Twitter what's going on LinkedIn I don't know what's marketing BS and what's not so I was like why not just bring on people who are working on the cool stuff at cool companies we will”
“of project that they want built for some use case and I'm going to understand what they want I'm going to make trade-offs and I'm going to help them build that so yeah I just felt like a natural extension of what I was doing and so back in January we started working with a couple private equity firms and a lot of people were just like hey like I'm following on what's going on on Twitter what's going on LinkedIn I don't know what's marketing BS and what's not so I was like why not just bring on people who are working on the cool stuff at cool companies we will”
Claim: The speaker began working with a couple of private equity firms in January.
“of project that they want built for some use case and I'm going to understand what they want I'm going to make trade-offs and I'm going to help them build that so yeah I just felt like a natural extension of what I was doing and so back in January we started working with a couple private equity firms and a lot of people were just like hey like I'm following on what's going on on Twitter what's going on LinkedIn I don't know what's marketing BS and what's not so I was like why not just bring on people who are working on the cool stuff at cool companies we will We'll talk about what they're working on, like we'll do a deep dive and we'll record it, we'll post it and hopefully people can take learnings away that they can apply to their businesses. And that's why I ended up starting doing a lot of the fireside panels and podcasts that I've been doing since then.”
“We'll talk about what they're working on, like we'll do a deep dive and we'll record it, we'll post it and hopefully people can take learnings away that they can apply to their businesses. And that's why I ended up starting doing a lot of the fireside panels and podcasts that I've been doing since then.”
“Yeah, so our very first episode was on the future of agentic engineering.”
Claim: The speaker's first podcast episode covered the future of agentic engineering.
“So we brought on companies like Factory Cognition, Composio,”
Claim: The speaker's first episode included companies named Factory, Cognition, and Composio.
“Soundgrap. We've done panels on voice agents, which is actually the one that we're going to be showing here. We've done some on computer use agents. We've done agents in the enterprise.”
Claim: The speaker's podcast has covered voice agents, computer-use agents, and enterprise agents.
“So we've done a ton of stuff just to get nitty gritty into the details of a lot of this stuff.”
Claim: The speaker's podcast aims to examine AI-agent topics in detailed, practical depth.
“Yeah, so we brought on some people from Decagon, Vappi, Retail Daily, and a company called Smallest AI.”
“Basically, we just talked about like, what is the state of the art in building voice agents?”
Claim: The speaker's discussion examined the state of the art in building voice agents.
“Because that's a very hot use case.”
Claim: The speaker characterized voice agents as a very hot use case.
“I think Sierra even talked about this is this is like one of the most competitive. markets in AI right now is like building voice agents.”
“And so I thought it would be useful to talk about how they're actually built and some of the things that, you know, engineers who are building in this space still have to contend with.”
“So for example, we talked about how like the state of the art right now in building voice agents is a cascaded pipeline. So we talked about how it's like a three-step process.”
“You go like speech to text, LLM, then text to speech.”
“And we talked about like, why is that versus why don't we just have a voice to voice model? Like, why don't we use that? that like the reality is those just are not super reliable right now we also talk about like the trade-offs that you have to make when you're building voice agents so you have to trade off like the intelligence of responses that you're getting versus the latency so you can get very intelligent responses but you're also going to trade off it's going to be a slower response and so for different use cases that might be good or bad we also talk about like reliability so there are some llm companies that may not be like super reliable in terms of like they're”
“that like the reality is those just are not super reliable right now we also talk about like the trade-offs that you have to make when you're building voice agents so you have to trade off like the intelligence of responses that you're getting versus the latency so you can get very intelligent responses but you're also going to trade off it's going to be a slower response and so for different use cases that might be good or bad we also talk about like reliability so there are some llm companies that may not be like super reliable in terms of like they're”
Claim: The speaker said some LLM companies may have unreliable infrastructure.
“that like the reality is those just are not super reliable right now we also talk about like the trade-offs that you have to make when you're building voice agents so you have to trade off like the intelligence of responses that you're getting versus the latency so you can get very intelligent responses but you're also going to trade off it's going to be a slower response and so for different use cases that might be good or bad we also talk about like reliability so there are some llm companies that may not be like super reliable in terms of like they're”
Claim: In voice-agent design, increasing response intelligence can come at the cost of higher latency.
“that like the reality is those just are not super reliable right now we also talk about like the trade-offs that you have to make when you're building voice agents so you have to trade off like the intelligence of responses that you're getting versus the latency so you can get very intelligent responses but you're also going to trade off it's going to be a slower response and so for different use cases that might be good or bad we also talk about like reliability so there are some llm companies that may not be like super reliable in terms of like they're”
“infrastructure and so like what do you do when opus goes down like you need to have a waterfall of models to basically pick it up so that your voice agents don't just stop working we also talk about like turn taking and how like it seems like a trivial problem where hey like you're talking to a voice agent and”
“infrastructure and so like what do you do when opus goes down like you need to have a waterfall of models to basically pick it up so that your voice agents don't just stop working we also talk about like turn taking and how like it seems like a trivial problem where hey like you're talking to a voice agent and let's say i pause so how does the voice agent know that hey like i should i should interject and i should actually like respond to you versus oh the person is just taking time to think so turn taking is actually like not a trivial problem to solve and so we talk a little bit about that and also we even talk about like how um i think pipette built a voice agent for the world's fair right so we even talked about that a little”
“many people actually called probably you know people were just kicking the tires uh you know like it's not it's not that serious like we it's a conference people know what it is but you know i think that's a good deployment of use case daily is a sponsor so why not i you know i i would say that vast majority of voice stuff is support”
Claim: Most voice AI deployments are used for customer support.
“support and like yeah oh my god there's you know so many call centers out there hundreds of billions of dollars spent on this stuff and they're all you know bad so hopefully we can sort of raise the the state of the art which you know i think that's why decadon and all these things people is like to me you know it's not as interesting if a bunch of people who are obviously selling you voice pipelines telling you that voice pipelines are the state of the art like yeah duh but you know having”
“support and like yeah oh my god there's you know so many call centers out there hundreds of billions of dollars spent on this stuff and they're all you know bad so hopefully we can sort of raise the the state of the art which you know i think that's why decadon and all these things people is like to me you know it's not as interesting if a bunch of people who are obviously selling you voice pipelines telling you that voice pipelines are the state of the art like yeah duh but you know having”
Claim: The speaker hopes voice AI will improve the state of the art in customer support.
“having the people actually focus on customer support and all those things also basically concludes like yeah like the models are not there yet they may never be and actually this is just the way that you got to do it and the people that really feel it are not the researchers because researchers just always want bigger models to solve everything and the product engineers will try to do it but the fdes the people actually dealing with customers will be like dude they made this like horrible mistake because can never happen again how can you guarantee me that”
Claim: The speaker characterizes researchers as tending to seek larger models to solve all problems.
“having the people actually focus on customer support and all those things also basically concludes like yeah like the models are not there yet they may never be and actually this is just the way that you got to do it and the people that really feel it are not the researchers because researchers just always want bigger models to solve everything and the product engineers will try to do it but the fdes the people actually dealing with customers will be like dude they made this like horrible mistake because can never happen again how can you guarantee me that”
“having the people actually focus on customer support and all those things also basically concludes like yeah like the models are not there yet they may never be and actually this is just the way that you got to do it and the people that really feel it are not the researchers because researchers just always want bigger models to solve everything and the product engineers will try to do it but the fdes the people actually dealing with customers will be like dude they made this like horrible mistake because can never happen again how can you guarantee me that”
“yeah yeah yeah so like actually we even talked about that a little bit so like we talked about like the inbound versus outbound use cases because we even did that once we like ran an outbound use case and like what you find is whenever somebody like picks up the phone and you do an outbound use case a lot of the times they just hang up as soon as they realize it's a bot so we even talked about that like how do you how do you solve for that”
“is typically voice input, it comes through a transport, it could be WebRTC, phone call, or WebSockets, it goes into speech-to-text, the transcription happens,”
“there can be some additional models right there for background noise removal, voice isolation.”
Claim: A cascade voice system may use additional models to remove background noise and isolate voices.
“since there are few people in in the foreground it will isolate to one person remove the background noise that there's a turn detection model unlike a text where you know when you're done you press enter and the message goes and the LLM starts to like execute with voice there is no it's not walkie-talkie or push-to-talk so there is no additional uh information apart from like if you were looking at the face you could actually figure out that the person stopped speaking but typically use something like a voice activity detection and a smart turn model to like figure out that the turn is complete um if i pause mid mid sentence that's different from like if i finish speaking and pause”
“Then when you have the text from the transcription, then you send it to an LLM. The LLM may respond with an inference. It may actually tell you it may infer to tool calls. It may do a bunch of things there.”
“You get that. When you have the full output, you text to speech, TDS, and it's usually faster than real time. So then you have to like figure out how to like stream it back to the end user. It'll go over the same patch transport that you spoke over and then it'll play it back on your speakers.”
“If you're doing an outbound use case, like you do an outbound call. That collection is quite commonly a use case for this. You get a call, they tell you you have like an unpaid bill, they ask you if you'll pay it or give you a certain set of options.”
“For that, the bot has really like if you go off the rails, like the human goes and start saying random stuff, it can just drop the call. It's not obligated to stay on the call in any way.”
“So guardrails are easy. It knows what the inputs it can accept. It knows what inputs it doesn't need to respond to.”
“It flips when you have an inbound use case because the bot has no context, like has some context of why it exists, but doesn't know why you, the person who's calling it.”
“if you're a mom and pop shop like a flower shop or or like a brace or something like that okay there's only like finite things that you can do there but if you are more complex let's say you're amazon right you're like a billion products on your platform um and Do you have different policies for different things?”
“Do you have different policies for different things? Is it a physical item? Is it a cheap physical item? Is it an expensive item? Is it an electronic item? Like all of that.”
“So it needs to pick.”
“And then when you said something like, you can't put all of that inside a single prompt because it will like LLMs are. you know goofy in that sense that always remember the first four percent and the last four percent and everything in between they kind of forget so if you have a return policy right in the middle of that context it's very likely it'll hallucinate so you have to do more smart things and that's where you have like more models come in you can have a compaction model like we're all using coding agents now right so you know like”
“you know goofy in that sense that always remember the first four percent and the last four percent and everything in between they kind of forget so if you have a return policy right in the middle of that context it's very likely it'll hallucinate so you have to do more smart things and that's where you have like more models come in you can have a compaction model like we're all using coding agents now right so you know like”
“If you're using like a 1 million context model, like at 25%, you should become nervous at that 25% mark. It's nowhere near 70% mark,”
“but at 25% mark, you're like, okay, should I start compacting and like saving my work so that this thing does not go off the rails?”
“I think YCR has been like one generation ahead of coding agents like we've all in the last two years solved things that felt like so alien but very tangible for us which now when we see coding agents you're like man compaction we we were doing compaction from day one like we knew like after five tons or ten tons when your context was only 250 tokens or 250,000 tokens or 50,000 tokens”
“I think YCR has been like one generation ahead of coding agents like we've all in the last two years solved things that felt like so alien but very tangible for us which now when we see coding agents you're like man compaction we we were doing compaction from day one like we knew like after five tons or ten tons when your context was only 250 tokens or 250,000 tokens or 50,000 tokens”
“you know models were really small like before and at 10 000 tokens they would like go off the rails so it's a lot of our work is i think about like making sure that the bot does not hallucinate and trying to like keep track of the conversation as the the turns progress and the turns are fairly short because”
“you know models were really small like before and at 10 000 tokens they would like go off the rails so it's a lot of our work is i think about like making sure that the bot does not hallucinate and trying to like keep track of the conversation as the the turns progress and the turns are fairly short because”
Claim: The speaker says earlier, smaller models could go off track at around 10,000 tokens of context.
“the either way the bot is more likely to speak more than the human did so you have to like kind of keep track of like what did the human say where are we in the conversation”
“Yes, that is one of the major offerings that we have cascading in speech to speech.”
Claim: The speaker says cascading is one of their company's major offerings for voice agents.
“And I feel like the easiest way to answer this is you can call into a really nice voice-to-voice demo and you're like, wow, it's like listening to me laugh and it's like responding to my tone and it's so snappy, it's so fast.”
Claim: A voice-to-voice demo can respond to a user's laughter and tone while feeling very fast.
“And I feel like the easiest way to answer this is you can call into a really nice voice-to-voice demo and you're like, wow, it's like listening to me laugh and it's like responding to my tone and it's so snappy, it's so fast.”
Claim: A voice-to-voice demo can respond to a user's tone and laughter while feeling very fast.
“But then, you know, I tell it that it asked me what day I want to schedule my appointment for and I say, you know, next week and it says, great, is that June 10th? And I'm like, no, it's the year's 2030. And it's like, you're right. It is 2030. So let's schedule this for June 10th, 2030.”
“But then, you know, I tell it that it asked me what day I want to schedule my appointment for and I say, you know, next week and it says, great, is that June 10th? And I'm like, no, it's the year's 2030. And it's like, you're right. It is 2030. So let's schedule this for June 10th, 2030. And I'm like, great, sounds good.”
“And so from that perspective, I am actually very curious, especially on the smallest side, how that's evolved over time, because we are very much keeping our eye on how these speech to speech models are performing.”
“And so from that perspective, I am actually very curious, especially on the smallest side, how that's evolved over time, because we are very much keeping our eye on how these speech to speech models are performing.”
“But overall, I think our current stance is the cascading model just allows you to enforce so many more rigid guardrails and just tight control over the ability to say,”
“But overall, I think our current stance is the cascading model just allows you to enforce so many more rigid guardrails and just tight control over the ability to say,”
“hey, this input is going to go through the same supervisor models to detect for, you know, prompt injection or social engineering, then it's going to go down the conveyor belt into intent selection,”
“hey, this input is going to go through the same supervisor models to detect for, you know, prompt injection or social engineering, then it's going to go down the conveyor belt into intent selection,”
“then we're going to optimize context by By checking conditions ahead of time to say, you know, in this complex process that we're going through with this user, maybe this prompt is applicable if they're this type of customer, but this prompt is applicable if they're that type of customer.”
“then we're going to optimize context by By checking conditions ahead of time to say, you know, in this complex process that we're going through with this user, maybe this prompt is applicable if they're this type of customer, but this prompt is applicable if they're that type of customer. So let's, you know, figure all this stuff out ahead of time, compact and compile a good system prompt for our message generation model and get a response back.”
“So let's, you know, figure all this stuff out ahead of time, compact and compile a good system prompt for our message generation model and get a response back. And then we can take that response and we can check a whole bunch of other things, right? We can check, is this grounded in truth? Is it 2030 or is it 2020, whatever year it is, 2026?”
“And then we can take that response and we can check a whole bunch of other things, right? We can check, is this grounded in truth? Is it 2030 or is it 2020, whatever year it is, 2026? And then it goes back out over the line, right?”
“And so the obvious constraint to that is how do you make that performant, right? How do you parallelize as many of those steps in the conveyor belt as possible?”
“And so the obvious constraint to that is how do you make that performant, right? How do you parallelize as many of those steps in the conveyor belt as possible?”
“I think the last six or so months has been a really amazing feat from our engineering team at least to find and shave off like 10 milliseconds at a time across every single part of this pipeline to make it feel snappy,”
“I think the last six or so months has been a really amazing feat from our engineering team at least to find and shave off like 10 milliseconds at a time across every single part of this pipeline to make it feel snappy,”
“even though there's a lot of things going on behind the scenes that. you know, you don't necessarily have to do with a speech-to-speech model.”
“even though there's a lot of things going on behind the scenes that. you know, you don't necessarily have to do with a speech-to-speech model.”
“Like let's talk about like why did people start thinking about speech to speech. And so the initial idea was simple that hey if you convert speech to text it's going to lose the emotional information, So if you say hey you might be saying it in a sad way or an excited way, the bot is going to answer in the same manner, right? So that was the obvious reason people started.”
“I think at some point of time, at least at smallest, like that has evolved to speech to speeches. A more natural way in which the human brain operates,”
“so for example, when you do the cascaded thing, you do speech to text, then you send the prompt to an LLM and then it responds, So we call that a synchronous architecture, like it's happening one after the other.”
“But our brain is thinking while listening. So as I'm speaking to you, you're already forming your thoughts and if I'm talking for too long, you'll interrupt. dropped me right or you might be taking notes in the back end right so you might be essentially doing tool calls while I'm speaking to you.”
“And so the whole idea is that if you ever want to pass the Turing test of how the human brain operates, you need something that is working asynchronously and cannot just understand emotions, but also operate like take in speech natively and give out speech natively asynchronously.”
“And so that's why we've been sort of building Hydra. Now, Hydra is our speech to speech model.”
“in terms of accuracy and interpretability and all those things, I think whenever there is like a new architecture that comes out, it's often sort of good in one parameter and then”
“regressed a little bit in other parameters right like so for example the speech-to-text accuracies for just a speech-to-text model might be way better than the encoder of a speech-to-speech model that you have and so I think the challenge is that while you made progress on the you know making it more natural and operate more human-like etc etc how could you keep the accuracy bar the same.”
“regressed a little bit in other parameters right like so for example the speech-to-text accuracies for just a speech-to-text model might be way better than the encoder of a speech-to-speech model that you have and so I think the challenge is that while you made progress on the you know making it more natural and operate more human-like etc etc how could you keep the accuracy bar the same.”
“And so I think a lot of that comes down to interpretability of such models like because you don't want it to be a black box so how can you understand where it is lacking and so that that's a lot of research that we do and how to make speech-to-speech models more interpretable the other constraint is I think initially the speech-to-speech models like I think sesame etc that came in that they were just speech-to-speech like they literally took in speech and gave out speech ours is like multi-model so”
“interpretability of such models like because you don't want it to be a black box so how can you understand where it is lacking and so that that's a lot of research that we do and how to make speech-to-speech models more interpretable the other constraint is I think initially the speech-to-speech models like I think sesame etc that came in that they were just speech-to-speech like they literally took in speech and gave out speech ours is like multi-model so”
“Text and speech and text gives out speech and text both so it can do tool call and take in text parallely. So if you want to put guardrails, if you want to do all those things, that constraint does not go away.”
“Text and speech and text gives out speech and text both so it can do tool call and take in text parallely. So if you want to put guardrails, if you want to do all those things, that constraint does not go away.”
Claim: The speaker says that multimodal input and output do not remove the need for guardrails.
“What we are also seeing is there is still at least in enterprises a lot more cascaded deployments compared to speech to speech model,”
“but speech to speech I think will be like an eventual future. That's my opinion.”
Claim: The speaker believes speech-to-speech will eventually become the future approach.
“So my take is that when you have like competing things you end up in a hybrid and I think the answer for the midterm will be some form of hybrid because the speech-to-speech are improving there are parts of the conversation loop where you would say like oh my use case of my workflow is complex I'm going to use multiple LLMs anyway so for the active loop like I'm talking to to do as a human you called me I'm answering your questions but whenever I have to do some kind of lookup or something I delegate to another LLM which will then do the cascade stuff so the wise in the first part of the loop keeps running and then you have like interesting kind of semaphores or like you can think of these as threads you want to interrupt the speech to speech model because you realize that this is a complex question and you want to Before the speech-to-speech response you say actually you cannot answer this question right and delegate to the cascade and so on so forth.”
“So my take is that when you have like competing things you end up in a hybrid and I think the answer for the midterm will be some form of hybrid because the speech-to-speech are improving there are parts of the conversation loop where you would say like oh my use case of my workflow is complex I'm going to use multiple LLMs anyway so for the active loop like I'm talking to”
“So typically I kind of say there are many use cases where you know if you wake me up in the middle of the night and ask me a bunch of questions there are definitely some class of questions that I can answer without thinking and we all do our jobs in a certain way where we can be an autopilot like 50% of our time.”
“right so as these use cases become emergent and you know you're fully deployed with the customer doing high volume you could essentially train specialized models that fully understand like that 50% use case very well and you could you could always ask the model like hey what is my return policy and I'll say like”
“this is metered on policy and it applies to like 70%. And I know which cues or products it belongs to or it applies to.”
“And if you ask me outside of that, then I have to like do all the crazy work.”
Claim: In the speaker’s example, questions outside the policy’s coverage require more extensive work.
“And if I don't, then I answer it, right?”
“If you're done of thinking then you can actually use any of the like if you want a fast model which responds to like the prompt and does not need to think because if you want it to think you actually start to think about paralyzing the work because you want the first model to be super fast.”
“I still like my Gemini 2.5 really well.”
Claim: The speaker expresses a favorable opinion of Gemini 2.5.
“assets like the 3.5 which they launch is like okay it's slower but the 2.5 is so good uh even the haiku is really really good uh they're still slower than what you would expect but yeah”
“You know, you can send the speech, like in the speech-to-speech model case, you would send the speech directly to one place, fork it into another place, run the std there.”
“And if you want multiple models, the text output from the first forks into multiple LLMs.”
Claim: A multi-model pipeline can send the first model’s text output to multiple LLMs in parallel.
“so at least for speech to speech right now, we are just fully focused on making it more intelligent in English and, you know, getting it really good in English like we don't want to do any other, we don't want to introduce any other variables because the technology in itself is, I would say, quite frontier,”
“like a lot of demand in so we have a lot of presence in India so we see like a lot of demand from India we see a lot of demand from Latin America etc I think US is more like mostly English and then there's some Spanish and then there's a lot of accents to it and I think yeah US at least we have not had any troubles in terms of”
Claim: The speaker's team reports substantial demand for its offering from India and Latin America.
“like a lot of demand in so we have a lot of presence in India so we see like a lot of demand from India we see a lot of demand from Latin America etc I think US is more like mostly English and then there's some Spanish and then there's a lot of accents to it and I think yeah US at least we have not had any troubles in terms of I mean, noise cancellation is probably the last mile of problem that is pending in terms of handling those things. And but yeah, that has nothing to do with accents. I'm curious what you guys have seen.”
“Maybe I can add some examples about that because on multilingual, I like to definitely leverage Cascade model because there are multiple levers you can pull.”
“it's a customer we were deploying for Japan. So you could typically go to your 4.1 and open AI or your deep gram for transcriber and that typically worked well. Now, the challenge became.”
“became on the voice piece so it turns out what we thought it was a state-of-the-art for something like English Spanish Portuguese Those type of, that are typical in the US didn't work at all.”
“So one of the benefits about this is that you can swap certain pieces that doesn't quite work for you.”
“At least in VAPI how we solve it is that we let customers bring their custom text-to-speech server.”
Claim: Vapi allows customers to integrate a custom text-to-speech server of their choice.
“It's also really hard to get it right in pronunciation of brands, addresses, and that really we cannot build expertise and optimize for every single use case.”
“Yeah, I think so they are building hardware is what I understand like they are putting those voice into some sort of a hardware. I'm not sure if it's glasses or what are they working on exactly but”
“I know few people who got hired there and they're all sort of focused on the hardware and voice sort of backgrounds”
“So I think it highly depends upon the use case and also the size of your prompt,”
Claim: The best prompt-design approach depends heavily on the use case and the size of the prompt.
“So I think it highly depends upon the use case and also the size of your prompt,”
“I have a counter approach to Varun in terms of that, like usually in inbound, I find, you know. You you're usually having dedicated lines and you know specifically where the workflows are going to go. Right. And so in these cases, if you know a little bit more about and you can predict where the conversation is going to go, then you can have a more of a node or graph builder kind of approach.”
“You you're usually having dedicated lines and you know specifically where the workflows are going to go. Right.”
Claim: Inbound workflows usually use dedicated lines with a known workflow path.
“And so in these cases, if you know a little bit more about and you can predict where the conversation is going to go, then you can have a more of a node or graph builder kind of approach.”
“But if you don't know what's going to happen, typically I think in outbound, yes, there is a dedicated like message, but what the caller says back. back you never know because they could just be frustrated they could be angry or they could be i don't know like annoyed and in that case when they're erratic when a person's erratic you never know so having it in a way that is like all in one one prompt allows you to have an or more like a central brain in terms of i can pick up components from my prompt that may not have been in a dedicated flow but”
“but what the caller says back. back you never know because they could just be frustrated they could be angry or they could be i don't know like annoyed and in that case when they're erratic when a person's erratic you never know so having it in a way that is like all in one one prompt allows you to have an or more like a central brain in terms of i can pick up components from my prompt that may not have been in a dedicated flow but”
“back you never know because they could just be frustrated they could be angry or they could be i don't know like annoyed and in that case when they're erratic when a person's erratic you never know so having it in a way that is like all in one one prompt allows you to have an or more like a central brain in terms of i can pick up components from my prompt that may not have been in a dedicated flow but actually can be utilized to enhance the response and then on top of that you have to think of your knowledge base and knowledge the things that you're at your expense right like in a inbound flow maybe you know exactly when you need to pull a specific part of your knowledge so you don't need to extract every single”
“actually can be utilized to enhance the response and then on top of that you have to think of your knowledge base and knowledge the things that you're at your expense right like in a inbound flow maybe you know exactly when you need to pull a specific part of your knowledge so you don't need to extract every single piece of context,”
“actually can be utilized to enhance the response and then on top of that you have to think of your knowledge base and knowledge the things that you're at your expense right like in a inbound flow maybe you know exactly when you need to pull a specific part of your knowledge so you don't need to extract every single piece of context,”
“whether it's your websites and documents, but sometimes in outbound, you should utilize all that information to get a better response and you will rely more on cosine similarity and stuff to make sure that hopefully your retrieval is good enough to respond to them.”
“but sometimes in outbound, you should utilize all that information to get a better response and you will rely more on cosine similarity and stuff to make sure that hopefully your retrieval is good enough to respond to them.”
“the limiting factor has been and will continue to be, although it's getting a lot better over time, the ability for these frontier LLMs to be able to take a massive wall of text and not just read the first three lines and the last three lines, but actually be able to properly reason and like localize themselves into, We are halfway through this procedure and these things have already happened and these things haven't happened yet and these things are true and these things are false about the situation and so therefore like this is the one thing that I should be really zooming in on right now, right?”
“I think we kind of take an approach that depends on the customer, but we do within our execution engine have the ability to make some of these rules or prompts or guidelines more situational.”
“situational so that we can you know check if these are relevant right now before we even include them or don't include them in the system prompt and so that was our solution that was especially relevant you know like seven months ago when the LLMs really were just like consistently skipping step for A1 but we have found over time now that”
“situational so that we can you know check if these are relevant right now before we even include them or don't include them in the system prompt and so that was our solution that was especially relevant you know like seven months ago when the LLMs really were just like consistently skipping step for A1 but we have found over time now that”
“And sometimes the reliability of deciding when these things are relevant or not is actually worse than just giving all of it to the model because the models have improved a lot since that time.”
“And so now we're going back, I think, swinging a little bit back towards just give the model everything and more or less it will figure it out.”
“But at the end of the day, you know, with non-deterministic systems, it really just ends up being customer by customer.”
Claim: The best approach to non-deterministic systems must be determined customer by customer.
“We have to really understand their use cases and at the end. At the end of the day, just build it and then test it a whole bunch of times and then use LLM as a judge to tell us, you know, did it work? Was it reliable or do we need to add a little bit more preprocessing and context optimization ahead of time in order to make it reliable?”
“So the real answer is most likely always just test and find out and then test again and then find out again and, you know, goes on forever.”
“I mean, I think the definition of what like benchmarking latency stats look like that are good have changed a lot and expectations are, you know, getting better and better and better from our customers.”
“I mean, I think the definition of what like benchmarking latency stats look like that are good have changed a lot and expectations are, you know, getting better and better and better from our customers.”
“it is normal for in normal conversation for humans to say a few words while they're thinking.”
Claim: People commonly say a few words while thinking during ordinary conversation.
“it is normal for in normal conversation for humans to say a few words while they're thinking.”
Claim: In ordinary conversation, humans commonly say a few words while thinking.
“And so we've actually found that when we have zero contextual fillers, sometimes it actually feels more rigid than when we don't.”
“And so we've actually found that when we have zero contextual fillers, sometimes it actually feels more rigid than when we don't.”
“And so we tend to find that having the right amount of fillers that kick in when, you know, we are experiencing some sort of latency because of some very large prompt or some very complex thing that the model is thinking through,”
“And so we tend to find that having the right amount of fillers that kick in when, you know, we are experiencing some sort of latency because of some very large prompt or some very complex thing that the model is thinking through, most commonly, you know, running a tool call where our customers API becomes the bottleneck and we're waiting, you know, five seconds for something to come back.”
“most commonly, you know, running a tool call where our customers API becomes the bottleneck and we're waiting, you know, five seconds for something to come back.”
“Being able to say, hey, just give me a sec looking at that. Okay, cool. Here's here's your answer, right? There's actually nothing wrong with that when we've tested with our customers.”
“Being able to say, hey, just give me a sec looking at that. Okay, cool. Here's here's your answer, right? There's actually nothing wrong with that when we've tested with our customers.”
“the limiting factor in that case then becomes how good are your fillers and how natural do they sound?”
“the limiting factor in that case then becomes how good are your fillers and how natural do they sound?”
“And so that's been a place where we've invested a lot of time as well to say, how do we create an execution engine where we really don't need fillers, but in those moments where we do, they're really good.”
“And so that's been a place where we've invested a lot of time as well to say, how do we create an execution engine where we really don't need fillers, but in those moments where we do, they're really good.”
“latency and how to find that trade-off, because in my opinion, I honestly think that that's the hardest problem to solve in voice deployments is just that trade-off that's always going to be there of like performance versus stability and reliability in general.”
“latency and how to find that trade-off, because in my opinion, I honestly think that that's the hardest problem to solve in voice deployments is just that trade-off that's always going to be there of like performance versus stability and reliability in general.”
“Yeah, I agree with all of that. Some of the ways that we think about it in VAPI is offloading some of that instruction. that you know 10-15 step workflow in the in the prompt one example I can talk about is this collections use case we do allow users to capture businesses to capture credit card information but not every call is about that some people already have a payment method maybe added or that so one way of doing that offloading into a specialized agents so we can start thinking about multi-agents and there are many architecture and patterns in there it constrains the instructions that you give it to it to that particular moment of the conversation and once that achieves its goal it can go back to that more free-form approach what has loaded like your longer's context something we're also being experimenting lately”
“specialized agents so we can start thinking about multi-agents and there are many architecture and patterns in there it constrains the instructions that you give it to it to that particular moment of the conversation and once that achieves its goal it can go back to that more free-form approach what has loaded like your longer's context something we're also being experimenting lately”
“specialized agents so we can start thinking about multi-agents and there are many architecture and patterns in there it constrains the instructions that you give it to it to that particular moment of the conversation and once that achieves its goal it can go back to that more free-form approach what has loaded like your longer's context something we're also being experimenting lately”
Claim: VAPI has recently been experimenting with the described multi-agent approach.
“To your point where you can have, think about it as a waterfall, you can actually stream to multiple places. So what if you have streaming down to a small language model, which can do inference in a very little time.”
“So think about classifiers to maybe collect intent while the pipeline is still maybe adding a filler word as well to keep a consistent experience.”
“by kicking off a background process that is still intelligent, still thinking. So those are still things that we'd like to invest a little bit more research on. constantly investing in that.”
“I think one thing that we haven't also factored in in terms of the latency trade-off is cost,”
Claim: Cost is another factor in the trade-off involved in designing for latency, according to the speaker.
“Now that you're putting such a big, large prompt into your LLM, you have to think about those calls where many people just hang up and predominantly calls just hang up 10 seconds into the call, Now you're having to pay for all those tokens to just get inputted in.”
“So like Stephen mentioned, splitting out your prompt, you are also playing to that strength of lower latency and lower cost.”
Claim: Splitting a large prompt into smaller parts can take advantage of lower latency and lower cost.
“So the idea there is that all our tooling is open source.”
Claim: VAPI's benchmarking tools are open source, according to the speaker.
“So with the LTT benchmarks, the TTS benchmarks and the LLM benchmarks, all the all the turns are actually open. So you can run it against any new model that comes out. You can run it locally,”
Claim: VAPI's benchmark turns are open, allowing users to run the benchmarks against new models or locally.
“you can look at the outputs from our graphs and compare that to what you're getting on your infrastructure if you're self-hosting or elsewhere.”
“So that's one important aspect to like consider if you know what your turns will look like you can just update or you know just download our benchmarks update the turns and see how that like goes for your own flows.”
Claim: Users can update the turns in VAPI's benchmarks to match their own flows and evaluate those flows.
“I think evals are really important.”
Claim: The speaker considers evaluations very important.
“And if you have like a like well talking to customers, if you can evaluate like what type of conversations or like what type of workflow they're going to have, like those are real direct inputs into like what you can directly start testing. even though you're not live with them or even they've not shown any intent so I think that's where like the SEs and the forward applied engineers can actually come to ford like you we all have cloud running in the background right so you can just say here is what we had a discussion like can you build an eval like based on our eval suit like can you like just run a smoke test with the conversation information that we already have”
“Yeah, just just adding to that, like we are seeing so we while we focus on models, we also have all the models sort of orchestrated on our platform. So you could build a voice agent on smallest.”
“We are seeing a lot of customers who just pair our models with our own like small language model it's called electron we fine-tune one of those to make it work for real-time voice use cases and see a lot of folks using that with better knowledge basis memory etc over gpt4 or 4.1 you know the popular real-time models just a from a cost perspective it's lower b from latency perspective it's like way lower”
“pair our models with our own like small language model it's called electron we fine-tune one of those to make it work for real-time voice use cases and see a lot of folks using that with better knowledge basis memory etc over gpt4 or 4.1 you know the popular real-time models just a from a cost perspective it's lower b from latency perspective it's like way lower”
“pair our models with our own like small language model it's called electron we fine-tune one of those to make it work for real-time voice use cases and see a lot of folks using that with better knowledge basis memory etc over gpt4 or 4.1 you know the popular real-time models just a from a cost perspective it's lower b from latency perspective it's like way lower”
“and then just more reliable I think I'm not sure if everyone faces this but like open AI API spike and then you have no control on those latencies with if you sort of have a self-hosted model you could scale up scale down based on your requirement so that's one thing we are seeing a lot with our customers.”
“and then just more reliable I think I'm not sure if everyone faces this but like open AI API spike and then you have no control on those latencies with if you sort of have a self-hosted model you could scale up scale down based on your requirement so that's one thing we are seeing a lot with our customers.”
“and then just more reliable I think I'm not sure if everyone faces this but like open AI API spike and then you have no control on those latencies with if you sort of have a self-hosted model you could scale up scale down based on your requirement so that's one thing we are seeing a lot with our customers.”
Claim: A self-hosted model can be scaled up or down to match requirements.