The speaker's team finds that appropriately timed fillers can make latency more manageable when a large prompt, complex model task, or customer API tool call causes a delay, including waits of about five seconds.

“And so we tend to find that having the right amount of fillers that kick in when, you know, we are experiencing some sort of latency because of some very large prompt or some very complex thing that the model is thinking through,”

“most commonly, you know, running a tool call where our customers API becomes the bottleneck and we're waiting,”

“you know, five seconds for something to come back.”