Benchmarks enable teams to measure whether changes to prompts, models, or agent loops improve user experience.

“And then second is, how do you own that intelligence instead of that all that data like going out, so that you can control it? And a big part of what machine learning, uh, what has driven progress in machine learning is codifying those things into benchmark, so that a large group of people can look at a number and say, "Everything that we are doing, we made an update to the prompt.”

“We swapped one model with another model. Instead of, uh, one loop of agent, we did two loops of agent." How did it all like actually improve the user experience?”

“That has to be codified into a way that people can look at some numbers and say, "Okay, this, uh, quantitatively helped move the, you know, improved user satisfaction," or and so on and so forth.”