The speaker says many agentic products lack robust evaluations, with some teams relying on user feedback until they can formalize evaluation practices.

“I mean, you probably wouldn't be surprised, but I think some people would be surprised about how few, you know, like products that exist in the world that are agentic that, like, don't have good evals [laughs] to be able- Oh, we had-...”

“to help test these models. We had Boris on the pod two years ago saying we don't have evals for Cloud Code. Yeah. [laughs] I mean, that's changed, but- That is true. [laughs] That's- that's changed.”

“But you know, like, uh, uh, yeah, a lot of things like, you know, your evals are your users, and you don't have time to formalize them, or you don't have the experience or whatever- Yeah... and that's fine.”

“At some point when you get serious, you- you'll have evals. Yeah, totally. Exactly.”