Evaluation needs vary by situation, but the speaker's general advice is to build a larger rather than smaller evaluation suite.

“There's a lot of different types of evals, right? Like, not all evals are made equal for different, you know, like, situations that you're in and things that you're trying to test for.”

“You obviously have just, like, static deterministic stuff, and then you have LLM as a judge. You have all these different strategies.”

“But I guess I would just say, like, what we found from most people is, like, you probably need more rather than less.”