The host said agent evaluations assess tool calling and effective context use rather than only testing knowledge.

“And instead of like testing knowledge, they're actually, you know, doing tool calling and”

“utilizing context very well that kind of stuff so yeah I think one of those things where there's one of those things where I think there's a lot of fresh area and potential for progress if you can do a credible eval and the other thing I like to see is an eval that has a cost awareness component because I think you know people don't really”