Anthropic minimizes the time between preparing a model for testing and customer access, choosing customers able to run evaluations and provide rapid, actionable feedback.

“Like, I do think a lot of this is vibes-based, and I think a lot of the conversations tend to be kind of like vibes-based and, and on the fly, and part of that is because we're, like, racing, right?”

“It's like we wanna minimize as much as possible the window between we have a version of the model that we've cut for testing and we give everyone access to that model, and so it is just like this really tight window, but the customers we tend to do it with are the ones who, like, we know they're gonna go like kick off their whole eval suite and be able to come back to us with like really actionable feedback as quickly as possible alongside the vibes, um- Yeah, yeah...”

“the vibes-based feedback. So, um, of course there's like definitely like processy things that we could do to make these things better over time, but I mean, you'd be surprised.”