Anthropic's model-launch process combines safety testing and evaluation with internal testing and customer feedback, including early access, before release.

“Uh- Minus the safety stuff, which I think is, is the more complicated part. But outside of those pieces, there is a lot of getting feedback from our customers.”

“So, you know, we make the model, there's a lot of safety testing. Um, there's also evaluation of where we think it needs to be.”

“We make the required trade-offs and decisions as necessary, and then we have to go through testing. Uh, we'll cut various versions of the model, we'll evaluate it.”

“We'll do a lot of internal, um, ant fooding, uh, is- which is what I call it, and that actually always really helps.”

“Like, we catch like really early things like, uh, this is, you know, maybe not the best, and we obviously use Claude a lot for our work internally, so we're able to actually see early signals around stuff.”

“Um, and it helps us detect like, you know, which, which variants might be better than others.”

“And then when we feel really confident with like a certain set of, of variants, uh, we will cut a version for our, for our customers.”

“Um, sometimes that will go through our first-party products, uh, like through Cloud Code or Cowork or things like that.”

“Um, in other cases it will go through our API customers, um, and we'll engage in a kind of like a, a early access program effectively with those customers.”

“They'll need to make changes into their harness, and we've engaged with, uh, a lot of customers like, like Cognition, like, like Hers, like these folks, and we have a bit of a process on how we handle that, um, so that we all kind of get...”

“They always get the feedback that we, we need. Uh, does it perform well in your evals? Do you like it? What don't you like? Some things are, are vibey, and we like to hear those things too.”

“Um, and some things are just like straight up evals, like, did it do well on the things that you care about? Yeah.”

“And we take that all together and once we've all feel good about like all those different pieces, we launch the model. The state of model feedback is- feels very unstructured still. Mm-hmm.”