A commercial partner validated Handshake AI’s evaluation set but said it did not help identify ways to improve the model’s compliance with the partner’s safety policy.

“Um, it's a much more holistic look at it, and we actually got this validated by one of our, our commercial partners. We shared the set with them, and their feedback wasn't that it helped them identify how to make their model better comply with their safety policy.”