The team concluded that its financial evaluations were saturated and that errors in answer keys, prompt specifications, or other evaluation flaws explained the questions the model missed.

“Uh, we, like, the-- Basically, we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, um, poorly specified prompts, or just some other flaw with the eval.”