On this benchmark, the three largest cross-model weaknesses were financial logic, contextual application of expertise, and scope and completeness.

“Um, the three biggest issues that we saw across models for this benchmark were financial logic, where it was unable to, um, unable to, you know,”

“exercise this kind of expertise, uh, in context with the environment, as well as scope and completeness. So it, it might, it might have gotten through, you know, uh, steps one through nine of... but it left h- step ten unfinished, right?”