GPT-5.4 had a failure rate of approximately 58–60%, and the failure rate was reduced substantially in the subsequent generation.

“Um, I think we can, you know, leave it there for people that wanna check out more. There is a paper, there's a data set. Um, you're right, the jumps are pretty crazy. Five four was like sixty percent failure rate, fifty-eight percent, and they were able to cut this down-”

“... pretty significantly. Uh, to tie it to handshake, people come to you saying they want evals on stuff like this. You externally run evals and then hand it off.”