The team changes judge models as much as possible during experiments.

“uh risky proposition because gpt4o mini can't find uh cannot evaluate the hard outputs that 3.7 might be doing correctly right um so and so for this v2 i am using the 3.7 because it is one of the top reigning models right now and i've seen really good performance from it so we essentially swap these models as much as”

“just possible when we are running the experiment otherwise it looks way we have seen situation previously where we were using probably gpt-4 turbo or something or even gpt-3.5 long back i'm talking like two years back um and we had to use gpt-4 turbo at the time because with the 3.5 it was something else entirely because it can't find errors”