For tasks combining verifiable and semi-verifiable components, graders must assess both; large training runs have failed when models performed well on the verifiable component but ignored the semi-verifiable one.

“So now the grader has to s- to both do semi-verifiable and verifiable in the same task. And if we-- he saw large training runs, uh, fail to produce results because the model was very good at verifiable, but it ignored the fact that there was a lot of semi-verifiable component in it as well.”