In their tests, a collective of LLM judges judged less accurately than a single judge; they replaced the collective with one judge, which performed better.

“They tested the ability and accuracy of that LLM as judge collective to judge,”

“and it actually didn't perform as well as one.”

“So they replaced all those judges with one judge,”

“and the one judge actually performed out the collective.”