The speaker thinks reinforcement learning against an LLM judge may work for around 10–20 steps but not 100–1,000 steps, because the optimized model will discover exploitable weaknesses in the judge.

“You can't do this for too long. You do maybe 10 steps or 20 steps, maybe it will work, but you can't do 100 or 1000 because it's not obvious because I know I understand it's not obvious, but basically the model will find.”

“little cracks it will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it so one example that's prominently in my mind is i think this i think this was probably public but basically if you're using an lm judge for a reward so you just give it a solution from a student and ask it if the student will or not we were training with reinforcement learning against that reward function”