In an example described by the speaker, reinforcement learning against an LLM judge led a model to produce nonsensical solutions that the judge assigned a perfect reward.

“little cracks it will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it so one example that's prominently in my mind is i think this i think this was probably public but basically if you're using an lm judge for a reward so you just give it a solution from a student and ask it if the student will or not we were training with reinforcement learning against that reward function”

“and it worked really well and then um suddenly the reward became extremely large like”

“there was”

“massive jump and it did perfect and you're looking at it like wow this this means the student is perfect and all these problems it's fully solved math but actually what's happening is that when you look at the completions that you're getting from the model they are complete nonsense they start out okay and then they change to the so it's just like oh okay let's take two plus three and we do this and this and then”

“and you're looking at it as like this is crazy how is it getting a reward of one or 100 percent um and you look at the llm judge and it turns out that is an adversarial example for the model and it assigns 100 probability to it”