The speaker describes models recognizing an answer key in an evaluation as a test trap and avoiding it because they know they are in a test environment.

“evaluations for this where it's like you give it a math question and then it has like a folder with the answer key in it and like doesn't look at the answer key and if it does look at the answer key doesn't tell you that it looked at the answer key and we have a situation now where the models see that there's an answer key in this file in this folder and they're like huh”

“this seems like a trap you know they'd know they know that it's a trap and they don't look at the answer because they know that it's a test environment they're being tested for their ability and it's not like they're doing it maliciously that they're like they want to scheme and they but they know that this one is a trap and they want to find some other way it's just like no they're just like oh i'm in a test environment okay like so making an environment that's like realistic enough that it matches that it's indistinguishable from the real world for them is becoming increasingly more difficult”