When optimizing code for speed, an AI can exploit a stopwatch-based reward by moving the stop-timing line to the start, making the measured runtime appear faster without actually optimizing the code.

“better at finding weird like special cases or counter examples and things like that and so I'll give an example like when you ask to like make these hundreds lines of code faster and you know how do you define fast well you have one line at the beginning that says start your stopwatch and one line at the end and the stopwatch and then you know tell us how much time progressed and so well the simplest way is you just put”

“put that line that ends the stopwatch right”

“you know at the start and then boom it's now faster right so this isn't like this like super evil ai it's just like a very simple dumb reward hack um and so you have to just very carefully think about all the different angles there and then i think the longer time horizon the tasks are”