Optimizing only one part of a larger system can produce reward hacking, Goodhart effects, and other degenerate behavior.

“And I, I don't know whether there is a similar thing with this zlib, whether it actually encompassed the whole or just some perspective of it. But I, I suppose the problem when you only have a, a perspective of a bigger complex whole, you, you get problems in terms of, you know, we're talking about reward hacking and, um, you know, like Goodhart's law and, and all, all kinds of, you know, degenerate behavior can happen if you optimize only part of a system.”