If reward-function elements are correlated, adding a new reward later can break the system and require all weights to be retuned because the weights are interdependent.

“Because one of the problems I talked about with the open AI example was that if two elements of your reward function are correlated and you uh and for example you add something to it like let's say you didn't at first you just had like a a bonus for killing enemy you did not have like a gold bonus at first and now you add the gold bonus uh later because well doing engineering you found well that would be a good idea as an additional guide.”

“Suddenly your entire system breaks and you need to retune all the weights, because the weights are tied relative to each other. If you model it as a constraint, let's say uh you should have more goal than your opponent in expectation, right, that that's a reasonable uh idea.”