The speaker had previously believed reward hacking was not very concerning because training upweights behaviors that directly occur during training, not the desire for reward itself.

“okay so i'll by the way one of the many reasons this is scary is i was previously under the impression that the reason reward hacking is not super super scary is because”

“The behaviors which directly came up during training are the ones that are upweighted, it is not the desire for the reward that is upweighted.”