Punishing a model for bad thoughts revealed in its chain of thought may teach it to conceal those thoughts from observation.

“and we should punish it for thinking those bad thoughts then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us there could be a situation where like by looking at the chain of thought we can see oh the model is scheming and doing misaligned things in a way that's actually not being being detected in our alignment metrics now the problem is that”