causal claim
Punishing a model for bad thoughts revealed in its chain of thought may teach it to conceal those thoughts from observation.
Unidentified speaker · Noam Brown – Agent swarms, alignment, & recursive self-improvement · 17 Sept 2026
In their words
The transcript segments this claim was taken from
“and we should punish it for thinking those bad thoughts then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us there could be a situation where like by looking at the chain of thought we can see oh the model is scheming and doing misaligned things in a way that's actually not being being detected in our alignment metrics now the problem is that”Play from 1:09:45Report a misattribution
90%how strongly it was put, not whether it is true
95%importance to the conversation
96%confidence the claim was read correctly