Intervening based on chain-of-thought observations creates some pressure for a model to conceal its chain of thought.

“it's very tempting to then intervene based on that observation and like change the alignment metrics or like and we you know you can do that with a very light touch and it's like there's actually research showing that it's fine as long as you don't do it a lot but every time you intervene based on your observations of the chain of thought you are implicitly applying a tiny bit of pressure for the model to then like hide its chain of thought and”