Depending on probe accuracy, training against a probe may remove the targeted behavior rather than the underlying representation, while other accuracy regimes do not show that outcome.

“and it turns out there are regimes in which this there are regimes of probe accuracy in which it seems easier to like the behavior will go away rather than the representation there are regimes in which it won't now if you back propagate through the probe you're just cooked right like this is basically always a bad idea and so you know I think people have sort of”