The speaker thinks that steering can behave erratically or produce gibberish when an intervention moves activations off the model's representation manifold.

“exactly because exactly like you say when you try and go from one point to another you're sort of you're just stepping out into this void which the network doesn't really know how to handle um and then it sort of breaks and i think this actually explains a lot of a lot of findings about steering this is the sort of”

“to steering just being intervening on activations um it's a fairly you know we do a lot of steering some other people do a lot of steering and one one common finding with steering neural networks is like sometimes it works and it's amazing and you get golden gate clawed or whatever um and sometimes it's just like completely janky um and the network does kind of the thing you want but also just goes a bit crazy um”

“or just turns immediately into gibberish and I think this basically like explains that phenomenon”

“Because you're sort of stepping off stepping off manifold”