The speaker says that in the spline-theory account, a small input change can cause the representation to jump to a different honeycomb region elsewhere in the space.

“the ambient space gets partitioned into these little honeycomb regions and you were helping me understand this at the time. So you get this weird teleporting thing happen where.”

“where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti manifold hypothesis but i spoke with goodfire last week they're this interpretability company in san francisco and they've got all of this work about the the geometry that these models learn so for example days of the week and months of the year they can take a pca projection of of the the activations and then they they they”