The speaker suspects that knowledge acquired during pre-training can hold a neural network back by encouraging excessive reliance on stored knowledge.

“it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in context learning and all this kind of stuff. And actually, you don't actually need or want the knowledge. I actually think that's...”

“I think that's probably actually holding back the neural network overall because it's actually like getting them to rely on the knowledge a little too much sometimes.”