The speaker says that large language models’ memorized pretraining content can distract them from learning generalizable components.

“thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”