The speaker says that training an LLM for as little as one or two iterations on a random sequence derived by hashing text can make it memorize and reproduce the whole sequence.

“You can give them completely.”

“completely nonsensical data like you can take um you can hash some amount of text or something like that you get completely random sequence if you train on it even just i think a single iteration or two it can suddenly regurgitate”

“the entire”

“thing it'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn those generalizable components whereas llms are distracted by all the memory that they have of the pre-training documents”