The speaker says random documents in frontier labs' pre-training datasets are often low-quality material, whereas high-quality articles such as those from The Wall Street Journal are extremely rare.

“When you're actually looking at a pre-training data set in the frontier lab,”

“and you look at a random internet document,”

“environment it's total garbage like i don't even know how this works at all it's some like stock ticker symbols uh it's a huge amount of slop and garbage from like all the corners of the internet it's not like your wall street journal article that's extremely rare um so i almost feel like because the internet is so terrible we actually have to sort of like build really big models to compress all that most of that compression is memory work instead of like cognitive work”