The speaker suspects that poor-quality internet training data drives the need for very large models, whose work is mostly memorizing and compressing data rather than cognition.

“environment it's total garbage like i don't even know how this works at all it's some like stock ticker symbols uh it's a huge amount of slop and garbage from like all the corners of the internet it's not like your wall street journal article that's extremely rare um so i almost feel like because the internet is so terrible we actually have to sort of like build really big models to compress all that most of that compression is memory work instead of like cognitive work”