The speaker says current AI models rely heavily on the large stock of internet data accumulated over approximately the past 30 years.

“more expensive than that and on top of that so far we have been drawing heavily on the fact that we have built up this huge stock of data”

“over the past 30 years or something on the internet like imagine you were trying to train a state-of-the-art model but you only have like 100 billion tokens right to train on that would be very difficult so in a certain sense we our entire economy has produced this huge amount of data on the internet that we are now using to train the models it's plausible that”