The speaker says it would be very difficult to train a state-of-the-art model using only 100 billion tokens.

“over the past 30 years or something on the internet like imagine you were trying to train a state-of-the-art model but you only have like 100 billion tokens right to train on that would be very difficult so in a certain sense we our entire economy has produced this huge amount of data on the internet that we are now using to train the models it's plausible that”