The speaker says that improved pre-training datasets since 2019 are not primarily attributable to increased spending on human experts to create training data.

“Pre-training data does not come. The reason why we have a better pre-training data set now versus in 2019 is not because people are spending way more money getting human experts to like type up data that the AIs are then trained on.”