Most improvements in pre-training data come, in the speaker’s view, from research into which datasets are useful and labor-intensive work to filter them.

“think the vast majority of pre-training data improvements are from science on better understanding what datasets are good and schleppy labor on figuring out how to filter down.”