The speaker says that filtering data to extract useful pre-training signal is fairly automatable.

“and that's quite an automatable process but like as the models get better as we enter mid-training and post-training the signal just like doesn't exist anywhere in the original data we have like no amount of filtering will like get this you know there's no like hidden proof of like the millennium prize problem sitting in common core we can just like”