The speaker says that for tasks of interest in pre-training, useful signal is already present in Common Core, and the challenge is filtering noise rather than obtaining signal.

“like the signal doesn't you know in pre-training the signal is like already in common core right like for the tasks that you care about in pre-training the problem is there's not just like getting signal at all it's like filtering out all the noise that exists”