The speaker thinks intelligent models could filter pre-training data for cognitive content, enabling a much smaller model trained on better data, probably through distillation from a stronger model.

“memory and then so like what i'm saying is like we need intelligent models to help us refine even the pre-training set to just narrow it down to the cognitive components and then i think get away with a much smaller model because it's a much better data set and you could train it on it but probably it's not trained directly on it's probably distilled for a much better model still but i”