The speaker argues that people gain knowledge by actively manipulating information and would like pretraining to include a stage in which a model reasons about material and reconciles it with existing knowledge.

“and by manipulating that information that you actually gain that knowledge and i think we have no equivalent of that again with other lumps they don't really do that but i'd love to see during pre-training some kind of a stage that thinks through the material and tries to reconcile with what it already knows and thinks through for like some amount of time and gets that to work and so there's no equivalent of any of this this is all research there's some subtle”