The speaker says naive next-token pretraining can generalize to tasks requiring deep input understanding or skills that were rare in the training data.

“Just pre-training on this very naive next token prediction objective to like various tasks of interest where some very that require understanding of the input in some deep way or learning some skill from pre-training that's like very rare and like not very like heavily represented and then like”