The speaker proposes that recognizing next-token prediction as an optimization target may have been a major AI breakthrough.

“um i actually john i'm curious for your like sort of uh wisdom of the ages of um or wisdom of like being in the trenches way back when but presumably a big breakthrough was realizing that next token prediction is the like you wouldn't have thought that nanos nano gpt speed run”

“is the thing to be optimizing for in”

“2014 but now that we have come to this new paradigm that's you would think to do a speed run on that and have you guys get really good at that but maybe there's like a next”