The speaker attributes some of the lower-than-expected increase in model scale and per-token cost to doing more work at small scale, which enables more training runs and iteration cycles.

“And I think we have seen this. Like I think one reason why the AIs have been scaled up less than you would have otherwise expected and like, for example, cost of per token hasn't increased as much as you might have thought is because there is a benefit to doing more of your work at small scale where you can run more training runs and get more cycles in. And so you're not as like,”

“you know, leaning as hard on like one big.”

“you know, really important training run.”