The speaker says that issues including cosine annealing and omitting embedding parameters distorted estimates for smaller models, because embedding parameters constitute a substantial part of a model.

“and bugs have their own clean scaling laws as well right like you know like with Kaplan for getting the cosine annealing thing or like even just like not considering embedding parameters I think and so that messed up the estimated smaller models because embedding parameters are a decent size of the model”