The speaker says bugs also exhibit clean scaling laws.

“and bugs have their own clean scaling laws as well right like you know like with Kaplan for getting the cosine annealing thing or like even just like not considering embedding parameters I think and so that messed up the estimated smaller models because embedding parameters are a decent size of the model”