The speaker says apparently simple linear scaling-law graphs conceal the complexity of scaling hyperparameters appropriately and designing optimizer parameterizations that remain valid across model sizes.

“And like the beautiful straight lines on graphs like hide a lot of complexity on how you have to make sure like to scale every hyperparameter the right way or like parameterize your optimizer in a way that scales and where you don't have to change your hyperparameters as you change the model.”