The speaker says model size should increase when data is the bottleneck, but decrease when compute is the bottleneck.

“mean this could well yeah this depends on like the ratio of like training and inference compute really it's like if your super bowl neck on data not on compute you should go bigger if a super bowl neck on compute you should always go smaller and then like yeah”