The extent of model growth depends in non-obvious ways on scaling laws, and data efficiency may matter more than compute efficiency as high-quality pre-training data becomes scarce.

“bigger but i would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways so one thing is that i think like data efficiency is going to be a bigger driver than compute efficiency of like the exact architectures people use now that we're getting to the regime where we're sort of running low on like high quality pre-training data so that might affect how sparse you want to make the model”