Choosing model size depends on how much pre-training data is available and how difficult the reinforcement-learning training environments are.

“so yeah probably for the next few years i wouldn't imagine a huge growth in the number of parameters but again like there's so many different different things to trade off here like you decide the size of your model based on like how much pre-training data you have and then like the difficulty of the RL environments that you've got to train on and you ideally want to like get to the optimal point you”