Training a 100-million-parameter model on three benchmark traces would not yield the same improvement as training a substantially larger model.

“so i mean on scale dependence i think like a lot of the like mid-training and post-training data we have now is like actually but gets better with scale because like a lot of it like the very long context horizon environment stuff really requires like big models to be able to like make use of them and like this is not you know if you try and train like your 100 million parameter model on like three bench traces it's not going to get anywhere like it's not going to show you the same kind of improvement that you would get if you train like an actual”