The speaker says pre-training scaling laws depend on current architecture choices remaining relatively stable; a complete architecture change would probably invalidate those laws or substantially alter the power-law relationship.

“you know maybe one model converges better than the other one like slightly but in general like if you were so for example pre-training scaling laws only hold because the architecture choices we have are somewhat stable right now but if you were to completely change the architecture pre-training scaling laws probably don't hold or like these kinds of this like power law is going to look very different and it's like the same thing with harnesses like i think all the harnesses we have right now for the most part”