Moore's law maintained a seemingly straight scaling trajectory for a long time, but sustaining it required many discrete innovations and discontinuities.

“with that paradigm where you know it's basically just like self-attention rl scaling up rl environments the i guess like if you think about what happened with moore's law right like we had this very like nice straight line and that held for a really really long time but there were so many like discrete like discontinuities and innovations that had to happen to keep that scaling law going and the same thing has kind of happened with llms like we had this pre-training like scaling law and then that was kind of like you know hitting the dimension”