Over the last 20 years, chip FLOPs increased about one million-fold, compared with about a 40-fold increase in memory bandwidth.

“this property of memory bandwidth which hasn't really been scaled so much on chips recently we've scaled flops like a million fold in the last 20 years memory bandwidth has gone up about 40x in the same time frame so as you scale that frontier you get to conserve more of the other thing we get to reduce the number of flops that we're using on these models to get to a certain level of intelligence and so that's the kind of really a multiplying factor on you”