At data-center scale and across thousands of users, inference economics largely reduce to the cost per gigabyte of the memory employed.

“of particular ineffable property which is incredibly high bandwidth to memory so we can load the weights we can load this state you know many thousands of times per second but also have like economical memory because when you look at what it looks like to run inference at data center scale for thousands of users the economics actually collapse down to essentially a kind of cost per gigabyte of the memory that you're employing”

“we're implying.”