High-bandwidth access between compute and model weights or KV cache can enable language models to run at thousands of tokens per second.

“And so you have this extremely high bandwidth between your compute where you're running the maths as you run these models.”

“and let's say the weights of your model or the kv cache of this model as you're rolling it out um and that's something which then then can drive you to thousands of tokens per second on these language models um but i think one of the things that we started to worry about in kind of towards the end of 2023 and certainly in 2024 was the scalability of this approach um and you know i think there are two things that”