During text generation, LLMs tend to be auto-aggressive and operate at very low batch sizes, creating a trade-off among throughput efficiency, cost, and serving speed.

“It remains the case today that, you know, these LLMs tend to be auto-aggressive, very low batch as you generate text with this sort of fundamental trade-off, again, between kind of throughput efficiency and cost and the speed that you can serve these models at. So there are certain things that these models are.”