Autoregressive inference maps poorly to GPUs because it is extremely memory-bound: most time is spent moving weights through the memory hierarchy and little on arithmetic.

“That kind of workload does not map well to GPUs. That kind of workload is extremely memory bound.”

“You're spending most of your time moving around weights across the memory hierarchy and you're doing very little.”

“a little arithmetic,”

“and that's a fundamental problem of autoregressive models.”