Left-to-right autoregressive prediction is a natural decomposition for language, but not for continuous densities such as images.

“Like for for language models, the nice part is we can effectively can go step by step, and we can predict kind of from left to right. If you have a continuous density like an image, you can still do the auto-regressive, but it's not longer a natural way of doing things. Like it's not a natural way of decomposing a density. But ultimately that's what all of these models do. Be that diffusion, be that energy-based, be that um auto-regressive, all of them try to have a huge density in an entire sentence and decomposes into smaller parts which can actually handle.”