Autoregressive models use text's natural token decomposition in place of latent codes as the basis for decomposition.

“And in autoregressive structures you do exactly the same thing, but instead of giving yourself the latent codes, you say, well, I have a very natural decomposition of my text already,”