Autoregressive language models model the probability of generating a sentence given a prompt by producing an answer token by token, which works well because language has a discrete measure.

“wha a to model the probability of generating a certain sentence given uh let's say a prompt. And what you end up doing is you end up kind of producing kind of step by step uh the the actual answer by going token by token by token. This works really nicely in something like uh language modelling, because you ultimately still have a discrete measure.”