The speaker believes that because labs control the best models, people have become accustomed to defining language models as autoregressive decoders.

“to give you a trade-off which is i will do very very fast inference over like if you have some prior about this problem like let's say i only need to make a binary classification am i going to ask my language model to do this and pay like a 400x cost no that's like silly right um and i think for the longest time because the labs are the only places that control you're never going to use something other than like gpt6 or”

“or fable because they're the best models but because of that like people have gotten kind of accustomed to this idea that a language model is just a auto regressive decoder like we have accepted this and i think when rlms came out it was the same thing like one of the comments like a very frequent criticism i got was like this is not a language model or like when i look at this like i thought it was a new architecture but it's actually not and my response to that is like well”