A language model models language and need not use a transformer-decoder architecture.

“or fable because they're the best models but because of that like people have gotten kind of accustomed to this idea that a language model is just a auto regressive decoder like we have accepted this and i think when rlms came out it was the same thing like one of the comments like a very frequent criticism i got was like this is not a language model or like when i look at this like i thought it was a new architecture but it's actually not and my response to that is like well”

“a language model is just modeling language it doesn't have to be this transformer decoder you know and jeb is really really interesting in that like we now have a new”