Joscha Bach says an LLM's text output does not reveal what the text indicates, because LLMs are trained on novels that can describe experiences their authors did not have while writing.

“Now, there is a problem with the LLM because the LLM is a text generator, and you don't know what the text indicates, right? The text is trained on novels, for instance, and in many novels, the authors write about stuff they don't experience when they write it, right?”