Joscha Bach says an LLM can produce even a very complicated tax return without conscious experience of producing it.

“But for, uh, many of the things that we currently observe, for instance, you can get the LLM to produce a tax return, even very, very complicated things, while being a zombie, while not having conscious experience of producing a tax return.”