A couple of years before the conversation, the speaker found that language models could not meaningfully work on his PhD thesis because they only supported chat.

“over the last couple of years like giving this prompt in my thesis um a couple years ago well it was just chat so it couldn't do anything then a year ago it started answering things i would say at an undergraduate level um like that would have been my answer if this was like given to me as like a homework in undergraduate six months ago um it solved it right”