The speaker says earlier, smaller models could go off track at around 10,000 tokens of context.

“you know models were really small like before and at 10 000 tokens they would like go off the rails so it's a lot of our work is i think about like making sure that the bot does not hallucinate and trying to like keep track of the conversation as the the turns progress and the turns are fairly short because”