Dario said that model performance tends to degrade at long contexts and that training and serving context lengths are distinct.

“is not infinitely strong but he was explaining why model performance tends to degrade at long context he said there's two things there's the context length you train at and there's a context length that you serve at if you train at a small context length and then try to serve at a long context length like maybe you get these degradations”