Neural-network training knowledge is a hazy recollection because the training data is compressed dramatically into the model.

“And that's because the compression is dramatic.”

“like you've you're taking 15 trillion tokens and you're compressing it to just your fine neural network with a few billion parameters so obviously it's a massive amount of compression going on uh so i kind of refer to it as like a hazy recollection of the internet documents whereas anything that happens in the context window of the neural network you're plugging all the tokens and building up all this kv cache representation is very directly accessible to the neural net so i compare the kv cache and the the stuff that happens at test time to like more like a working memory like all the stuff that's in the in um”