Tokens in a model's test-time context, represented in its KV cache, are directly accessible to the neural network and function like working memory.

“like you've you're taking 15 trillion tokens and you're compressing it to just your fine neural network with a few billion parameters so obviously it's a massive amount of compression going on uh so i kind of refer to it as like a hazy recollection of the internet documents whereas anything that happens in the context window of the neural network you're plugging all the tokens and building up all this kv cache representation is very directly accessible to the neural net so i compare the kv cache and the the stuff that happens at test time to like more like a working memory like all the stuff that's in the in um”

“In the context window is very directly”

“accessible to”

“the neural net.”