If in-context learning improves across longer time horizons, it may be unnecessary to distill all deployment-time learning into model weights.

“Similarly, people would say,”

“look,”

“continue learning.”

“This capability I keep harping about where the model's weights get updated based on what it's learning from deployment may simply not be necessary because if in-context learning gets so good across longer and longer time.”

“or time horizons, then you don't need to distill back everything the model is learning on the job into the weight.”