Continually mid-training the same base model indefinitely eventually reaches an asymptote, limiting its ability to learn new information.

“yeah i mean i think like definitely as i was saying like you can do continual mid-training for like a long time and you can like roll back to a checkpoint give a new mid-training data but at the same time like you can't do this like indefinitely like if you just keep continuing mid-training the same base forever it'll just like get it does it sort of asymptotes at some point like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you just keep mid-training the same base forever”