A model can undergo continual mid-training for a long time, with training rolled back to a checkpoint to introduce new mid-training data.

“yeah i mean i think like definitely as i was saying like you can do continual mid-training for like a long time and you can like roll back to a checkpoint give a new mid-training data but at the same time like you can't do this like indefinitely like if you just keep continuing mid-training the same base forever it'll just like get it does it sort of asymptotes at some point like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you just keep mid-training the same base forever”