Much of the apparent success of reinforcement learning comes from high-quality mid-training data, such as synthetic reasoning data and environments that prepare models for reinforcement learning.

“I mean so I think the success of RL comes down to a bunch of different things so first I think what is slightly underestimated is actually the mid training so an awful lot of like what we see as successes of RL actually comes from like very very good mid training data which is basically where we're like essentially doing pre-training but on like synthetic reasoning data and like the kind of environments that like get the model warm started for RL and so this actually takes the model like almost like 80% of the way to like the final RL checkpoint often.”