Supervised fine-tuning includes the correctness information in the answer token while also training the model to match the exact reasoning tokens in a trace.

“well there's two things so yes one it's exactly the bits about how to get the answer right but like this is not exactly how you think of it because in sot you have a trace right you have like a bunch of math reasoning and then the answer at the end the bit is still there like you're still sot on the answer token so that bit is still there what's important is that the object”

“ignores all the other bits so in sft you like have like you know to try and match like the exact reasoning tokens that the model produces so you're essentially getting like too many bits about like the exact way this other model you're training on reasons for rl you only get the one bit and that means that like this that signal is not drowned out in the noise of like all the other bits the model has and so that's what really like it's really super dramatic like increase into the signal to noise ratio during training which is why like rl is like”