OPSD offers denser supervision than naive RL by using per-token teacher–student probability discrepancies instead of a single trajectory-level reward.

“And two,”

“OPSD provides a much denser supervision signal than naive RL.”

“Instead of projecting a single reward through the whole trajectory,”

“you can train on the per-token probability discrepancy between the teacher and student.”