Karpathy describes reinforcement learning as using a single final reward to upweight or downweight an entire trajectory, thereby spreading sparse supervision across all its steps.

“only to find a single at the end you get a single number of like oh you did correct and and based on that you weigh that entire trajectory as like upweight or downweight and so you're the way i like to put it is you're sucking supervision through a straw because you've done all this work that could be a minute of rollout and you're you're like sucking the bits of supervision of the final reward signal through a straw and you're like putting it you're like you're basically like um yeah you're broadcasting that across the entire trajectory and using that”

“using that to upweight or downweight that trajectory.”