Disagreement
The behavioral impact of small reinforcement-learning updates
The disagreement concerns whether a small amount of reinforcement learning on a strong model necessarily produces only a small behavioral effect. Dwarkesh Patel emphasizes that an RL step changes relatively few parameters, an unresolved speaker describes RL as making very small model changes, and Beren Millidge characterizes post-mid-training RL as policy adjustment rather than learning behaviors from scratch. Millidge nevertheless argues that even a small amount of learning can substantially change behavior when the starting model is strong.
2speakers
1show
1–3claims for and against
0.90confidence, no higher than its weakest claim
The split
One segment per claim
At issueStarting from a strong model, a small amount of reinforcement learning does not necessarily produce only a small behavioral effect.
For it: 1 · Beren MillidgeAgainst it: 3 · Dwarkesh Patel, Unresolved speaker and Beren Millidge
When each side was argued
For it · 1 claim
“can still be like super dramatic you know even if it's like even like one bit can change like your function space a lot and it can like rule out like half the hypothesis space which is huge so like i don't think it's necessarily the case it's like small amounts of bits small amounts of rl once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily”
Against it · 3 claims
The speaker says reinforcement learning is attractive because it makes only very small changes to a model.
“That's why actually very few parameters are changed. during an RL training step.”
“often and then what rl does on top of that is it does like a lot of you know essentially tweaking to the policy and so this is one of the reasons why it doesn't need like as many bits as you would naively think it doesn't have to learn all of these behaviors from scratch it needs just like a few bits from these episodes which you do get and then the other thing that i really point out in my blog is that these bits are actually extremely high signal compared to like regular like pre-training which is why you need rl at all versus just like sfting on like successful reasoning traces”
“Like the reason RL is so nice is because like, yeah, it changes a very, very small amount about the model and there's like a lot of evidence for why this is the case.”