Disagreement
Human discovery as a benchmark for judging language models
Scott Alexander and Dwarkesh Patel disagree over whether a language model's failure to make the discovery under discussion warrants a negative judgment. Alexander's position is that humans also cannot make that kind of discovery; Patel's counterargument is that humans have demonstrated the ability, so models should be judged more negatively for failing to do so. The evidence captures their opposing positions in the same show.
2speakers
1show
1–1claims for and against
0.90confidence, no higher than its weakest claim
The split
One segment per claim
At issueThe speaker's example was that humans also cannot make the kind of discovery under discussion.
For it: 1 · Scott AlexanderAgainst it: 1 · Dwarkesh Patel
When each side was argued
For it · 1 claim
“Yeah, and then the example I gave was that humans also can't do this.”
Against it · 1 claim
The speaker argues that language models should be judged more negatively for failing to make discoveries of this kind, since humans have demonstrated the ability to do so.
“You have a blog where you do this. This is your job, Scott. So why shouldn't we hold the fact that language models can't do this more against them?”