opinion claim
Benchmark-based forecasting has a streetlight-effect limitation: a model reaching 100% on a benchmark may appear to have solved the task, but that benchmark result can be misleading.
Unidentified speaker · Three Red Lines We're About to Cross Toward AGI (Daniel Kokotajlo, Gary Marcus, Dan Hendrycks) · 24 Jun 2025
In their words
The transcript segments this claim was taken from
“And I think that has limitations,”
“in particular a streetlight effect where,”
“when it gets 100% that will suggest that it has the task.”
“or that suggests it's solved the task.”Play from 1:19:10Report a misattribution
85%how strongly it was put, not whether it is true
75%importance to the conversation
90%confidence the claim was read correctly