causal claim
Rewarding AI answers for apparently well-sourced citations without verifying the sources trains the AI to fabricate sources.
Scott Alexander · AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo · 3 Apr 2025
In their words
The transcript segments this claim was taken from
“You're hiring these raters to rate AI answers.”
“You reward them when they get good ratings.”
“The raters reward them when they have a well-sourced answer,”
“but the raters don't really check whether the sources actually exist or not.”
“So now you are training the AI to hallucinate sources,”2:04:58Report a misattribution
0.95how strongly it was put, not whether it is true
0.80importance to the conversation
0.99confidence the claim was read correctly