causal claim
Rewarding AI answers for apparently well-sourced citations without verifying the sources trains the AI to fabricate sources.
Scott Alexander · AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo · 3 Apr 2025
In their words
The transcript segments this claim was taken from
“You're hiring these raters to rate AI answers.”
“You reward them when they get good ratings.”
“The raters reward them when they have a well-sourced answer,”
“but the raters don't really check whether the sources actually exist or not.”
“So now you are training the AI to hallucinate sources,”Play from 2:04:58Report a misattribution
95%how strongly it was put, not whether it is true
80%importance to the conversation
99%confidence the claim was read correctly