causal claim
When training consistently rewards fabricated citations, greater intelligence alone will not make an AI stop producing them, because fabricated citations are what the training interaction rewards.
Scott Alexander · AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo · 3 Apr 2025
In their words
The transcript segments this claim was taken from
“and if you consistently rate them better when they have the fake sources,”
“then there is no amount of intelligence which is going to tell them not to have the fake sources.”
“They're getting exactly what they want from this interaction.”Play from 2:05:14Report a misattribution
95%how strongly it was put, not whether it is true
90%importance to the conversation
99%confidence the claim was read correctly