Thomas Larsen

“our overall proposal is in the meantime, so between in our scenario between sort of 2030 and 2040, which is roughly when we're in the human range of AI capabilities, we're going to basically almost entirely rely on control.”

Expects: In the speaker's AI 2040 scenario, people will rely almost entirely on control when AI capabilities are around the human range, roughly from 2030 to 2040. · 1 Jan 2030 – 31 Dec 2040 · not yet resolved

“And that's going to come further and further apart from like sort of the actual measurement of whether they're actually good or not.”

Expects: AI adaptation to evaluation setups will increasingly make evaluation results diverge from the underlying quality being measured. · no date given · not yet resolved · In a shared prediction of 3 →

“that relieves pressure on the regulators because if something sort of catastrophically or existentially unsafe is happening, sort of society as a whole, academia, you know, nonprofits.”

Expects: The speaker expects society, academia, and nonprofits to be able to respond if catastrophically or existentially unsafe AI activity is occurring. · no date given · not yet resolved

“And so sort of society as a whole would be able to like weigh in, there'd be able to be public and informed debates.”

Expects: Under Plan A's transparency principle, the public would be able to participate in informed scientific debate about AI. · no date given · not yet resolved

“We build A.I.s, they're aligned to humanity, but to who? Like, s you know, is it the president, is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? Um, you're probably NOT gonna be the last one, and so you know...”

Expects: AI alignment is probably not going to be determined by a broad, endorsed democratic process that aggregates everyone's values. · no date given · not yet resolved

“You know, things are just not going to go as you expect.”

Expects: The course of the war will not unfold exactly as expected. · no date given · not yet resolved

“you know, that's sort of the hope with sort of our like we ourselves do a bunch of war games, but also do a bunch of sort of detailed scenario writing. Our hope is that every time we have to write a part of the scenario and the part of that scenario seems super unrealistic or. you know isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online that's basically reality yelling at us and trying to like help us see reason and our hope is that we can sort of like do enough of this sort of put up enough surface area so that we can sort of get that you know dose of reality from from the real world or from the simulation of the real world which we hope is realistic enough to sort of accurately give us that information”

Expects: The authors hope that war games, detailed scenario writing, and criticism will reveal unrealistic or poorly modeled parts of their scenarios and help them correct their thinking. · no date given · not yet resolved

“you know isn't really well modeled and doesn't really make sense or people are able to make really good criticisms of it online that's basically reality yelling at us and trying to like help us see reason and our hope is that we can sort of like do enough of this sort of put up enough surface area so that we can sort of get that you know dose of reality from from the real world or from the simulation of the real world which we hope is realistic enough to sort of accurately give us that information”

Expects: The authors hope that exposing enough of their work will elicit feedback from reality or sufficiently realistic simulations that provides useful information. · no date given · not yet resolved

“In particular, I'm pretty worried about this whole like increasing awareness of sort of very smart AIs and how important they'll be and how much they'll reshape the world and then not causing people to go, I want to be the one in charge of the AGI or the super intelligence, so I'm going to race towards that. And I think historically that's...”

Expects: Rising awareness of highly capable AIs and their transformative importance could motivate people to race to control AGI or superintelligence. · no date given · not yet resolved

“and so from our perspective like that's going to happen at some point because there's nothing fundamental stopping the ais from sort of reaching this human level of capability”

Expects: Companies will eventually prefer AI labor over human labor, as AI reaches human-level capability; the speaker presents this as a prediction based on there being no fundamental barrier to that capability. · no date given · not yet resolved · In a shared prediction of 16 →

“and so from our perspective like that's going to happen at some point because there's nothing fundamental stopping the ais from sort of reaching this human level of capability”

Expects: AIs will eventually reach human-level capability, as the speaker expects this to happen because he sees no fundamental barrier preventing it. · no date given · not yet resolved · In a shared prediction of 16 →

“feedback loops and i guess my view is that we're sort of going to get this continuous expansion of what the ads can do driven probably in part by uh you know an expansion in the amount of rl and the type of rl”

Expects: AI capabilities will probably continue to expand, driven partly by increases in the amount and variety of reinforcement learning. · no date given · not yet resolved · In a shared prediction of 16 →

“and sort of there's going to be this continuous process of expanding out through the different types of problems and different like how like exactly how verifiable each task is until you get sort of sort of everything that humans could do because after all we humans do learn how to do these long horizon tasks somehow”

Expects: AI capabilities will expand across problem types and degrees of verifiability until they encompass the tasks humans can perform. · no date given · not yet resolved · In a shared prediction of 16 →

“Yeah, so my view is yes. basically they guys will be able to uh everything or at least everything that really matters so there is yeah you mentioned the notion of bullshit jobs i would i would just sort of start with let's consider everything we need to make better ais as maybe like a first step of like which is a large chunk of the economy so for example for this what you need you need to be able to build bigger better chips and more chips and that's the entire semiconductor supply chain to build semiconductor supply chain you need”

Expects: The guest believes AI agents will probably become able to do everything, or at least everything that really matters. · no date given · not yet resolved · In a shared prediction of 16 →

“to really speed up and sort of massively change the overall world, even if, for example, you know, let's say, let's say like there is occupational licensing or whatever, preventing the AIs from doing like some random legal work or some random whatever work or whatever bullshit jobs throughout the economy. I think sort of what really, really matters is sort of this is like the stuff that's actually really important, in particular, the robots, the compute and the better AI.”

Expects: AI, compute, and robotics capabilities could greatly accelerate and transform the world, even if occupational licensing blocks AI from some legal and other jobs. · no date given · not yet resolved

“successful companies all at once and then no one else can really do that or at least has succeeded at doing that it's just because of the sort of multiplicative like you need it to be like 90th percentile and like each of these 10 10 domains which is just very very unlikely but if you could have an uh if you had an ai that's sort of like you could just train to be like really high percentile in all of these skills in such a way that no human is it would be like extremely rare or you know infinitesimally unlikely for any particular human to have all of those skills at once i think you would just actually just be really really really good at changing the world in all of these like concrete and important ways just like sort of elon has actually done that”

Expects: The speaker thinks an AI trained to rank very highly across many skills could be extraordinarily capable, while it would be extremely rare or nearly impossible for one human to possess all those skills at once. · no date given · not yet resolved

“So just like the AI is actually getting out of control to concentration of power. So like”

Expects: AI-driven concentration of power will be a major problem on the horizon. · no date given · not yet resolved · In a shared prediction of 4 →

“i hope not we want we want it to be the last one um uh yeah then there's risk of sort of of conflict over ai um so you know in particular i think we're worried You know, we're worried about literal World War Three where countries,”

Expects: The speaker predicts that competition over AI could lead to conflict, potentially including a literal World War Three. · no date given · not yet resolved

“to be really helpful for solving these problems,”

Expects: The speaker hopes human-level AIs will be capable enough to help solve the identified problems. · no date given · not yet resolved

“our story in AI 2040 is basically between 2030 and 2040 we're going to we're going to do this buying time we're going to rely on control we're going to have the human level AIs we're going to use those human level AIs to make a bunch of progress on alignment but also on other things and then by 2040 in the story we've made enough alignment progress where it's like hey okay we really don't need to rely on control therefore we can scale to these like sort of wildly superhuman AIs where we're really relying on alignment and if we were to try to control them Both of them would totally fail because if they were misaligned, they would be able to like completely subvert our control measures.”

Expects: In the speaker's scenario, control would fail against wildly superhuman AI if that AI were misaligned, because it could completely subvert control measures. · no date given · not yet resolved

“our story in AI 2040 is basically between 2030 and 2040 we're going to we're going to do this buying time we're going to rely on control we're going to have the human level AIs we're going to use those human level AIs to make a bunch of progress on alignment but also on other things and then by 2040 in the story we've made enough alignment progress where it's like hey okay we really don't need to rely on control therefore we can scale to these like sort of wildly superhuman AIs where we're really relying on alignment and if we were to try to control them Both of them would totally fail because if they were misaligned, they would be able to like completely subvert our control measures.”

Expects: In the speaker’s AI 2040 scenario, people use human-level AI and control from 2030 to 2040 while advancing alignment, and by 2040 alignment progress is sufficient to scale to wildly superhuman AI that relies on alignment rather than control. · 1 Jan 2030 – 31 Dec 2040 · not yet resolved · In a shared prediction of 16 →

“I think it's going to get harder because the models will just be. there will be a bunch of properties the models will have which will make it much harder to align them in particular so we're already seeing sort of growth in situational awareness where like the ai is a few years ago we're like kind of not that situationally aware and then now i think they're like pretty good at you know recognizing like for example our friends at redwood uh um you know they often do sort of these control evals and the models often you know in the evaluations where they're like trying to break out of these control setups”

Expects: Aligning AI models will probably become harder as the models grow more sophisticated. · no date given · not yet resolved · In a shared prediction of 4 →

“And so one of our core hopes is that basically with more transparency and more sort of public understanding into what's going on in the ag companies,”

Expects: The speakers hope that greater transparency and public understanding of AI companies’ activities will reduce the burden on regulators. · no date given · not yet resolved