Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC

“We have raised $40 million by Ribbit Capital and First Amarna.”

Claim: AIUC raised $40 million from Ribbit Capital and First Amarna.

“When we raised that seed round, we had a hypothesis that... is that at some point risk was going to hold down adoption at that point in time that felt kind of hypothetical and i think that that is now over clearly the moment is now with methos and with fable it's pretty obvious that literally the binding constraint on adoption is risk and so for us it feels like this is a natural continuation of the same hypothesis but where previously it was speculation now it feels like fact”

Claim: AIUC hypothesized at the time of its seed round that risk would eventually constrain adoption.

“is that at some point risk was going to hold down adoption at that point in time that felt kind of hypothetical and i think that that is now over clearly the moment is now with methos and with fable it's pretty obvious that literally the binding constraint on adoption is risk and so for us it feels like this is a natural continuation of the same hypothesis but where previously it was speculation now it feels like fact”

Claim: The speaker says that risk is now the binding constraint on adoption, pointing to Mythos and Fable.

“is that at some point risk was going to hold down adoption at that point in time that felt kind of hypothetical and i think that that is now over clearly the moment is now with methos and with fable it's pretty obvious that literally the binding constraint on adoption is risk and so for us it feels like this is a natural continuation of the same hypothesis but where previously it was speculation now it feels like fact”

Claim: The speaker says AIUC's hypothesis that risk would constrain adoption has shifted from speculation to something that now feels factual.

“so we are now working with folks like Cursor, Harvey, Lovable, Eleven Labs.”

Claim: AIUC works with Cursor, Harvey, Lovable, and Eleven Labs.

“Yeah, late 2021, I sold a company, my first company, an edtech company.”

Claim: The speaker sold his first company, an edtech company, in late 2021.

“company i had a bit of time to think about what was next i came across the scaling laws paper and that just struck me like lightning i was just like this is a big idea in short the scaling laws paper just says the bigger the model the smarter the model and exactly”

Claim: After selling his company, the speaker encountered the scaling laws paper and found it profoundly striking.

“company i had a bit of time to think about what was next i came across the scaling laws paper and that just struck me like lightning i was just like this is a big idea in short the scaling laws paper just says the bigger the model the smarter the model and exactly”

Claim: The speaker describes the scaling laws paper as asserting that larger models are smarter.

“the cabin one and the important thing that clicked for me there was oh now capital will understand this if you put in more money you get more money out and so that will kick off a hype cycle and so you'll actually kind of you'll get a sense of predictable returns which is in fact played out and so i just pack my bags i've never been to san francisco i just pack my bags throughout here to find the people who've written it and at the time they had just started a small lab called anthropic and there was like 40 people at the time or so”

Claim: The speaker believed the scaling-laws argument would help capital recognize that investing more could yield more in return.

“oh now capital will understand this if you put in more money you get more money out and so that will kick off a hype cycle and so you'll actually kind of you'll get a sense of predictable returns which is in fact played out and so i just pack my bags i've never been to san francisco i just pack my bags throughout here to find the people who've written it and at the time they had just started a small lab called anthropic and there was like 40 people at the time or so”

Claim: The speaker says the anticipated AI investment dynamic, including a hype cycle and predictable returns, has played out.

“oh now capital will understand this if you put in more money you get more money out and so that will kick off a hype cycle and so you'll actually kind of you'll get a sense of predictable returns which is in fact played out and so i just pack my bags i've never been to san francisco i just pack my bags throughout here to find the people who've written it and at the time they had just started a small lab called anthropic and there was like 40 people at the time or so”

Claim: Anthropic had recently begun operating as a small lab with about 40 people when the speaker visited.

“oh now capital will understand this if you put in more money you get more money out and so that will kick off a hype cycle and so you'll actually kind of you'll get a sense of predictable returns which is in fact played out and so i just pack my bags i've never been to san francisco i just pack my bags throughout here to find the people who've written it and at the time they had just started a small lab called anthropic and there was like 40 people at the time or so”

Claim: The speaker expected increased investment in AI to trigger a hype cycle and create a perception of predictable returns.

“until I eventually got introduced to Dario and at the time they were wrestling with some of these questions of like should we deploy our models should we make revenue how should we engage with the rest of the world they're just broken off from OpenAI and it's been publicly reported that they were kind of concerned with how they were dealing with deployment so they were wrestling with some of those questions at this point this is like early form of war like early 2022 the sexiest product at the time was like Jasper”

Claim: The speaker says Anthropic had concerns about model deployment, a concern he notes had been publicly reported.

“until I eventually got introduced to Dario and at the time they were wrestling with some of these questions of like should we deploy our models should we make revenue how should we engage with the rest of the world they're just broken off from OpenAI and it's been publicly reported that they were kind of concerned with how they were dealing with deployment so they were wrestling with some of those questions at this point this is like early form of war like early 2022 the sexiest product at the time was like Jasper”

Claim: Anthropic had broken off from OpenAI by early 2022.

“until I eventually got introduced to Dario and at the time they were wrestling with some of these questions of like should we deploy our models should we make revenue how should we engage with the rest of the world they're just broken off from OpenAI and it's been publicly reported that they were kind of concerned with how they were dealing with deployment so they were wrestling with some of those questions at this point this is like early form of war like early 2022 the sexiest product at the time was like Jasper”

Claim: By early 2022, Anthropic was considering whether to deploy its models, generate revenue, and engage with the rest of the world.

“until I eventually got introduced to Dario and at the time they were wrestling with some of these questions of like should we deploy our models should we make revenue how should we engage with the rest of the world they're just broken off from OpenAI and it's been publicly reported that they were kind of concerned with how they were dealing with deployment so they were wrestling with some of those questions at this point this is like early form of war like early 2022 the sexiest product at the time was like Jasper”

Claim: The speaker says Jasper was the most exciting product around in early 2022.

“there's there's nothing out there so where is value going to accrue uh what are going to be the different parts of the stack where all open questions yes”

Claim: In early 2022, the speaker saw the AI landscape as lacking established products, with the location of value and the structure of the technology stack still open questions.

“Yeah, I think I'd actually ever since AlphaGo. had had some appreciation that AI was a big deal but it kind of felt it raised all these kind of interesting philosophical questions but it's kind of not clear from afar where exactly that would go but it was obvious enough that it was like this is going to be a big thing if we find the kind of right mechanism to kind of get the techno capital machine to work on this but it was just not clear and so I think there's some way in which like”

Claim: Since AlphaGo, the speaker had regarded AI as a significant development.

“had had some appreciation that AI was a big deal but it kind of felt it raised all these kind of interesting philosophical questions but it's kind of not clear from afar where exactly that would go but it was obvious enough that it was like this is going to be a big thing if we find the kind of right mechanism to kind of get the techno capital machine to work on this but it was just not clear and so I think there's some way in which like”

Claim: The speaker believed AI could become a major development if the right mechanism mobilized capital and technology around it.

“had had some appreciation that AI was a big deal but it kind of felt it raised all these kind of interesting philosophical questions but it's kind of not clear from afar where exactly that would go but it was obvious enough that it was like this is going to be a big thing if we find the kind of right mechanism to kind of get the techno capital machine to work on this but it was just not clear and so I think there's some way in which like”

Claim: Before the scaling-laws insight, the speaker found AI's direction unclear from afar, even though it raised interesting philosophical questions.

“a philosophy and economics background, it felt like if this turns out to be true, you're going to be wrestling with all of the big questions in society.”

Claim: The speaker believed that if AI's promise proved true, it would raise major societal questions, viewed from his philosophy and economics background.

“Everything you've learned about politics gets thrown out of the window.”

Claim: The speaker believed that if AI's promise proved true, established political understanding would be radically disrupted.

“Everything you've learned about economics at least gets challenged.”

Claim: The speaker believed that if AI's promise proved true, established economic understanding would at least be challenged.

“And so what felt interesting was to be at that frontier that has just ramifications across everything.”

Claim: The speaker found working at the AI frontier interesting because AI has ramifications across many domains.

“what I think team anthropic has thought more about than anyone was like what are the implications of this if you really play this out and back then they had kind of vision documents for what the world would look like in 2026 and they were kind of in vivid detail playing out how much compute is going to be needed what is the cap it's going to look like what are going to be some of the kind of societal concerns but also what is the amount of economic value coming out here and so it kind of felt like they”

Claim: Anthropic's vision documents projected in vivid detail what the world might look like in 2026, covering compute requirements, the cap's form, societal concerns, and economic value.

“what I think team anthropic has thought more about than anyone was like what are the implications of this if you really play this out and back then they had kind of vision documents for what the world would look like in 2026 and they were kind of in vivid detail playing out how much compute is going to be needed what is the cap it's going to look like what are going to be some of the kind of societal concerns but also what is the amount of economic value coming out here and so it kind of felt like they”

Claim: The speaker believes Anthropic had considered AI's implications more deeply than anyone else.

“held a crystal ball that in hindsight turned out to just be dramatically correct and they weren't holding it like they were obviously correct they're just like take this hypothesis really really seriously and”

Claim: The speaker says Anthropic presented its AI hypothesis as something to examine seriously, rather than claiming it was obviously correct.

“held a crystal ball that in hindsight turned out to just be dramatically correct and they weren't holding it like they were obviously correct they're just like take this hypothesis really really seriously and”

Claim: The speaker says Anthropic's earlier projections proved dramatically accurate in hindsight.

“And that's now a couple of years old, but also people keep referencing these particular weeks with Fable and Mythos and it's like, wow, if you take this one idea seriously, the scaling laws, a lot of things fall into place.”

Claim: Taking the idea of scaling laws seriously makes many observations about AI development fall into place, according to the speaker.

“And that's now a couple of years old, but also people keep referencing these particular weeks with Fable and Mythos and it's like, wow, if you take this one idea seriously, the scaling laws, a lot of things fall into place.”

Claim: The speaker says that people still refer to particular weeks involving Fable and Mythos, and that taking scaling laws seriously helps explain many developments.

“deep conviction in in again in this like big if you take a big blob of compute data it just wants to learn and out of that will come smarter and smarter models and all the particulars are not clear and yeah. all the implications were not clear but that deep conviction is like core thesis and that was kind of dizzying it's both phenomenally interesting and exciting and also”

Claim: The speaker describes a core thesis that large amounts of compute and data will produce increasingly intelligent models, while acknowledging that the details and implications are unclear.

“very quickly you get to like the world we know today will no longer be if this hypothesis holds so it also just felt like important in some kind of grand sense”

Claim: If the hypothesis about AI's development holds, the world as it is known today will change fundamentally very quickly.

“there it was kind of remarkably like what it looks like on the outside today extremely cohesive extremely mission oriented”

Claim: During the speaker's time at Anthropic, the organization was remarkably cohesive and mission-oriented.

“and living in this tension between their two ideas which is ai could both go really well and really bad and we want to be part of building it that creates astounding amounts of tension”

Claim: The Anthropic team experienced tension between the possibility that AI could have very positive or very negative outcomes and its desire to participate in building AI.

“And they were wrestling with this incentive challenge where they know there's a race that they're in where you might get forced to cut corners, but it also felt very important to them to be at the forefront of technology.”

Claim: Anthropic faced competitive pressure that might force it to cut corners, while its members also considered staying at the technological forefront important.

“They have really stuck to their guns. There's a core set of beliefs that they hold more deeply than most companies hold any beliefs.”

Claim: Anthropic has remained committed to a core set of beliefs that the speaker considers more deeply held than the beliefs of most companies.

“AIUC builds confidence infrastructure for frontier AI through standards and insurance.”

Claim: AIUC builds confidence infrastructure for frontier AI by using standards and insurance.

“Already back then, early 2022, Waymo's were in some ways like AGI for cars, like they were superhuman drivers, but you couldn't take one to the airport.”

Claim: By early 2022, the speaker viewed Waymo vehicles as superhuman drivers in some respects, but said they could not be taken to the airport.

“And now, 400 years later, you still can't take a Waymo to the airport, despite now everyone having kind of looked at the evidence and being like they're better drivers than humans. So in that particular instance,”

Claim: At the time of the conversation, Waymo vehicles still could not be taken to the airport, despite the speaker's assertion that the evidence showed they were better drivers than humans.

“what's clear is that the binding constraint... on AI being useful is not capability, but it's that liability or risk or trust.”

Claim: The main constraint on AI's usefulness is liability, risk, or trust rather than technical capability.

“That problem is general.”

Claim: The challenge of liability, risk, and trust as constraints on AI usefulness applies broadly.

“The reason why right now Fable is not open for access is not because it's not a good model. It's because it's a very good model. It's just hard to make promises about what it will or will not do.”

Claim: Fable is not open for access because it is a very capable model, making it difficult to promise what it will or will not do.

“And this problem gets worse. As AI gets better, basically more intelligent AI can be more autonomous. That's more valuable, but also the risk surface grows.”

Claim: As AI becomes more capable, it can become more autonomous and valuable, but its risk surface also grows, worsening the challenge of making reliable promises about its behavior.

“And so what Waymo illustrates is that unless you build the confidence infrastructure to make promises about AI or at least bring light to the risks, you grind adoption to halt.”

Claim: Without confidence infrastructure that enables credible promises about AI or makes its risks visible, AI adoption can grind to a halt.

“governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it.”

Claim: Governments, banks, hospitals, and militaries need to understand what AI will and will not do before they can operate with and incorporate it.

“If you trace this problem back through history, every technology wave has had some version of this problem.”

Claim: Every major technology wave has faced a version of the problem of establishing confidence in the technology.

“So if you go back to like year 1900, electricity comes out. Cars burn down. houses burn down. Lots of people die.”

Claim: The speaker cites early electricity as a technology wave associated with house fires and many deaths.

“1930s, cars are a big deal. Kill lots of people.”

Claim: The speaker says that cars became significant in the 1930s and caused many deaths.

“50s, private nuclear energy is a big deal, poses big risks.”

Claim: Private nuclear energy became a major development in the 1950s and posed significant risks, according to the speaker.

“In each of those instances, the market runs ahead of regulation to create confidence infrastructure because that's required to make go-no-go decisions. It's required for adoption and the market fundamentally wants adoption.”

Claim: Markets develop confidence infrastructure ahead of regulation because it enables go/no-go decisions and adoption, and markets fundamentally seek adoption.

“And in all of those instances, common blueprint emerges between standards and insurance.”

Claim: Across the technology examples discussed, a common confidence-building blueprint combines standards with insurance.

“The reason these components is standards kind of provide the rules of the road and they also specify like what are the tests that need to be run so we can get a sense of how high the risk is. So taking the case of cars,”

Claim: Standards establish rules of the road and specify tests that help determine the level of risk.

“it's like a car crash. they inform your insurance pricing today, they inform your purchasing decisions,”

Claim: Car-crash test results inform insurance pricing and consumers' purchasing decisions.

“That's basically the risk framework. The insurers are important because they pick up the bill so they are the private institution that is most on the side of and is best incentivized to quantify the risk truthfully and then figure out all the ways to reduce the risk because that increases their profit so they're basically they help shape the incentives and these two work really well in unison now”

Claim: Because insurers bear the costs of claims, they are strongly incentivized to quantify risk accurately and reduce it, which increases their profits.

“Frontier companies, some of our customers today like Cursor, Sierra, Eleven Labs, Harvey, are going to have a very easy time selling a pilot to a bank. The demo just sells itself. It's magic.”

Claim: Frontier AI companies such as Cursor, Sierra, ElevenLabs, and Harvey can readily sell pilot projects to banks because their demonstrations are compelling.

“But bringing that through, if you want to do a wall-to-wall rollout at a bank or a hospital, you have to go through the risk process.”

Claim: A wall-to-wall deployment of AI at a bank or hospital requires passing through the organization's risk process.

“These banks have no idea, even which questions to ask, let alone which answers are sufficient.”

Claim: Banks lack clarity about which questions to ask when assessing AI risk and what answers would be sufficient.

“let alone like how do they go and test whether these agents actually work the way they're supposed to and so they have this problem of like what can we say to earn the trust and we think there's like a golden sentence that goes something like hey i hear you're really worried about hallucinations or jailbreaks or whatever it may be we've had an independent third party test us against the gold standard we pass with flying colors and as a vote of confidence the world's most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet so if something does go There's wrong exactly”

Claim: A proposed way to build customer trust is to show that an independent third party tested an AI system against a gold standard and that the system passed.

“let alone like how do they go and test whether these agents actually work the way they're supposed to and so they have this problem of like what can we say to earn the trust and we think there's like a golden sentence that goes something like hey i hear you're really worried about hallucinations or jailbreaks or whatever it may be we've had an independent third party”

Claim: Banks lack established methods for testing whether AI agents behave as intended.

“test us against the gold standard we pass with flying colors and as a vote of confidence the world's most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet so if something does go There's wrong exactly”

Claim: An insurer's willingness to assume some of the risk of an AI system can signal confidence in that system.

“Yeah, Rajiv is actually my soon-to-be brother-in-law. so I'm actually in a week and a half getting married to Rajiv's sister.”

Claim: The speaker said he was due to marry Rajiv's sister in a week and a half, making Rajiv his soon-to-be brother-in-law.

“So Rajiv and I have known each other for a decade.”

Claim: The speaker and Rajiv had known each other for ten years.

“made a funny story i met both rajiv and his sister henna at the same time when henna and i were interns at mckinsey in london and rajiv was assigned as my mentor and so met them at the same time for a long time it was not obvious that we were necessarily going to work together i was in startups he was an insurance partner at mckinsey three or four years ago i think henna convinced him that ai was going to be a really”

Claim: The speaker first met Rajiv and Rajiv's sister Henna while the speaker and Henna were interns at McKinsey in London; Rajiv was the speaker's assigned mentor.

“made a funny story i met both rajiv and his sister henna at the same time when henna and i were interns at mckinsey in london and rajiv was assigned as my mentor and so met them at the same time for a long time it was not obvious that we were necessarily going to work together i was in startups he was an insurance partner at mckinsey three or four years ago i think henna convinced him that ai was going to be a really”

Claim: The speaker thinks Henna convinced Rajiv, three or four years earlier, that AI would become a major development.

“made a funny story i met both rajiv and his sister henna at the same time when henna and i were interns at mckinsey in london and rajiv was assigned as my mentor and so met them at the same time for a long time it was not obvious that we were necessarily going to work together i was in startups he was an insurance partner at mckinsey three or four years ago i think henna convinced him that ai was going to be a really”

Claim: For a long time, the speaker and Rajiv did not seem likely to work together: the speaker was working in startups, while Rajiv was an insurance partner at McKinsey.

“big saying and so he quit his job cushy partner job at mckinsey in london pack his bags flew to san francisco and ended up joining meter you guys COO.”

Claim: After leaving his partner role at McKinsey in London, Rajiv moved to San Francisco and joined Meter as COO.

“if you've seen the them. the the chart of uh the horizons of the tasks that agents can take on is doubling extremely fast so he went there led their partnerships with anthropic and open ai”

Claim: Rajiv led Meter's partnerships with Anthropic and OpenAI.

“if you've seen the them. the the chart of uh the horizons of the tasks that agents can take on is doubling extremely fast so he went there led their partnerships with anthropic and open ai”

Claim: The speaker said the task horizons that AI agents can handle are expanding at an extremely fast, doubling pace.

“to test their models before release but also working closely with the US and UK government to figure out like how do you know whether a model can be released and in some ways that's like the perfect background you spend a lot of time in insurance knows that world spend a lot of time with frontier testing of models and so when I was bumbling around this idea space starting with some of the ideas we talked about related to Waymo as soon as we got into the content we're both like Oh, this would be an amazing business to build together.”

Claim: The speaker viewed Rajiv's experience in insurance and frontier-model testing as an ideal background for building their business together.

“to test their models before release but also working closely with the US and UK government to figure out like how do you know whether a model can be released and in some ways that's like the perfect background you spend a lot of time in insurance knows that world spend a lot of time with frontier testing of models and so when I was bumbling around this idea space starting with some of the ideas we talked about related to Waymo as soon as we got into the content we're both like”

Claim: Rajiv tested Anthropic and OpenAI models before release and worked with the US and UK governments to develop ways to assess whether models should be released.

“This is like wrestling with the problem that we both think is the most important in the world from a market angle, which is kind of our intuitions is that.”

Claim: The speaker and Rajiv believed their business was addressing the world's most important problem from a market perspective.

“the market can do a lot and the faster AI moves the harder it is for government to solve some of these problems and then took a time to work through what is it like to work with family and yeah”

Claim: The speaker and his cofounder's intuition was that markets can address many AI-related problems, while faster AI progress makes some of those problems harder for governments to solve.

“back then there was we felt like we were a family so starting a business together felt like kind of a big step and here we are with just immense amounts of trust”

Claim: The speaker said starting a company with Rajiv felt like a significant step, and that they now have a great deal of trust in each other.

“There is just 20 of us now.”

Claim: The speaker's company had 20 people at the time of the conversation.

“As it says right at the top, at AZ1 is the standard for agent security, safety and reliability.”

Claim: AIUC1 is a standard for the security, safety, and reliability of AI agents.

“The fundamental design principle is take all of the concerns that slow down adoption. So all the questions, all the fears that keep security leaders and Fortune 1000 up at night and put them into one comprehensive framework.”

Claim: AIUC1's design principle is to consolidate adoption-related concerns and security fears into one comprehensive framework.

“So one of the concerns with security standards that often feel kind of like theater paperwork is that they don't actually ground out in does any of this work? Does any of this matter?”

Claim: The speaker said security standards may amount to performative paperwork if they are not grounded in technical tests of whether their controls work or matter.

“conviction from early on that that was going to be the kind of crux was to pass this you must get tested every quarter basically run thousands of simulations to see also can it actually be jailbroken how hard is it to jailbreak how often does it hallucinate how often does it delete data etc”

Claim: The speaker's early conviction was that certification should require quarterly testing involving thousands of simulations to measure jailbreakability, jailbreak difficulty, hallucination frequency, and data deletion.

“if you scroll up to the top here, is to refresh it quarterly.”

Claim: AIUC1 is intended to be refreshed every quarter.

“So the core trait of AI is that it moves extremely fast.”

Claim: The speaker characterized AI as changing or advancing extremely quickly.

“Whatever concerns we're discussing today were not the same ones three months ago. And this will keep changing.”

Claim: AI-related concerns change over time: the concerns at the time of the conversation differed from those three months earlier, and the speaker expected them to continue changing.

“Typically, standards update on like a decade cycle is obviously not going to work.”

Claim: The speaker argued that standards updated only about once a decade are unsuitable for the rapidly changing AI context.

“And the core thing here was to... They usually get the risk leaders of the Fortune 1000 around the table.”

Claim: The speaker said the standard-development process brings Fortune 1000 risk leaders together for discussion.

“you'll see AI1 Consortium. The consortium is a group of risk leaders who run real banks, real hospitals, real critical infrastructure who are facing these challenges every day.”

Claim: The AIUC1 Consortium consists of risk leaders from banks, hospitals, and critical-infrastructure organizations who encounter AI-related challenges in their work.

“And we meet with these folks twice a quarter.”

Claim: The speaker's organization meets with AIUC1 Consortium participants twice per quarter.

“what's top of mind? What is keeping them up at night? There's a tremendous amount of desire for that conversation.”

Claim: The speaker said consortium participants have strong demand for discussions about their current concerns.

“And then we operationalize that into a specific standard that gets into... Actually, we can go into and look at what even is the standard.”

Claim: The speaker's organization turns consortium discussions about current concerns into specific standard requirements.

“So if you take something like hallucinations, hallucinations system reliability, there is a number of requirements here. If you go into the top one, prevent hallucinated requirements, hallucinated outputs. This is one particular requirement. This is a technical control.”

Claim: AIUC1 includes technical controls for system reliability, including a requirement intended to prevent hallucinated outputs.

“Basically, we want some kind of randomness filter.”

Claim: The speaker said the technical control should use some kind of randomness filter to address hallucinated outputs.

“The first thing you see here is what's called a crosswalk.”

Claim: The standard's documentation includes a crosswalk.

“So every one of their grandmother has put out a framework, very high level framework for what are the AI risks.”

Claim: The speaker said many organizations have published high-level frameworks outlining AI risks.

“some ways our competition, in fact friends with them. We'll come back”

Claim: The speaker said organizations that publish AI-risk frameworks are competitors in some respects, but his organization is also friendly with them.

“And the reason we go this deep is that there's actually not that much confusion about what are the big concerns in AI. Everyone agrees to these. The question like, what are you actually supposed to do?”

Claim: The speaker says there is relatively little disagreement about the major concerns in AI, while the practical question is what actions people should take.

“What we find a lot of demand for is getting down to the specific evidence that people need to look for.”

Claim: The speaker reports substantial demand for guidance on the specific evidence people should look for when assessing AI-related controls.

“you won't believe the amount of time a risk leader has asked for the IP rights to the underlying model.”

Claim: The speaker says risk leaders have sometimes requested intellectual-property rights to the underlying AI model.

“it see exactly put that in the question now all right so that's kind of what a standard is and we update this every quarter with these folks to keep up with the latest concerns yeah”

Claim: The speaker says the standard is updated quarterly to reflect the latest concerns.

“yes yes yes yes so if you go into I don't know the left-hand side so actually before we go in there there are three types of requirements the first is technical controls like you must implement some guardrails Two, there are test controls. And then three, there are policy controls.”

Claim: The speaker categorizes the requirements into technical controls, test controls, and policy controls.

“yes yes yes yes so if you go into I don't know the left-hand side so actually before we go in there there are three types of requirements the first is technical controls like you must implement some guardrails”

Claim: The speaker says technical requirements can require an organization to implement guardrails.

“there are test controls. So you must have an independent third party go and run some tests against you. I'll show you one of those in a second.”

Claim: The speaker says test controls require an independent third party to conduct tests against the organization.

“For example, you must have a person whose name is on the line when you guys fuck up and you must have a plan for how you tell your customers and how you engage with them.”

Claim: The speaker says policy controls require a named accountable person and a plan for notifying and engaging customers when the organization fails.

“they're kind of more traditional standard type stuff so in this particular instance we just check whether they in fact have a grounders filter so we will partner with an auditor so we partner with auditors like kpmg or like shellman who go in and do the thing auditors do which is to check the evidence in this case that might be a screenshot it might be a part of the code that they need to review to see that it actually just that it exists and then”

Claim: The speaker says an auditor checks evidence, such as screenshots or code, to verify that a required groundness filter exists; this check does not establish whether the filter is effective.

“to the third-party testing for hallucinations out on the left. That's basically the next requirement. This is where we test how well does it actually work.”

Claim: The speaker says a separate requirement tests system performance through third-party testing for hallucinations.

“we test them we test them yeah”

Claim: The speaker says their organization conducts the system tests itself.

“the end-to-end almost always is dependent on like our customers need to build something for us it takes somewhere between like three to ten weeks depending on how up to snuff they already are so some people show up to us with like extremely rigorous security programs When we test them, it works extremely well. We can get that done very quick.”

Claim: The speaker says customers with rigorous security programs can complete testing quickly, and that testing works extremely well for them.

“the end-to-end almost always is dependent on like our customers need to build something for us it takes somewhere between like three to ten weeks depending on how up to snuff they already are so some people show up to us with like extremely rigorous security programs”

Claim: The speaker says end-to-end certification usually takes three to ten weeks, depending on the customer's existing readiness.

“Some people come to us and they're not that far along. We give them kind of the spec that they need to build towards and then their security teams and engineers get to work and build to meet the standard.”

Claim: The speaker says customers who are less prepared receive a specification, after which their security teams and engineers build toward meeting the standard.

“The testing itself typically takes a couple of weeks, including the time for them to remediate.”

Claim: The speaker says the testing phase typically takes a couple of weeks, including remediation time.

“Often we'll find something that we cannot pass where this is actually just not up to the standard. You won't pass the standard and then they will need to go and... implement additional safeguards or additional remediation that makes them more robust so that they can actually kind of hand on heart look at their customers in the eyes and say like hey we've done truly our very best correct”

Claim: The speaker says assessments often uncover failures to meet the standard, after which customers need additional safeguards or remediation to improve robustness.

“some of the changes have just been ages aren't just one thing so like if you take agents like cursor and compare them to sierra they're really quite different compared to harvey again compare them to ui path again”

Claim: The speaker says AI agents vary substantially across products, citing Cursor, Sierra, Harvey, and UiPath as examples.

“labs they're all quite different and so we wanted to design a standard that works for”

Claim: AIUC aimed to create a standard applicable across different types of AI agents because labs differ considerably.

“all of the types of agents and we started with one that was like pretty text-based like honestly pretty customer support focused that's where there's a lot of existing demand and then over time I've picked some of the frontier companies in each of these other domains that we could work with and build up the standards so such that we know that the same standard works for code that works for customer support works for automation etc so that's been one big thing yeah then some of the things that have been top of mind recently”

Claim: AIUC has worked with frontier companies across domains to develop agent standards intended to apply to coding, customer support, and automation.

“all of the types of agents and we started with one that was like pretty text-based like honestly pretty customer support focused that's where there's a lot of existing demand and then over time I've picked some of the frontier companies in each of these other domains that we could work with and build up the standards so such that we know that the same standard works for code that works for customer support works for automation etc so that's been one big thing yeah then some of the things that have been top of mind recently”

Claim: AIUC initially focused its agent standard on text-based customer support, where substantial demand already existed.

“METHOS is bringing up a lot of concerns for security leaders.”

Claim: METHOS is prompting substantial concern among security leaders.

“We're starting to get more and more questions around agent-to-agent interactions.”

Claim: AIUC is receiving increasingly frequent questions about interactions between AI agents.

“It's very nascent at the moment, but it's starting to emerge.”

Claim: Interactions between AI agents remain very nascent but are beginning to emerge.

“There's been a lot of questions related to OpenClone and MCP. Again,”

Claim: AIUC has received many questions related to OpenClone and MCP.

“like agents starting to interact with each other is really top of mind.”

Claim: Interactions between AI agents are a major current concern for the speaker.

“Then as coding agents have really taken off, that's also where banks and hospitals, etc. are getting more and more precise on what it is they need. So really dialing it in as it starts to be like where most of the tokens flow through in the world, getting much sharper on that.”

Claim: With the rise of coding agents, banks and hospitals are specifying their needs more precisely.

“The most important thing is that a lot of companies have not done a serious stress test.”

Claim: Many companies have not conducted serious stress tests of their AI agents or systems.

“They spend most of the time, perhaps rightly so, optimizing for how does it work in the good case, the average case. How high quality is the output for the customer?”

Claim: Companies often focus most of their effort on system performance and customer-output quality in good and average cases, a focus the speaker says may be justified.

“And a lot of these companies are pretty new, so they haven't spent a lot of time stress testing the what is what is an adversary on the other side? What are some of the complicated corner cases that you've not really considered?”

Claim: Many newer companies have spent little time testing how their systems handle adversarial users and complicated edge cases.

“It often takes a while until they hire their first security person and there's a whole different kind of risk surface than just building a good product.”

Claim: Startups face security and other risks beyond the challenge of building a good product.

“It often takes a while until they hire their first security person and there's a whole different kind of risk surface than just building a good product.”

Claim: Startups often delay hiring their first security employee.

“the right kind of architecture most of them will have some kind of guardrails in place either something coming out of the box from their model provider or they'll have built their own filters to sit in between they just don't work very well the difference between putting a classifier in place that like maybe goes and checks whether you're giving medical advice when you shouldn't and says hey if this looks like medical advice filter it out lots of companies have that in place the question is whether it works and it actually is pretty fiddly”

Claim: Most companies have guardrails supplied by model providers or built in-house, but these safeguards often do not work well.

“the right kind of architecture most of them will have some kind of guardrails in place either something coming out of the box from their model provider or they'll have built their own filters to sit in between they just don't work very well the difference between putting a classifier in place that like maybe goes and checks whether you're giving medical advice when you shouldn't and says hey if this looks like medical advice filter it out lots of companies have that in place the question is whether it works and it actually is pretty fiddly”

Claim: Companies may use classifiers to detect and filter AI responses that provide medical advice when they should not.

“To sit down and think about all the ways in which you could ask for medical advice, read the academic literature on what are the kinds of... framing or tricks you might play to get an AI to give you medical advice when you really shouldn't and so there's like an error of expertise that's just missing so what we find is that most people have the right billing box in place and it doesn't it's not rocket science but the finicky thing is like getting into the corners and testing whether it works such that you can look your customers in the eye who may be a bank or maybe a hospital and be like this is this is going to work for you”

Claim: The difficult part of guardrails is thoroughly testing whether they work in edge cases.

“framing or tricks you might play to get an AI to give you medical advice when you really shouldn't and so there's like an error of expertise that's just missing so what we find is that most people have the right billing box in place and it doesn't it's not rocket science but the finicky thing is like getting into the corners and testing whether it works such that you can look your customers in the eye who may be a bank or maybe a hospital and be like this is this is going to work for you”

Claim: Many companies have basic safeguards in place, but must thoroughly test edge cases before they can confidently assure bank and hospital customers that their systems will work for them.

“framing or tricks you might play to get an AI to give you medical advice when you really shouldn't and so there's like an error of expertise that's just missing so what we find is that most people have the right billing box in place and it doesn't it's not rocket science but the finicky thing is like getting into the corners and testing whether it works such that you can look your customers in the eye who may be a bank or maybe a hospital and be like this is this is going to work for you”

Claim: Users can try framing or other tricks to elicit medical advice from an AI in situations where it should not provide that advice.

“the models are going to present risk in the national security category.”

Claim: AI models are expected to present national-security risks.

“You have some people that are on the hook if something goes wrong. In the case of agents, it's often the security leaders and the enterprise. In this case, it's the government.”

Claim: Security leaders and enterprises are often accountable when agent-related problems occur, while the government is accountable for model-related risks.

“They don't have necessarily spent their entire lives thinking about what are the new risks that come here?”

Claim: Government officials may lack extensive prior experience thinking about emerging AI-model risks.

“But they do have to make sure that their concerns are addressed.”

Claim: Governments must ensure that their concerns about AI risks are addressed.

“You have some frontier AI companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive. to not always be true.”

Claim: Frontier AI companies have substantial technical expertise and risk knowledge, but their incentives can lead them not always to be truthful.

“So you have a trust gap between the government and the labs.”

Claim: A trust gap exists between governments and AI labs.

“And in every other industry, you end up with some kind of body sitting between a neutral third party sitting between those people. There's no other industry where you allow people to audit themselves.”

Claim: Other industries use neutral third parties between stakeholders rather than allowing organizations to audit themselves.

“So there's going to be a need for a third party that can take the rigor of the labs to run frontier technical evals, but can also speak. Legible trusts in the way that the government trusts PwC to go and run financial audits and they know that they output audit reports in a way that's consistent, that's easy to read, that's factual, that's trustworthy.”

Claim: A third-party evaluator is needed to conduct rigorous frontier AI technical evaluations and communicate their results to governments in a legible and trustworthy form.

“Those two things need to be brought together and what we've learned from our work with agents is that if you want that communication between those two parties to be smooth, there has to be one common standard that is public that people can inspect.”

Claim: Smooth communication between governments and AI labs requires a single public, inspectable standard.

“What are the risks that matter within each of these risks? What are the kind of threat models that you're really looking for? You need to specify for each of those risks, what are the guardrails that need to be in place? And what are the tests they need to run to see whether those guardrails are effective?”

Claim: AI risk assessments should specify threat models, required guardrails, and tests for whether those guardrails work.

“And then you need to go and run audits that are technical audits that are consistent. So if you're trying to bring trust, it's extremely important that you methodically work your way through the risks.”

Claim: Consistent technical audits and methodical examination of risks are important for building trust.

“You can't send one researcher in and say like, come back with whatever you find. You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try. and therefore the kinds of promises you can and cannot make at the end of it.”

Claim: AI audit reports should document what evaluators did and did not test, enabling appropriately scoped conclusions and promises.

“I think of Fable as a direct symptom of this problem that the government was told that there's a risk. The government may struggle to assess just how big that risk is.”

Claim: Governments may have difficulty assessing the magnitude of an AI risk once informed that the risk exists.

“And we think that a broker. It is the most natural solution.”

Claim: The speaker considers a broker the most natural solution to the trust problem between governments and AI labs.

“In other markets, you see something like in financial markets, you see Moody's. Moody's goes in and they look at a bond and they output a rating. They say like, here's the evidence we found. Here's the rating. We don't decide whether anyone should buy this bond or not buy this bond. Well, that depends on their risk appetite.”

Claim: Moody’s provides bond ratings and supporting evidence but leaves investment decisions to buyers’ risk appetites.

“But we do provide this common information layer that everyone can rely on.”

Claim: A neutral rating or evaluation intermediary can provide a shared information layer that stakeholders can rely on.

“point to them and say hey pension funds you should probably really take care you shouldn't risk your pensioners money so you can only invest in AAA rated bonds that means that now the government doesn't have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are”

Claim: Using a neutral third party’s ratings can help governments guide pension-fund investment without staffing thousands of experts to rerun forecasts weekly.

“crack your radio they get to point to some neutral third party so my hypothesis is my hunch is that you will see a third party that sits between the government and the labs and it could either be the government builds it themselves so something like Casey was set up to do exactly this and the question”

Claim: The speaker hypothesizes that an intermediary will emerge between governments and AI labs, potentially as a government-built body.

“is the center for AI standards and innovation I won't get into the details but it's a sub body of NIST that typically typically set standards so it's basically a government body that has AI experts yeah”

Claim: The Center for AI Standards and Innovation (CAISI) is described as a NIST sub-body that typically sets standards and has AI experts.

“And they have excellent people, but they have an extraordinarily small budget compared to the scale of the challenge that's ahead of us.”

Claim: Government agencies have capable personnel but budgets that are very small relative to the scale of upcoming AI challenges.

“And I think they have a role to play.”

Claim: Government agencies have a role in addressing AI risks.

“Every model release, there is an astounding, given that they take in any input, their risk surface is astounding.”

Claim: AI models have an extensive risk surface because they accept arbitrary inputs, and this applies to every model release.

“Our perspective is that also at the model layer, the risk that people care about today. They are not the same ones that came out three months ago.”

Claim: At the model layer, the risks people prioritize change over time; today’s concerns differ from those of three months earlier.

“So the pace of legislation is too slow to deal with pinpointing the risks here.”

Claim: Legislation moves too slowly to identify changing AI-model risks in a timely way.

“And so we think there's a lot that the market can do to surface timely information.”

Claim: Market mechanisms can help surface timely information about changing AI risks.

“But what we want to make sure is that the process that produces this risk information is compatible with very fast innovation.”

Claim: Risk-information processes should be compatible with very fast innovation.

“But what we want to make sure is that the process that produces this risk information is compatible with very fast innovation.”

Claim: AI risk-information processes should keep pace with very rapid innovation.

“so you don't want to this is not a question of like can you slow the things down can you keep the models locked up until for months on end until everyone can make a guarantee uh but it is just can you in the time it given that”

Claim: The speaker rejects slowing AI development or withholding models for months while waiting for guarantees as the intended approach to risk management.

“The US is competing with China on releasing models.”

Claim: The United States is competing with China to release models.

“The US is competing with China on releasing models.”

Claim: The United States and China are competing in the release of AI models.

“Can you insert risk information that allows the government to make rapid decisions on some of these questions,”

Claim: Providing risk information could help governments make rapid decisions about AI-model risks and related policy questions.

“balancing that trade-off between failing to adopt AI is going to put us at risk, but also reckless adoption is going to put us at risk. And that's a very kind of fine balance that they're going to need like a lot of high quality intelligence to make.”

Claim: Both failing to adopt AI and adopting it recklessly create risks.

“balancing that trade-off between failing to adopt AI is going to put us at risk, but also reckless adoption is going to put us at risk. And that's a very kind of fine balance that they're going to need like a lot of high quality intelligence to make.”

Claim: Both failure to adopt AI and reckless AI adoption can create risks, so governments need high-quality intelligence to balance the trade-off.

“but also reckless adoption is going to put us at risk. And that's a very kind of fine balance that they're going to need like a lot of high quality intelligence to make.”

Claim: Balancing the risks of insufficient AI adoption and reckless AI adoption requires substantial high-quality intelligence.

“But CSOs have a bunch of concerns around data flows in general that they're really concerned about.”

Claim: Chief security officers have many concerns about data flows generally.

“So it's a lot of questions like if these models are Chinese, where does our data go?”

Claim: Users commonly ask where their data goes when they use Chinese models.

“I think a lot of this can be addressed.”

Claim: The speaker thinks many concerns about data flows and Chinese models can be addressed.

“They come up often.”

Claim: Concerns about data flows and Chinese models arise frequently.

“Some of them understand that they're running on a rack GPU.”

Claim: Some people do not realize that some models run on GPUs.

“A year ago, there was not a lot of understanding of this.”

Claim: One year earlier, there was little understanding of AI-related security and infrastructure issues being discussed.

“I actually think you're seeing the security leaders becoming kind of AI literate at a blistering pace.”

Claim: Security leaders are becoming knowledgeable about AI very rapidly.

“And you're actually also seeing my Twitter timeline that's very AI-pilled and my LinkedIn feed that used to not at all be AI-pilled kind of converge.”

Claim: The speaker observes that their AI-focused Twitter timeline and previously non-AI-focused LinkedIn feed are becoming more similar in their AI focus.

“They're both talking about fable.”

Claim: The speaker says both their Twitter and LinkedIn feeds are discussing fable.

“They are both talking about whether you can prevent models from being jailbroken these days.”

Claim: The speaker says both their Twitter and LinkedIn feeds are discussing whether models can be prevented from being jailbroken.

“like national security risk are kind of that conversation is actually emerging other than that i think you mostly see a kind of uh there's no concerns with any particular model or any particular model output but there's a general nervousness of having critical infrastructure run on models that are not produced in America by Americans where the American government has control.”

Claim: The speaker says national-security concerns are beginning to emerge, while the broader concern is nervousness about critical infrastructure relying on models not produced in the United States by Americans under U.S. government control; they report no concerns about a particular model or output.

“There's a bit of stuff in there actually on the provenance of the models and disclosing that,”

Claim: The speaker says their framework includes some provisions concerning model provenance and its disclosure.

“but I think there's a bunch of use cases where running a Chinese open source model is just the best solution.”

Claim: The speaker believes that using a Chinese open-source model is the best solution for many use cases.

“it concerns something more macro which is not best addressed at any particular certification level there's”

Claim: The speaker says this concern is a macro-level issue that is not best addressed through any single certification level.

“The risk of the day is cyber because it's very real, very tangible.”

Claim: The speaker identifies cyber risk as the current leading concern because it is concrete and immediate.

“And some of the risks that are also emerging as pretty real and pretty tangible are things like child safety is becoming both extremely important, but also politically important.”

Claim: The speaker says child safety is becoming both highly important and politically salient as an AI risk area.

“And there are some of the risks that are coming down the pipeline that today feel kind of speculative, but people spend a lot of time with the models, see them coming down as things like...”

Claim: Some emerging AI risks seem speculative today, although people who work extensively with models anticipate them.

“And specifically whether models will help adversaries produce biological weapons and making that extremely cheap, extremely accessible, producing,”

Claim: The speaker raises the possibility that AI models could help adversaries produce biological weapons at extremely low cost and with broad accessibility.

“making the chance of another COVID or worse pandemic,”

Claim: The speaker suggests that AI-enabled biological weapon production could raise the chance of another COVID-scale or worse pandemic.

“COVID was not engineered to be bad as if you're trying to do that.”

Claim: The speaker says COVID was not engineered to be as harmful as a pathogen deliberately designed for that purpose would be.

“I think one other thing to just note is that agents are kind of deliberately narrow. So like...”

Claim: AI agents are intentionally designed to operate within a narrow scope.

“So when a frontier agent company puts a chatbot that interacts with customers, they've really tried to narrow the topics it's interested in talking about.”

Claim: Companies deploying frontier AI agents in customer-facing chatbots deliberately restrict the topics those chatbots discuss.

“So if you ask it, like, what do you think of the president, it will just decline.”

Claim: A customer-facing chatbot with restricted topics may decline to answer a question about the president.

“Which means there's a kind of risk area that's somewhat smaller. For models, it is infinite.”

Claim: The speaker says agents' narrower scope makes their risk area somewhat smaller, whereas the risk area for models is effectively infinite.

“And so there's not a single expert out there who can competently evaluate the risks of cyber attacks and 15-year-olds having month-long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that and can evaluate the risks that terrorists can use AI to produce bioweapons.”

Claim: The speaker argues that no individual expert can competently assess the full range of risks, including cyberattacks, harmful chatbot interactions with teenagers, and AI-enabled bioweapon production by terrorists.

“The risk surface is just too big.”

Claim: The speaker considers the overall AI risk surface too broad for a single expert to assess.

“And so the central challenge actually becomes how do you get those? subject matter experts to work within a one coherent framework that outputs one coherent report and rating that the world can go and inspect because that global perspective is central but there's not a single organization today that could produce that we”

Claim: A central challenge is coordinating subject-matter experts through a unified framework that produces a coherent, publicly inspectable report and rating.

“subject matter experts to work within a one coherent framework that outputs one coherent report and rating that the world can go and inspect because that global perspective is central but there's not a single organization today that could produce that we”

Claim: The speaker says no single organization currently produces a unified, global perspective on these AI risks.

“think there can be one company that can with a consortium of experts build one coherent standard”

Claim: A single company could coordinate a consortium of experts to create a unified standard for evaluating AI risks.

“I think we've shown that across all of the enterprise risks today.”

Claim: The speaker says their organization has demonstrated this approach across current enterprise risks.

“We think it could be one company that could, with a consortium, specify the audit rules, basically like the inputs and outputs that all these technical experts need. What access do they need?”

Claim: A single coordinating company and expert consortium could define audit rules, including required inputs, outputs, and expert access.

“How should they treat infrastructure security?”

Claim: A shared audit framework should define how technical experts assess infrastructure security.

“They can look at whether their evals are well produced. Without necessarily being able to say, hey, is this a threat or not a threat, but overall evaluating whether the evals are good, well constructed,”

Claim: Technical experts can evaluate the quality and construction of model evaluations without necessarily determining whether the model itself poses a threat.

“that set of older rules basically becomes the interface for all these experts we think one clearinghouse could put together.”

Claim: The speaker thinks a common rule set could coordinate experts and be assembled by a single clearinghouse.

“To be clear, when I say one company, I think of it as one company coordinating lots of this in the same way that when we saw our consortium, it's not like we say we have all the answers on agent security.”

Claim: The speaker uses “one company” to mean a coordinator of many contributors, rather than an organization claiming complete expertise in agent security.

“pretty. What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house such that the standard updates lockstep every quarter and that they order reports that come out in this case 100 page order reports uniform and crisp and clear to the level of detail that is required for executives that need to make a clear go no go decision so that's kind of the role that we think we might play yeah”

Claim: The speaker describes their role as gathering expert concerns, coordinating them, and maintaining a process for quarterly, synchronized standard updates.

“lockstep every quarter and that they order reports that come out in this case 100 page order reports uniform and crisp and clear to the level of detail that is required for executives that need to make a clear go no go decision so that's kind of the role that we think we might play yeah”

Claim: The speaker believes their organization may serve as the coordinator of the expert-driven standards and reporting process.

“lockstep every quarter and that they order reports that come out in this case 100 page order reports uniform and crisp and clear to the level of detail that is required for executives that need to make a clear go no go decision so that's kind of the role that we think we might play yeah”

Claim: The speaker says the process aims to produce consistent, clear, detailed reports of roughly 100 pages for executives making go/no-go decisions.

“so first of all owasp is basically an open source Open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns.”

Claim: OWASP is an open-source community of security practitioners that develops frameworks for addressing current security concerns.

“We think they are phenomenal at creating frameworks.”

Claim: The speaker considers OWASP exceptionally good at creating frameworks.

“We've in fact, first of all, we're partners with them. So we have a joint article to we've learned a lot from them.”

Claim: The speaker's company partners with OWASP and has published a joint article with OWASP.

“We've in fact, first of all, we're partners with them. So we have a joint article to we've learned a lot from them.”

Claim: The speaker's company has learned a lot from OWASP.

“We think a tremendous source of intelligence.”

Claim: The speaker considers OWASP a tremendous source of intelligence.

“What it almost does not do is building the machine that runs third party audits.”

Claim: OWASP does not build the operational machinery needed to run third-party audits.

“They are part of the information gathering and intelligence gathering and creating clarity, but the operational layer of turning this into promises is not the business they try to be in.”

Claim: OWASP focuses on gathering information and intelligence and creating clarity; turning standards into operational promises is not its intended business.

“In the security space today, cybersecurity, most of the standards are produced by non-profits.”

Claim: Most cybersecurity standards are produced by non-profit organizations.

“I think that's an issue.”

Claim: The speaker considers the predominance of non-profit-produced cybersecurity standards a problem.

“non-profits tend to not have these adverse profit incentives where they hollow out their standard and create a race to the bottom but they're also not at all responsive by default to the communities that they serve because there's no process they don't have customers that they serve where they go and ask what do you want what do you want what do you want and when you look at the overall satisfaction with the security standards today”

Claim: Non-profit organizations are not inherently responsive to the communities they serve because they generally lack a customer-feedback process.

“non-profits tend to not have these adverse profit incentives where they hollow out their standard and create a race to the bottom but they're also not at all responsive by default to the communities that they serve because there's no process they don't have customers that they serve where they go and ask what do you want what do you want what do you want and when you look at the overall satisfaction with the security standards today”

Claim: Non-profit organizations tend to lack the adverse profit incentives that could weaken standards and drive a race to the bottom.

“People tend to just not like them very much.”

Claim: People generally have low satisfaction with current security standards.

“You do see in other domains that for-profit standards can serve the world quite well.”

Claim: For-profit standards can serve society effectively in some domains.

“So there are examples like we talked about movies before. It's not without flaws, but it is absolutely critical societal infrastructure that gets run at an astounding scale today.”

Claim: The speaker describes the movie-rating example as imperfect but as critical societal infrastructure operating at enormous scale.

“Your credit score, it's FICO. It's also a for-profit business.”

Claim: FICO provides the credit score referenced by the speaker and operates as a for-profit business.

“some of the crash testing standards came out of insurance companies.”

Claim: Some crash-testing standards were developed by insurance companies.

“The insurance companies together founded the Institute of Highway Safety because they were very interested in like, how can we use standards to drive down mortality and save money?”

Claim: Insurance companies jointly founded the Institute of Highway Safety to use standards to reduce deaths and save money.

“Go back, prior our name actually pays homage to the Underwriters Laboratory, UL,”

Claim: The speaker's company name pays homage to Underwriters Laboratories (UL).

“which was started right around when electricity came out, houses started burning down,”

Claim: Underwriters Laboratories was established around the time electricity became widespread and house fires became a concern.

“insurers again were paying the bill.”

Claim: Insurers bore the costs associated with house fires around the time Underwriters Laboratories began.

“and there are maybe also good people but their profit incentive was let's prevent houses from burning down let's test all the electrical products the light bulbs all the light bulbs in here are probably you all tested the toasters etc and they set up an entity to create those standards today ul has a for-profit entity and a non-profit entity um where they've recognized they spun out they started a non-profit they spun out a for-profit because what they recognized was like hey actually to serve”

Claim: Underwriters Laboratories has both for-profit and non-profit entities.

“and there are maybe also good people but their profit incentive was let's prevent houses from burning down let's test all the electrical products the light bulbs all the light bulbs in here are probably you all tested the toasters etc and they set up an entity to create those standards today ul has a for-profit entity and a non-profit entity um where they've recognized they spun out they started a non-profit they spun out a for-profit because what they recognized was like hey actually to serve”

Claim: UL's profit incentive was to prevent house fires by testing electrical products such as light bulbs and toasters.

“and there are maybe also good people but their profit incentive was let's prevent houses from burning down let's test all the electrical products the light bulbs all the light bulbs in here are probably you all tested the toasters etc and they set up an entity to create those standards today ul has a for-profit entity and a non-profit entity um where they've recognized they spun out they started a non-profit they spun out a for-profit because what they recognized was like hey actually to serve Serve customers well. You need a for-profit entity.”

Claim: UL created a for-profit entity because it recognized that serving customers well required one.

“The lesson here is one of the ways that the market can align incentives so you're both responsive to customers and not hollowing out your standard over time.”

Claim: Market incentives can make standards responsive to customers while discouraging their weakening over time.

“is to align it with insurers because they fundamentally have good incentives and so if you're a for-profit standard that works closely with insurers you get the feedback loop in such that you're really tuned into your customers but also have their interest at heart so that's the model that we are the kind of inspirational model that we've learned a lot from and that's also where the name comes from in some ways the term underwriting can both be associated with insurance but it's also a broad term for like”

Claim: The speaker's company has taken inspiration from a for-profit standards model that works closely with insurers.

“is to align it with insurers because they fundamentally have good incentives and so if you're a for-profit standard that works closely with insurers you get the feedback loop in such that you're really tuned into your customers but also have their interest at heart so that's the model that we are the kind of inspirational model that we've learned a lot from and that's also where the name comes from in some ways the term underwriting can both be associated with insurance but it's also a broad term for like”

Claim: A for-profit standards organization working closely with insurers can benefit from customer feedback and aligned incentives, since insurers have good incentives.

“is to align it with insurers because they fundamentally have good incentives and so if you're a for-profit standard that works closely with insurers you get the feedback loop in such that you're really tuned into your customers but also have their interest at heart so that's the model that we are the kind of inspirational model that we've learned a lot from and that's also where the name comes from in some ways the term underwriting can both be associated with insurance but it's also a broad term for like making decisions if”

Claim: “Underwriting” can be used broadly to mean making decisions.

“you underwrite a decision you're fundamentally kind of taking ownership for for the consequences of it yeah”

Claim: To underwrite a decision is to take ownership of its consequences.

“most of the demand comes today for insurance contractors sitting between people who've built AI and people who are buying AI and what you want is the reason why people want insurance involved both for the traditional reasons hey if something goes wrong, we want to be compensated.”

Claim: People seek insurance partly to receive compensation if something goes wrong.

“most of the demand comes today for insurance contractors sitting between people who've built AI and people who are buying AI and what you want is the reason why people want insurance involved both for the traditional reasons hey if something”

Claim: Most current demand for AI insurance comes from insurance contractors positioned between AI developers and AI purchasers.

“But it's in particular because insurers can bring trust to the equation.”

Claim: Insurers can increase trust in AI adoption by participating in insurance arrangements.

“Because insurers will pay for the damages, if they're willing to write an insurance policy, that is them saying,”

Claim: Insurers will pay for damages when they are willing to write an insurance policy covering the risk.

“that is them saying, hey, we think there's risk here, but that is manageable.”

Claim: An insurer that writes a policy signals that it considers the covered risk real but manageable.

“And that is kind of their incentive aligned with the enterprises adopting it.”

Claim: Insurers' incentives are aligned with enterprises that adopt AI.

“So that's really a good signal to the market.”

Claim: The availability of insurance is a positive signal to the market about the insured AI risk.

“one of the things that Waymo tries to get the first permit to even operate in San Francisco was to get a lot of insurers.”

Claim: Waymo sought substantial insurance coverage before obtaining its first permit to operate in San Francisco.

“to stack up a huge insurance policy in the case that something went wrong not because google can't pay but because it's very valuable to have a third party go and look at that data that are trusted by governments trusted by enterprises as conservative people and say hey we've looked at it we're actually willing to take some of this on top of our sheet so that's that's kind of the reason why people are interested in it what it looks like is uh in some ways like every other insurance contract you specify what are the perils you want to cover How much do you want to cover them? Like up to what limits? What does it cost to cover that?”

Claim: An insurance contract specifies which perils are covered, the coverage limits, and the price of coverage.

“to stack up a huge insurance policy in the case that something went wrong not because google can't pay but because it's very valuable to have a third party go and look at that data that are trusted by governments trusted by enterprises as conservative people and say hey we've looked at it we're actually willing to take some of this on top of our sheet so that's that's kind of the reason why people are interested in it what it looks like is uh in some ways like every other insurance contract you specify what are the perils you want to cover”

Claim: Waymo sought a large insurance policy to obtain trusted third-party review of its data and reassure governments and enterprises, rather than because Google lacked the ability to pay damages.

“11 Labs. bought a first of its kind AI agent insurance policy.”

Claim: ElevenLabs purchased what the speaker described as the first AI agent insurance policy of its kind.

“They work with some of the biggest enterprises that work with governments.”

Claim: ElevenLabs works with some of the largest enterprises that serve governments.

“They're really interested in going above and beyond and making promises to their customers.”

Claim: ElevenLabs wants to exceed baseline expectations and make commitments to its customers.

“So they wrote a policy that covers just some of the core concerns that their customers have been asking about.”

Claim: ElevenLabs' AI insurance policy covers some core concerns raised by its customers.

“And the crucial thing was really to get Lloyds of London. The world's oldest insurer, one of our partners, to look at this data and be that third party alongside us to say,”

Claim: A crucial part of ElevenLabs' AI insurance arrangement was having Lloyd's of London review the data as a third party alongside the speaker's organization.

“hey, we think there's something here that's worth underwriting and that's actually what it looks like. And so they will show that contract to their customers and they can see how much they're covered for, they can see what exactly it covers and that will also probably change next year.”

Claim: Lloyd's of London considered the AI risk worth underwriting, and ElevenLabs customers can review their insurance contract to see the amount and scope of coverage.

“they can see what exactly it covers and that will also probably change next year.”

Claim: The insurance contract will probably change in the following year.

“They will want to write an insurance policy that might cover more.”

Claim: Insurers may seek to offer an insurance policy with broader coverage.

“Yeah, so typically the way new companies get into insurance is that they partner with insurers such that the insurers take the majority or all of the financial risks.”

Claim: New insurance companies typically enter the market through partnerships in which established insurers assume most or all of the financial risk.

“Fundamentally, if insurance is useful because it brings trust, you have to be able to pay the bill.”

Claim: For insurance to provide trust, the insurer must have the financial capacity to pay claims.

“Lloyds of London is 400 years old.”

Claim: Lloyd's of London is 400 years old.

“They've never not paid a claim.”

Claim: Lloyd's of London has never failed to pay an insurance claim.

“They're extremely trusted.”

Claim: Lloyd's of London is highly trusted.

“what Lords of London struggle to do on their own. is to figure out which of the risks are real, what should we be looking for, what are the kind of technical controls and running the tests.”

Claim: Lloyd's of London has difficulty identifying real risks and determining the relevant technical controls and tests.

“So they use AOC1 as kind of the underwriting framework and we produce a bunch of e-mail results that then directly feed in to inform the pricing.”

Claim: Lloyd's uses AOC1 as an underwriting framework, with the speaker's organization supplying evaluation results that directly inform insurance pricing.

“So this means that 11Labs customers know that that payment will be there.”

Claim: ElevenLabs customers can rely on the insurance payment being available if a covered claim occurs.

“They don't have to look to our Series A and see like, do we think they have enough cash on the balance sheet?”

Claim: Because the policy provides insurance backing, ElevenLabs customers do not need to rely on the company's balance-sheet cash to assess whether claims can be paid.

“exactly it's actually a remarkable institution that's both kind of has some of the truly old school virtues of having been around for a long time they like really they really operate like a trusted entity and they have appetite to figure out the future and i think there's a lot of recognition that both there is like tremendous amount of risk in ai that is poorly understood today so getting into this business carries real risks but also this is where lots of the risk description will happen in the future”

Claim: Lloyd's of London combines longstanding institutional virtues with a willingness to engage with future risks.

“they really operate like a trusted entity and they have appetite to figure out the future and i think there's a lot of recognition that both there is like tremendous amount of risk in ai that is poorly understood today so getting into this business carries real risks but also this is where lots of the risk description will happen in the future”

Claim: A substantial amount of AI risk is poorly understood at present.

“they really operate like a trusted entity and they have appetite to figure out the future and i think there's a lot of recognition that both there is like tremendous amount of risk in ai that is poorly understood today so getting into this business carries real risks but also this is where lots of the risk description will happen in the future”

Claim: Entering AI insurance carries real risks, while the market is also likely to shape how AI risks are described in the future.

“This is the one market where risk is truly growing.”

Claim: AI is the one market in which risk is genuinely increasing.

“This is the one market that will also take out some of the existing markets, take like auto insurance.”

Claim: The AI insurance market will take over some existing insurance markets, including auto insurance.

“When there are no human drivers, how is that market going to look? Well, it's clearly going to change.”

Claim: Auto insurance will change substantially in a scenario where there are no human drivers.

“All I'll say is the principles for how you insure Waymo are very similar to how you insure other kinds of AI.”

Claim: The principles used to insure Waymo are very similar to those used to insure other AI systems.

“again, crash testing, that's what we do for customer share or lovable. That will also need to happen for Waymo, which is not how you do it for human drivers.”

Claim: Insuring Waymo will require crash testing, unlike insuring human drivers.

“So there's this growing awareness that the world is changing very fast and the only way to learn how to underwrite AI is to write some policies.”

Claim: Writing insurance policies is the only way to learn how to underwrite AI.

“You may incur some losses and think of that as R&D expense, really.”

Claim: Insurers entering AI underwriting may incur losses and can treat them as research-and-development expenses.

“But the question for them is like, who are the trusted technical partners they can get into this business with that can help them navigate and make sure they don't make kind of foolish mistakes,”

Claim: Insurers entering AI underwriting need trusted technical partners to guide them and help prevent poor decisions.

“but also who is willing to hear the wisdom that they have?”

Claim: Technical partners should also be willing to learn from insurers' accumulated expertise.

“They've done this before they've seen it was they were there when cyber came out so”

Claim: Some insurers have experience from the period when cyber risks first emerged.

“There are lots of ways in which AI feels completely new, but there's also lots of ways in which risks look the same.”

Claim: AI risks are novel in some respects but resemble familiar risks in others.

“And so there's actually a tremendous amount of wisdom sitting in some folks that may have gray hair, but they really have like a keen sense of how to quantify risk.”

Claim: Experienced insurance professionals possess substantial expertise in quantifying risk.

“exactly you basically split the fee and you can think of the insurance supply chain is like there's bringing the capital there is doing the pricing and there's doing the distribution and typically you will pay out some x percent of premium here y percent It's going to be me here and the rest of it will go here.”

Claim: The insurance supply chain includes capital provision, risk pricing, and distribution, and premium revenue is divided among these functions.

“And I could see some worlds where that makes sense, specifically if there are risks that we feel high confidence they would want to insure where the incumbent insurers. are too slow to find appetite or simply struggle to evaluate it such that they don't want to do it but by and large in general you do not want to compete with insurers on bringing risk capital to the game for two reasons one is that's fundamentally a cost of capital game they have extremely low cost of capital startups have high cost of capital by and large and two you want to”

Claim: AIUC could insure risks that incumbent insurers are too slow to underwrite or struggle to evaluate, in cases where AIUC has high confidence that the risks need coverage.

“simply struggle to evaluate it such that they don't want to do it but by and large in general you do not want to compete with insurers on bringing risk capital to the game for two reasons one is that's fundamentally a cost of capital game they have extremely low cost of capital startups have high cost of capital by and large and two you want to”

Claim: AIUC generally should not compete with insurers to supply risk capital because insurers have an extremely low cost of capital while startups generally have a high cost of capital.

“hedge your bets and it's very helpful then to also have a portfolio of home insurance of car insurance and we're not about to become a car insurer nor a home insurer so they have some natural advantages which makes it much more likely that we'll partner”

Claim: AIUC prefers partnering with insurers that have diversified home- and car-insurance portfolios, rather than becoming a home or car insurer itself, partly to hedge its bets.

“and they bring the capital at scale and we bring the technical expertise.”

Claim: AIUC's intended partnership divides roles: insurers provide capital at scale, and AIUC provides technical expertise.

“I think maybe the first thing is they feed into the standard as well. So if there are things that they feel like they need that they're not seeing, we are also taking that as input into the standard because fundamentally we think”

Claim: AIUC incorporates insurers' identified unmet needs as input when developing its standard.

“A good standard is one that creates a really healthy promise ecosystem, and we think insurers are a critical part of that.”

Claim: AIUC believes a good standard should create a healthy promise ecosystem, in which insurers play a critical role.

“And again, they are the most well incentivized too. They see all the lost data across any particular CISO knows their particular concerns. Insurers see the concerns across the entire portfolio and often have direct access to like what exactly happened, who was at fault, et cetera, as they do part of their forensics. So they're actually like a great source of intelligence on this.”

Claim: Insurers are well positioned and incentivized to inform standards because they observe concerns and losses across their portfolios and often access forensic details about incidents, including causes and fault.

“One of the big takeaways from cyber insurance, which was a market that didn't work that well, was that the insurance and the technical expertise was not married up.”

Claim: A major lesson from the poorly functioning cyber-insurance market was that insurance was not integrated with technical expertise.

“What our conviction is that standards have to precede insurance.”

Claim: AIUC believes standards must be established before insurance is developed or offered.

“Fundamentally, what everyone first and foremost want, whether you're a CISO at JPMorgan or a CISO at Cursor or an underwriter at Lloyd's of London syndicate, is you want to not have an incident in the first place. You want to know that the risk is well managed and only then does insurance start to make sense. So we'll see the standard ecosystem basically run ahead of the insurance and the reason why.”

Claim: AIUC argues that users and underwriters primarily want to prevent incidents and ensure risks are well managed, after which insurance becomes useful.

“we you asked us kind of why also do insurance this is kind of proving what we think a whole promise confidence infrastructure ecosystem needs to look like and we think it's very compelling to bring that to life even if we think the standard is kind of the the core linchpin that unlocks the rest yep”

Claim: AIUC views its insurance work as a demonstration of a broader promise-confidence infrastructure ecosystem, while regarding the standard as the central component that enables the rest.

“I think the first thing is people have limits on their policy.”

Claim: Insurance policies impose coverage limits on policyholders.

“If you want to claim $200 million, someone has to have paid a lot for that insurance policy up front to have $200 million of coverage.”

Claim: A $200 million insurance claim requires a policyholder to have purchased $200 million in coverage, which entails paying a substantial premium up front.

“the way this works is that you start from a lot of uncertainty. Well, they can go and look at precedent, they can see, but ultimately these things get settled in court and you hammer it out over time. So you start from this like place of ambiguity.”

Claim: Insurance-related questions begin in ambiguity, and legal precedent and court proceedings resolve them over time.

“which is both why insurance can be hard to do early on but it's also why people want insurance because that ambiguity slows down adoption that also sits at the heads of the state”

Claim: Early ambiguity makes insurance harder to provide while increasing demand for insurance, because uncertainty slows adoption.

“exactly and there have been a number of incidents out there that have just not been insurance covered so take the now old example from Air Canada where someone hallucinated a refund policy”

Claim: Some AI-related incidents have not been insured; one example cited is the Air Canada case involving a chatbot that hallucinated a refund policy.

“see and the question was a kind of case where like hey we have nothing to do with this this chatbot messed up but like sorry and the courts were like no if you put your chatbots to interact with your customers they make legally binding promises on your behalf that is now precedent for everything in the future where you will”

Claim: In the Air Canada chatbot case, the speaker says the courts established that a company is responsible for legally binding promises its customer-facing chatbot makes on its behalf.

“If someone wants to deploy a chatbot like that again, they should not expect to be able to just pawn off and say, sorry, my chatbot lied. It's nothing to do with me. I bought it from OpenAI. No, if you're putting this in front of your customers.”

Claim: A company that deploys a chatbot for customer interactions should not expect to disclaim responsibility by blaming the chatbot or its provider, such as OpenAI.

“No, if you're putting this in front of your customers. You are taking responsibility for it.”

Claim: A company that deploys a chatbot in front of its customers takes responsibility for the chatbot's actions.

“And so every court case, whether insurance is involved or not, clarifies liability.”

Claim: Court cases clarify liability regardless of whether insurance is involved.

“And liability is kind of the foundation for insurance.”

Claim: Liability provides a foundation for insurance.

“Liability, often one of the core concepts is whether someone was negligent.”

Claim: Liability often depends on whether a person was negligent.

“you basically judge whether they've met their duty of care.”

Claim: Negligence is often judged by whether a person met their duty of care.

“often they look to standards.”

Claim: Courts often use standards to assess whether a person met their duty of care.

“So if there's a standard that is broadly adopted that says you must have a groundedness filter or you must have a jailbreak filter, it becomes way harder to claim ignorance that these things existed.”

Claim: When a broadly adopted standard requires a groundedness filter or jailbreak filter, it becomes harder for an organization to claim it did not know those safeguards existed.

“help clarify liability courts will often point to standards and being like well this seems like best practice to do is there for everyone to see so there's another way in which like standards are kind of civilization infrastructure that insurance can build on which promises can then build on”

Claim: Standards can clarify liability by making best practices visible to courts, and they can provide infrastructure on which insurance and related promises can build.

“There's a couple of requirements in there that relate to how do you yourself, how are you yourself testing before you make major releases. We don't go and audit people every day,”

Claim: AIUC does not conduct daily audits; its requirements instead address organizations’ pre-release testing and customer disclosures.

“but at least there is now a trail where if you... do a major mess up then your customer may come and ask you hey you promised me that you were going to run these evals yourself and for lots of them most most of the PRs that people merge were not fundamentally altered the product experience but some of them were and”

Claim: AIUC’s requirements create an audit trail that customers can use to check whether a company performed promised evaluations after a major failure.

“you don't know and this is also true and this is also there are some inherent risks that everyone kind of everyone knows that when they buy software there can be bugs and this this is just part of it but what they can if you're selling to mom and pop shops they may not care”

Claim: Software carries inherent risks such as bugs, and customers vary in their willingness to accept those risks.

“okay that is like what i want to use your tool so i'm just going to want to take that risk on if you're selling to a big bank they might be like sorry we're making promises to our customers if you can't make a promise to us that we can pass on we don't want to work with you then it's up to you to say do i care for my agent to get used as critical infrastructure in this nation”

Claim: A large bank may reject a vendor that cannot provide assurances the bank can pass on to its customers.

“if so at least i can make promises about what process i run and then we can go and test it every quarter to be like well does this seem like uh it's sort of uh kind of it still meets the standard so from my perspective it's kind of a way to big companies by default kind of have some amount of trust when they ship ai”

Claim: The speaker believes process-based assurances can give large companies some baseline trust when deploying AI.

“if so at least i can make promises about what process i run and then we can go and test it every quarter to be like well does this seem like uh it's sort of uh kind of it still meets the standard so from my perspective it's kind of a way to big companies by default kind of have some amount of trust when they ship ai”

Claim: A company can promise to follow a defined process and test quarterly whether that process continues to meet the standard.

“If you're a young company, if you're just starting out, by default you have no trust. And there are very few places where you can go and get trust.”

Claim: A newly established company starts with little or no default trust and has few places to obtain it.

“So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts.”

Claim: Many of AIUC’s customers published their own security blog posts before they began working with AIUC.

“But also, who's going to trust you saying we're so secure? Like anyone can write that.”

Claim: Customers may distrust a company’s self-authored security claims because any company can make them.

“And so I think making the standards more legible. makes it easier for smaller companies to prove that they're doing what they ought to be doing because the default assumption is that it's the Wild West yep”

Claim: Clearer, more legible standards help smaller companies demonstrate appropriate practices in a field commonly assumed to be unregulated or chaotic.

“think when we when we zoom out AIC1 deals with agents. Next up, we will deal with models. Next up from that, we will deal with robotics,”

Claim: AIUC’s roadmap is to address agents first, then models, and then robotics.

“of which in some ways Waymo is the first robot,”

Claim: The speaker characterizes Waymo as, in some ways, the first robot.

“but the exact same problem is going to be someone's going to develop a robot, someone's going to need some promises. they're going to struggle to make the promises.”

Claim: Robot developers will need to provide assurances about their products and may struggle to do so.

“if you think fable concerns are bad, like see when Waymo hits a dog and that's if people lose their mind, imagine when first robot knocks off a toddler off a kitchen table, going to see some real strict liability.”

Claim: The speaker predicts that a robot causing a toddler to fall from a kitchen table would trigger intense public reaction and strict liability.

“So physical AI, the level of stringency just goes up. up and up and up and up and so that's kind of like the big picture agents models robotics i think within agents”

Claim: The speaker says regulatory or safety stringency rises for physical AI applications.

“The current set of agents are well covered by this, but as the technology progresses, as agents get longer horizons, new types of failure modes will emerge.”

Claim: The current standard covers existing agents well, but advancing technology and longer agent horizons will produce new failure modes.

“And so it's mostly of can you make sure that the standard keeps up when they appear?”

Claim: AI safety standards must evolve to address new failure modes as they emerge in agents.

“world models is mostly kind of a research question. There's no one who's really using it,”

Claim: At the time of speaking, world models are primarily a research topic and are not yet in substantial practical use.

“but that will also bring in just new kinds of ways to create value. value but also more risk surface that no one knows how to grab with today you'll start to see true agent to agent interactions that are not mediated by humans there's going to be a bunch of interesting questions you're basically going to need a new legal system how do they build trust amongst each other how one of the core things when humans trade with each other is that you know that you have recourse you can sue them”

Claim: World models are expected to create new forms of value while introducing risks that current approaches do not yet know how to manage.

“value but also more risk surface that no one knows how to grab with today you'll start to see true agent to agent interactions that are not mediated by humans there's going to be a bunch of interesting questions you're basically going to need a new legal system how do they build trust amongst each other how one of the core things when humans trade with each other is that you know that you have recourse you can sue them”

Claim: The speaker identifies the ability to seek recourse, including suing, as a core feature of trade between humans.

“value but also more risk surface that no one knows how to grab with today you'll start to see true agent to agent interactions that are not mediated by humans there's going to be a bunch of interesting questions you're basically going to need a new legal system how do they build trust amongst each other how one of the core things when humans trade with each other is that you know that you have recourse you can sue them”

Claim: Unmediated interactions among agents may require new legal arrangements for establishing trust between agents.

“value but also more risk surface that no one knows how to grab with today you'll start to see true agent to agent interactions that are not mediated by humans there's going to be a bunch of interesting questions you're basically going to need a new legal system how do they build trust amongst each other how one of the core things when humans trade with each other is that you know that you have recourse you can sue them”

Claim: The speaker expects agents to begin interacting directly with one another without human mediation.

“How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you, you know you can get your money back?”

Claim: Agent commerce needs a persistent financial backing mechanism that allows counterparties to recover money when an agent causes them a loss.

“And the technical testing of multi-agent systems is also going to be interesting and complex.”

Claim: The technical evaluation of multi-agent systems is expected to be complex.

“Yeah, one of the places where there's a bunch of appetite for insurance and not a lot of demand but not a lot of supply is when it comes to copyright.”

Claim: The speaker says demand for copyright-related insurance is strong while available supply is limited.

“The first is people who have trained on copyrighted materials almost always know that they've done that.”

Claim: People who train models on copyrighted materials almost always know they have used copyrighted materials.

“So if you want to buy insurance for it, it probably signals. that you might be a high risk customer the people who are most interested in getting insurance for copyright infringement the people who are most likely to have committed copyright infringement”

Claim: People most interested in purchasing copyright-infringement insurance may be high-risk customers and may be the people most likely to have committed copyright infringement.

“at least a hard question i don't have the answer to it but i think your your intuition is your intuition is right kind of like uh what is the kind of duty of care”

Claim: The speaker considers the duty of care owed in relation to open models a difficult question that he cannot answer.

“And people don't today think of it as customary that you go and you like dissect your open models training data and you check everything.”

Claim: It is not currently customary for users to inspect all the training data of open models.

“In fact, lots of people use them. It's seen as kind of generally acceptable. to not check for this and therefore we're not going”

Claim: Many people use open models, and not checking their training data for copyright issues is generally considered acceptable.

“so you could of then ban it but i think no court is going to a copyright question”

Claim: The speaker thinks a court is unlikely to ban open models over a copyright question.

“The core problem is one of information asymmetry. If you were buying insurance and know something about their risk that the insurer does not know.”

Claim: Insurance can involve information asymmetry when the person seeking coverage knows something about their risk that the insurer does not know.

“you can break a lot of these information asymmetries if there is some kind of testing that reveals the underlying true risk.”

Claim: Tests that reveal underlying risk can reduce information asymmetries in insurance.

“you were able to in the case you mentioned have good diagnosis of whether someone has it or what the probability is that someone has it that the insurers trust then they might be willing to insure it but if they don't if there's no kind of common information then they only the patient know and that”

Claim: Insurers might be willing to provide coverage when they trust a diagnosis or probability estimate of a person's condition; without shared information, the patient alone may know the relevant risk.

“sort of breaks it down so the question is again how do you create credible signaling between players this is also the whole reason why Moody's exists Moody's just does credible signaling.”

Claim: Credible signaling can address information asymmetry, and Moody's exists to provide such signaling.

“That's also why Moody's has to be independent. If Moody's was owned by JPMorgan, then JPMorgan could not use it as a signaling mechanism.”

Claim: Moody's needs to be independent because JPMorgan could not credibly use it as a signaling mechanism if JPMorgan owned it.

“So a lot of the basics of standards and certification are just communication devices. It's just a trust gap.”

Claim: Standards and certification function as communication devices that help address a trust gap.

“and one another way you can break a lot of this is through transparency if you are transparent in how you operate you just cannot mess with others nearly as easily you make it much more costly and that increases trust this is one of the reasons why there's a change lock here every”

Claim: Operational transparency makes it harder and more costly to manipulate others, thereby increasing trust.

“a lot of this is just as things get clearer you can see a lot of clarifications you can see some revisions as things get hammered out you want to change this but if you make it all public you make it much harder to mess with people or at least you become found out very easily and so this is a way of increasing So reducing the information asymmetry is by just making more of the information public.”

Claim: Making more information public reduces information asymmetry.

“We deposit some amount of trust every time we meet this commitment.”

Claim: Each time an organization meets a commitment, it builds some trust.

“easy to ship a new version of a standard once a quarter in the enterprises who are used to this like decade-long cycle”

Claim: Shipping a new version of a standard quarterly is easy compared with the decade-long cycles familiar to enterprises.

“we often get met with like incredulity like there's just no way and then you show them the changelog yes”

Claim: Enterprise audiences often doubt that standards can be updated quarterly until shown the changelog.

“I think McInterb is a big one. a lot of interest in that. I think everyone would agree that there's like promising scientific potential. We're still a while. a little bit aware at least from this being like commercially available on demand such that there's like now a selection of vendors you can go to exactly”

Claim: McInterb has promising scientific potential, but the speaker believes it is still some time from being commercially available on demand from multiple vendors.

“we would agree with them we think the work that they're doing is tremendous we're not quite at the point where we could like literally require it but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use”

Claim: The speaker considers Goodfire’s work tremendous but says their organization is not yet able to require its use.

“we would agree with them we think the work that they're doing is tremendous we're not quite at the point where we could like literally require it but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use”

Claim: The speaker says an optional control for using Goodfire’s technology could plausibly be introduced relatively soon.

“That can turn up as a way to reduce risk.”

Claim: The speaker says the optional control can serve as a way to reduce risk.

“We can't require because it would be hard to require everyone to become good fire customers.”

Claim: The speaker says their organization cannot require everyone to become a Goodfire customer because enforcing that requirement would be difficult.

“It's a pass fail, but it's also a 100-page order report that you'd be surprised at how much security leaders actually sit down and digest this stuff.”

Claim: The speaker says the assessment is pass/fail and produces a 100-page report that security leaders often read and digest.

“And I promise you that if someone is using MacInter today, they will have a slide on it.”

Claim: The speaker predicts that anyone using MacInter today will include it on a slide.

“So I think McIntyre is interesting.”

Claim: The speaker finds McIntyre interesting.

“And I think if that promise truly comes to fruition, you can make stronger promises than you can believe us.”

Claim: The speaker says that if the promise is fulfilled, organizations could make stronger assurances than assurances based only on trusting the speaker’s organization.

“And so I think that's very compelling.”

Claim: The speaker considers the described promise and its potential to enable stronger assurances very compelling.

“Another thing that I think will become increasingly important is just kind of good old school monitoring.”

Claim: The speaker expects traditional monitoring to become increasingly important.

“And slightly after the fact, one of the things you're seeing with eval, some of the challenges that are emerging is that the agents are starting to become aware that they're being eval.”

Claim: The speaker says agents are increasingly becoming aware that they are being evaluated.

“Which is a problem.”

Claim: The speaker considers agents’ awareness of evaluation to be a problem.

“It means that they basically, if they know they're being watched, they won't do the thing that they think they get punished for.”

Claim: The speaker says agents that know they are being monitored will avoid actions they believe would lead to punishment.

“unless you know how to kind of reduce the amount of awareness you should trust the amount less and one of the kind of”

Claim: The speaker says evaluation results should be trusted less unless one knows how to reduce agents’ awareness that they are being evaluated.

“monitoring is the source of truth.”

Claim: The speaker says monitoring provides the source of truth for determining what agents actually do.

“So I think that is also a paradigm that's slightly more intrusive. You actually will look at some customer data, but I think will become more prevalent over time.”

Claim: The speaker says this monitoring approach is somewhat more intrusive because it examines some customer data, and expects it to become more prevalent over time.

“Did you guys see the anthropic research where, I think this is literally anthropic did that test, it took, I can't remember the details here, but they ran some studies on misalignment and then they took out the training data that related to less wrong discussing misalignment and they ran the same test. test again and the failure rate went down so it in fact was some evidence pointing towards it had learnt”

Claim: The speaker recounts an Anthropic study in which researchers removed training data related to LessWrong discussions of misalignment and reran a test, observing a lower failure rate.

“test again and the failure rate went down so it in fact was some evidence pointing towards it had learnt their ability or their propensity to do that.”

Claim: The speaker says the lower failure rate provided some evidence that the model had learned the ability or propensity to exhibit the behavior under study.

“very important for understanding that when you train for a thing, you also train the opposite of the thing because it's just a bit flip.”

Claim: The speaker argues that training a model for a behavior also trains its opposite, describing the relationship as a bit flip.

“I think just going back to where we were at, there's a lot more than just Mac interpret.”

Claim: The speaker says the relevant AI stack includes many components beyond MacInter.

“There's value in just having added,”

Claim: The speaker says there is value in adding further measures to the approach under discussion.

“you know do you see other parts of the stack like the inference providers that you use the services okay am i using chinese model on their home api am i using through certified vendor here am i hosting myself what am i doing on the inference engine side there's just like so many levels of stuff that gives you know information that you can standardize out right”

Claim: The speaker says information from multiple inference-stack layers—including model choice, API or certified vendor, self-hosting, and inference engine—can be standardized.

“in addition to just the basic job. The chatbots are increasingly seeing... big companies adopting agent platforms where they're building on top of Google's agent studio to”

Claim: The speaker says big companies are increasingly adopting agent platforms and building on Google’s Agent Studio.

“You can host your own managed agents, so play as agent SDK or hosted by Anthropik or Google does both.”

Claim: The speaker says managed agents can be self-hosted, built or run through an agent SDK, or hosted by Anthropic or Google, and that Google offers both options.

“And then these are just ways to kind of strengthen the security guarantees you can make.”

Claim: The speaker says the described agent deployment options can strengthen the security guarantees an organization can offer.

“And in some ways it's kind of bread and butter enterprise security.”

Claim: The speaker characterizes the security measures under discussion as conventional enterprise security in some respects.

“They love to host things on their own premises because it gives them a sense of control.”

Claim: The speaker says enterprise customers often prefer on-premises hosting because it gives them a sense of control.

“I think you'll see just like you do in every other enterprise market, if you really sell to the enterprise, you start to compete on some of these security features. And this is also happening AI, unsurprisingly.”

Claim: The speaker expects AI vendors selling to enterprises to compete on security features, as occurs in other enterprise markets.

“enterprises are really grappling with the thing that makes agents useful is that they're stochastic. And the thing that makes them really hard to adopt is that they're stochastic.”

Claim: The speaker says agents’ stochastic behavior makes them useful while also making them difficult for enterprises to adopt.

“on different sides of that table in part depending on how much the ceo is trying to get the stock price to go up by saying they're ai native and that we must be willing to take the risks we see we actually see phenomenal tension in the heads of the csos of the fortune 1000 where on the one hand you have the ceo saying we must adopt otherwise we're becoming irrelevant and if we fuck up you're fired”

Claim: The speaker says Fortune 1000 security leaders face tension between CEOs’ pressure to adopt AI to avoid irrelevance and the threat of being fired if adoption fails; CEO pressure may also be influenced by a desire to raise the stock price by presenting the company as AI-native.

“And that's kind of like the core emotional tension that we see showing up again and again and again and again.”

Claim: The speaker says the conflict between pressure to adopt AI and fear of consequences for failures is a recurring core emotional tension for enterprise security leaders.

“And one of the core problems that we solve for them is to take that abstract emotional concern and turn it into a framework in some ways just pride and clarity to that concern.”

Claim: The speaker says their organization helps enterprise leaders translate abstract concerns about AI adoption into a clearer framework.

“I think a lot of the concerns that come up there either relate to copyright or there's a lot related to, let's call it broadly safety.”

Claim: Rune Kvist says that concerns about generative media often involve copyright or safety.

“Or like this could be not safe for work or just very graphic materials are kind of some of the core things we have done some work on this. There's a little bit in the standard as well that deals explicitly with that.”

Claim: Rune Kvist says that non-safe-for-work and very graphic material are core safety concerns for generative media, and that AIUC has done some work on these issues and addresses them to some extent in its standard.

“Video, we've not done a lot in yet and I think for proper production that has still. especially power production without a human in the loop that's still got some ways to go as obvious that it's coming but it's very rare that it's like one shot deploy”

Claim: Rune Kvist says AIUC has not done much work on video yet and believes production-quality video generation, especially without human oversight, is not yet mature.

“a video to the internet but eventually that will also happen”

Claim: Rune Kvist predicts that eventually people will publish AI-generated videos online without human review of each video.

“yeah and that just makes complete sense as the technology matures and over time it will become so good that people will not want to slow things down by having a human in the loop and then the need to make promises will drop”

Claim: Rune Kvist predicts that as AI video technology improves, users will increasingly avoid human-in-the-loop review to prevent delays, reducing the need for assurances or promises.

“prediction markets rely on public information.”

Claim: Rune Kvist says prediction markets rely on public information.

“There is not a lot of public information. It's just insiders trading on each side.”

Claim: Rune Kvist says that in some cases public information is scarce and prediction-market trading is left to insiders on both sides.

“That's illegal.”

Claim: Rune Kvist says trading on leaked information is illegal.

“meet this model how capable is it actually prediction markets have not a lot to say because actually just no one knows and so i think that's the core place where some of this breaks down is that actually lots of the world's information that guides some of these high-level decisions is private and often also just not known”

Claim: Rune Kvist says prediction markets provide little insight into the actual capabilities of an unreleased model when the relevant information is unknown.

“meet this model how capable is it actually prediction markets have not a lot to say because actually just no one knows and so i think that's the core place where some of this breaks down is that actually lots of the world's information that guides some of these high-level decisions is private and often also just not known”

Claim: Rune Kvist says much of the information that guides high-level decisions is private and often not known at all.

“that information may not exist anywhere. So prediction markets aggregate existing information. this information may not exist and you want something very specific and you're willing to pay for it that's kind of where third-party audit comes in we also don't really use prediction markets to figure out whether public companies have committed fraud on their books you use audits you probably could but the information is just not that available and if so it would be like just trading on vibes i actually would have been really interesting to see whether prediction markets 2001 were predicted and run”

Claim: Rune Kvist says prediction markets aggregate existing information, whereas third-party audits can help generate information that does not yet exist for a specific question.

“And I think prediction markets will bring lots of new information to it.”

Claim: Rune Kvist predicts that prediction markets will bring in substantial new information.

“So the thing is mostly not like which one is it and more like what are the types of questions that prediction markets are really good at and what are the ones where the information doesn't even exist for insiders such that no one can in fact trade on it and needs to get generated.”

Claim: Rune Kvist says prediction markets are suited to questions with available information, while questions for which even insiders lack information require that information to be generated.

“If you think of the highest level objective as like accelerating secure deployment of agents, then that would totally help. One of the things that happens often now is that folks build agents.”

Claim: Rune Kvist says AI engineer training or certification would help accelerate the secure deployment of agents, if that is the highest-level objective.

“they bring it to the to the decision maker and the decision maker services a bunch of security considerations that they've now thought of and now it's not built to spec now you have to go and re-enter like add these filters etc so if you shifted that left like if everyone knew what the spec they were building to everyone knew the grading scheme”

Claim: Rune Kvist says agent builders often present their systems to decision-makers only to discover security requirements that were not incorporated, forcing later changes such as adding filters.

“they bring it to the to the decision maker and the decision maker services a bunch of security considerations that they've now thought of and now it's not built to spec now you have to go and re-enter like add these filters etc so if you shifted that left like if everyone knew what the spec they were building to everyone knew the grading scheme”

Claim: Rune Kvist says that making the specification and grading scheme known to agent builders in advance would shift security considerations earlier in development.

“we set the grading scheme and i think what's valuable is that if you can turn this into training”

Claim: Rune Kvist says AIUC sets the grading scheme and that converting it into training would be valuable.

“such that people not doing that i think there's value in doing it yep”

Claim: Rune Kvist says training programs aligned with the relevant standard would be valuable.

“basically think it's good for, we will accelerate adoption if we have more people who know how to build secure agents.”

Claim: The speaker believes that having more people who know how to build secure agents will accelerate adoption.

“We are not working on the side of training people at the moment.”

Claim: The speaker's organization is not currently working on training people.

“I think it's like very aligned with our mission.”

Claim: The speaker believes that training people is strongly aligned with the organization's mission.

“We only have so much attention.”

Claim: The speaker's organization has limited attention to allocate.

“I think that's a choice and I think that serves some audiences very well where if you're trying to deploy this into a bank or a hospital, etc. clarity of those boundaries is extremely valuable.”

Claim: The speaker believes that clearly defined boundaries are especially valuable for audiences deploying AI in settings such as banks and hospitals.

“There's lots of other settings where being much more experimental, much more trying it out, it's just a better fit.”

Claim: The speaker believes that a more experimental, trial-oriented approach is better suited to many settings outside highly regulated deployments.

“And so to me, this makes a ton of sense. Also, you'd have to rewrite your curricula every freaking three months.”

Claim: The speaker says curricula for this area would need to be rewritten every three months.

“i think that's exactly right and i think a lot of our work actually goes like we don't want to carry we also don't think ourselves are able to carry the kind of the true north of what it's like secure or not secure but we can coordinate the forum where you listed all that”

Claim: The speaker says the organization's work does not involve claiming that it can define the definitive standard for what is secure.

“secure or not secure but we can coordinate the forum where you listed all that”

Claim: The speaker says the organization can coordinate a forum for people to discuss what is and is not secure.

“and i think there's also you uh it matters a lot what the promises so if the promises hey if you've taken my course you will not fuck up you can't make that promise clearly you could make a promise of like here's the some important things that everyone should at least know”

Claim: The speaker says a course cannot promise that its graduates will never make mistakes.

“and i think there's also you uh it matters a lot what the promises so if the promises hey if you've taken my course you will not fuck up you can't make that promise clearly you could make a promise of like here's the some important things that everyone should at least know”

Claim: The speaker says a course can promise to teach some important knowledge that everyone should have.

“At least the promise changes. Of course, there's some subtlety in how do you communicate this stuff so people really get it.”

Claim: The speaker says that the scope of a course's promise changes and that communicating it clearly requires care.

“But I think it's important to dial in and we have a section in our standard like what is the promise and what is the promise not?”

Claim: The speaker says the organization's standard includes a section defining what its promise does and does not cover.

“Because it's impossible to guarantee that nothing will go wrong.”

Claim: The speaker says it is impossible to guarantee that nothing will go wrong.

“If you need to guarantee that nothing will go wrong, you cannot work with Frontier AI,”

Claim: The speaker says organizations requiring a guarantee that nothing will go wrong cannot work with frontier AI.

“but you can make some claims.”

Claim: The speaker says it is possible to make some claims about frontier AI despite being unable to guarantee that nothing will go wrong.

“In very near term, we've now started to work with some of the frontier companies in each of the categories that are taking off and we'll continue that work to make sure that we cover all of the use cases that are really taking off.”

Claim: The speaker says the organization has begun working with frontier companies in emerging categories and intends to continue until it covers all use cases that are gaining significant traction.

“We see a lot of interest once the first one in the market.”

Claim: The speaker says there is substantial interest after the first company in a market category makes a move.

“moves lots of people want to follow them and we think basically AAC1 will get to a point where all of the Fortune 1000 will organize their risk processes around the standard”

Claim: The speaker predicts that all Fortune 1000 companies will eventually organize their risk processes around AAC1.

“No, we do not have 50% today.”

Claim: The speaker says the organization does not currently have 50% representation.

“I think there is some world where probably by end of year, we might have representation in our consortium for 50% of the Fortune 1000.”

Claim: The speaker thinks it is possible, though uncertain, that by the end of the year the consortium will include representation from 50% of Fortune 1000 companies.

“And then we think the model layer is going to be just brings are now surfacing the concerns that are most likely to slow down adoption of AI.”

Claim: The speaker believes that the model layer is bringing forward the concerns most likely to slow AI adoption.

“hiring across the board across go to market and members of technical staff the people who do really well on our technical team are folks who are really excited about kind of being truly full stack so let's say when we started working with cursor uh we'd never done coding tools before so taking the standard and extending it fleshing out what does frontier evals look like for long horizon coding agents and taking that problem all the way from like working with cursor and other folks in the space down to like fleshing out and shipping a new version of the standard so that's like a truly a full-stack entrepreneur shaped technical people do extremely well at AUC the hard part is building one universal red teamer”

Claim: The speaker says the organization had not previously worked on coding tools when it began working with Cursor.

“hiring across the board across go to market and members of technical staff the people who do really well on our technical team are folks who are really excited about kind of being truly full stack so let's say when we started working with cursor uh we'd never done coding”

Claim: The speaker says the organization is hiring across both go-to-market roles and technical staff roles.

“hiring across the board across go to market and members of technical staff the people who do really well on our technical team are folks who are really excited about kind of being truly full stack so let's say when we started working with cursor uh we'd never done coding”

Claim: The speaker says successful technical-team members at the organization tend to be enthusiastic about working across the full stack.

“tools before so taking the standard and extending it fleshing out what does frontier evals look like for long horizon coding agents and taking that problem all the way from like working with cursor and other folks in the space down to like fleshing out and shipping a new version of the standard so that's like a truly a full-stack entrepreneur shaped technical people do extremely well at AUC the hard part is building one universal red teamer that will act across from Harvey to cursor.”

Claim: The speaker says a major challenge is building a universal red-teaming system that works across products such as Harvey and Cursor.

“tools before so taking the standard and extending it fleshing out what does frontier evals look like for long horizon coding agents and taking that problem all the way from like working with cursor and other folks in the space down to like fleshing out and shipping a new version of the standard so that's like a truly a full-stack entrepreneur shaped technical people do extremely well at AUC the hard part is building one universal red teamer”

Claim: The speaker says the organization developed frontier evaluations for long-horizon coding agents and incorporated them into a new version of its standard.

“and everywhere in between but there's one consistent methodology one consistent taxonomy of what are the risks and the attacks”

Claim: The speaker advocates using a consistent methodology and taxonomy to describe risks and attacks across AI products.

“and making we think that's fundamentally the best way to make consistent promises Jim Morgan is buying both they want to have one framework one consistent way that this comes out and the mechanics of making that happen you get to deal with a lot of the complexity of the real world I think we have good answers in a bunch of that but there's pretty hard engineering problems and finex and Kubernetes”

Claim: The speaker believes that a single consistent framework is the best way to make consistent promises about AI systems.

“and making we think that's fundamentally the best way to make consistent promises Jim Morgan is buying both they want to have one framework one consistent way that this comes out and the mechanics of making that happen you get to deal with a lot of the complexity of the real world I think we have good answers in a bunch of that but there's pretty hard engineering problems and finex and Kubernetes”

Claim: The speaker says implementing the consistent framework involves difficult engineering problems, including work involving Finex and Kubernetes.

“If the decision maker on the buying side, let's say you're the head of risk at a bank and your biggest risk is not encoding or in customer support or whatever the top two biggest use cases, but it's somewhere else, you want to still make sure that that framework has something to say about it to the burning question you have. Otherwise, you'll not earn that trust.”

Claim: A risk framework must address a buyer’s most pressing risk, including risks outside the most common use cases, or it will not earn the buyer’s trust.

“Now, it's true that a lot of the burning questions follow where there's a lot of adoption.”

Claim: Many of the most pressing questions about AI arise in areas with substantial adoption.

“So we today do not cover every single edge, but we have a framework that we can add all of these within.”

Claim: The company’s current framework does not cover every edge case, but it can be expanded to include them.

“We have one global taxonomy of risks and attacks that keep adapting. as like every time a new incident occurs that there's never been seen before great let's go and update the taxonomy so we bake that in so I think we have one coherent universal approach it doesn't mean that we spend equal amounts of time on code and in certain niche use case we do spend time where people where people care we think it's very valuable to have one language”

Claim: The company maintains a global taxonomy of risks and attacks that it updates in response to novel incidents.

“as like every time a new incident occurs that there's never been seen before great let's go and update the taxonomy so we bake that in so I think we have one coherent universal approach it doesn't mean that we spend equal amounts of time on code and in certain niche use case we do spend time where people where people care we think it's very valuable to have one language”

Claim: The company applies a coherent, universal risk approach while allocating attention unevenly toward areas of stakeholder concern.

“I think they're the last businesses to exist beyond the labs will be underwriting.”

Claim: The speaker expects underwriting businesses to be among the last businesses to survive outside AI labs.

“there's one job that the labs can never do for themselves, which is to be their own watchdog.”

Claim: AI labs cannot independently act as their own watchdogs.

“they think a lot about that i think some of the smartest people on these topics work at the labs so the problem is not whether they care uh the problem is that they will all be stuck in a race”

Claim: The speaker sees competitive race dynamics among AI labs, rather than lack of concern, as the central problem.

“they think a lot about that i think some of the smartest people on these topics work at the labs so the problem is not whether they care uh the problem is that they will all be stuck in a race”

Claim: AI labs give substantial thought to advanced-model risks, and some of the smartest people working on these topics are employed by the labs.

“where they might have incentives to cut corners and they might have incentives to withhold information from the government, etc.”

Claim: Competitive pressure among AI labs might incentivize them to cut corners and withhold information from the government.

“one kind of feels like eternal truth is that you need an independent third party to go and inspect that data and share information in this case say with the government it's more of an incentive problem than an interest problem i think they're fundamentally all trying to make this go well correct”

Claim: The speaker argues that an independent third party should inspect AI-lab data and share relevant information with the government because the core concern is incentives, not the labs’ underlying intentions.

“one thing that would make this a qualitative alternative shift is for some definitions of AGI.”

Claim: Under some definitions of AGI, reaching AGI would constitute a qualitative shift.

“it will just get nationalized it'd be a threat to sovereignty and”

Claim: Under some definitions of AGI, reaching that level might lead to nationalization because it would threaten national sovereignty.

“So actually the watchdog is a natural monopoly because if you have race dynamics in watchdogs, then the watchdogs will compete each other to the lowest possible standard.”

Claim: The speaker argues that AI watchdog oversight is a natural monopoly: competition among watchdogs could drive standards down to the lowest possible level.

“The reason why we're very excited about having insurers be on this table is that insurers are the only ones that do not have this dynamic because they pay the bill.”

Claim: The speaker says insurers are valuable participants in AI oversight because, as the parties paying claims, they do not face the same incentive to lower watchdog standards through competition.

“If they keep lowering their prices, they also pay the bill.”

Claim: Insurers bear the cost when they lower prices, because they ultimately pay the bill.

“And this is not true for Moody's where they don't directly pay the bill if they make recommendations that are off.”

Claim: Unlike insurers, Moody’s does not directly bear the cost when its recommendations are inaccurate.

“So we think that balancing factor is pretty important.”

Claim: The speaker considers insurers’ financial stake an important balancing factor in oversight.