⚡ Building ChatGPT for Finance — Joseph Kim, OpenAI's Head of Financial Services

“I, I really do think that there's gonna be a billion-dollar business built on top of a, a ChatGPT plugin. But ninety-five percent of what we built was something that absolutely a startup could do, um, another company could do.”

Claim: The speaker expects a billion-dollar business to be built on top of a ChatGPT plugin.

“I, I really do think that there's gonna be a billion-dollar business built on top of a, a ChatGPT plugin. But ninety-five percent of what we built was something that absolutely a startup could do, um, another company could do.”

Claim: The speaker says that 95% of what the team built could also have been built by a startup or another company.

“At OpenAI, we ran into problems with pre-training and therefore had difficulty scaling the model. In two thousand twenty-six, we've been able to solve many of these issues.”

Claim: OpenAI says it had solved many of its pre-training and model-scaling issues by 2026.

“At OpenAI, we ran into problems with pre-training and therefore had difficulty scaling the model. In two thousand twenty-six, we've been able to solve many of these issues.”

Claim: OpenAI encountered pre-training problems that made it difficult to scale the model.

“Sure, yeah, I can give a quick thirty-second intro to kind of what we just chatted about. Um, one, I would say, like, we just released a new product, ChatGPT for Financial Services.”

Claim: OpenAI has released a product called ChatGPT for Financial Services.

“This is an enterprise-focused product that's really trying to bring the p- mix the power of ChatGPT, kind of the tools and things that you really, you know, like to use in a consumer product, but also add in the enterprise layer.”

Claim: ChatGPT for Financial Services is an enterprise-focused product that combines consumer ChatGPT tools and experience with an enterprise layer.

“So that's firm-specific templates, um, you know, sources like Daloop, uh, S&P, et cetera. Uh, and so really exciting. Um, we've been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that's been incredible.”

Claim: The speaker says reception to ChatGPT for Financial Services has been incredible.

“So that's firm-specific templates, um, you know, sources like Daloop, uh, S&P, et cetera. Uh, and so really exciting. Um, we've been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that's been incredible.”

Claim: ChatGPT for Financial Services includes firm-specific templates and sources such as Daloop and S&P.

“So that's firm-specific templates, um, you know, sources like Daloop, uh, S&P, et cetera. Uh, and so really exciting. Um, we've been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that's been incredible.”

Claim: OpenAI has worked with Morgan Stanley and Evercore as design partners for ChatGPT for Financial Services.

“So that was kind of the first half of the talk. And then the second half of the talk, really much more, um, forward-looking, so thinking deeply about, um, some of the trend lines that we're seeing in terms of model improvements and drawing those trends line out to the future and trying to think a little bit about, okay, like, what are these models gonna be able to do at the end of, you know, mid '27, end of '27?”

Claim: The speaker’s keynote examined model-improvement trends and projected what models might be able to do by mid-2027 or the end of 2027.

“And what does that mean for folks that are building products on top of that model? And how can you kind of start to plan ahead to the intelligence that's coming in, you know, just a matter of months?”

Claim: The speaker’s keynote considered how product builders could plan for AI capabilities expected to arrive within a matter of months.

“And what does that mean for folks that are building products on top of that model? And how can you kind of start to plan ahead to the intelligence that's coming in, you know, just a matter of months?”

Claim: The speaker’s keynote considered the implications of future model capabilities for people building products on those models.

“Yeah, and it's, and it's a great question. I mean, one thing I will say is that, like, you know, um, our-- all of the training data that we do for financial, um, you know, reasoning analysis, um, we do upstream into what we call the Big Run, which is the, the main kind of run that becomes, you know, something like 6-AshQ or 6-1-AshQ, et cetera.”

Claim: OpenAI puts its training data for financial reasoning and analysis into the main training run, called the Big Run, which produces models such as 6-AshQ or 6-1-AshQ.

“And so even though we're adding a bunch of financial data to the model, um, it's going into the main, main model, and so therefore it's available to folks that wanna build on top of it. And that is something that's very important to us.”

Claim: OpenAI considers access to its financial data-informed main model for external builders very important.

“And so even though we're adding a bunch of financial data to the model, um, it's going into the main, main model, and so therefore it's available to folks that wanna build on top of it. And that is something that's very important to us.”

Claim: OpenAI adds financial data to its main model, making that data-informed model available to people building on it.

“Um, you know, at OpenAI, we really do want that intelligence to be accessible, not only to our products that we're building in-house in first-party ways, but also for folks that want to build startups, build businesses on top of that, as well as, you know, financial firms that wanna integrate that directly into their companies.”

Claim: OpenAI wants its AI capabilities to be accessible through its own products, to startup and business builders, and to financial firms integrating the capabilities into their companies.

“And so, um, we will make all that intelligence accessible there. Um, I think you touched a little bit on product and research, and I think that's, like, really core to what we do at OpenAI.”

Claim: The speaker says research and product are central to OpenAI’s work.

“And so, um, we will make all that intelligence accessible there. Um, I think you touched a little bit on product and research, and I think that's, like, really core to what we do at OpenAI.”

Claim: OpenAI says it will make its AI capabilities accessible to external builders and financial firms.

“Um, internally, we call ourselves a research and deployment company, and so with that-- And we kind of structure that in order on purpose, like research comes first and then deployment comes second. Um, and so as we think about, you know, at the end of the day, trying to change the way that people do knowledge work, we kind of work in that order, right?”

Claim: OpenAI describes itself as a research and deployment company and deliberately prioritizes research before deployment.

“Um, internally, we call ourselves a research and deployment company, and so with that-- And we kind of structure that in order on purpose, like research comes first and then deployment comes second. Um, and so as we think about, you know, at the end of the day, trying to change the way that people do knowledge work, we kind of work in that order, right?”

Claim: OpenAI aims to change how people perform knowledge work.

“We think about the model capabilities, where they're headed, what kind of new things can they enable? So that's one. And then two, we think very deeply about the product f- um, features and surface areas that we should build to make that model intelligence, um, accessible to more people.”

Claim: OpenAI considers which product features and interfaces to build to make model capabilities accessible to more people.

“We think about the model capabilities, where they're headed, what kind of new things can they enable? So that's one. And then two, we think very deeply about the product f- um, features and surface areas that we should build to make that model intelligence, um, accessible to more people.”

Claim: OpenAI considers the direction of model capabilities and the new applications those capabilities may enable.

“And I think you alluded to, to a slide in my com-- my, um, keynote where, you know, we have this chart of Codex adoption within OpenAI. Uh, and it's funny because, uh, you can, you can see the, the software engineering adoption curve, and it's just like up and to the right, pretty steady.”

Claim: The speaker says Codex adoption for software engineering at OpenAI was increasing steadily.

“People are picking it up and, and incorporating it into their daily routines. Um, but if you look at finan- uh, finance, legal, and sales, um, you know, it's pretty much at zero, and suddenly it kind of jumps up to ninety percent.”

Claim: The speaker says people were adopting Codex and using it in their daily routines.

“People are picking it up and, and incorporating it into their daily routines. Um, but if you look at finan- uh, finance, legal, and sales, um, you know, it's pretty much at zero, and suddenly it kind of jumps up to ninety percent.”

Claim: The speaker says Codex adoption in finance, legal, and sales at OpenAI rose from nearly zero to 90%.

“And if you, and if you look at that line and try to see what happened around the time where it jumped up, we, we had released Codex desktop app, um, you know, uh, in February, and that was when that adoption jumped up.”

Claim: The speaker says Codex adoption in finance, legal, and sales jumped when OpenAI released the Codex desktop app in February.

“And so I do think it kind of shows the importance of co-developing those things. And, uh, the, the purpose of the product is to make that model intelligence accessible to as many users as possible.”

Claim: The speaker believes the adoption pattern demonstrates the importance of co-developing AI models and products.

“And so I do think it kind of shows the importance of co-developing those things. And, uh, the, the purpose of the product is to make that model intelligence accessible to as many users as possible.”

Claim: The product is intended to make model capabilities accessible to as many users as possible.

“I would say, like, Codex was really, like, our biggest and main one, and, um, it was the first time we developed a product where the product, the model harness, and the, the research was all kind of under one house and under one roof, and it really gave us a lot of flexibility to, um, go about building things.”

Claim: Codex was OpenAI's biggest and main first-party product, and the first product OpenAI developed with its product, model harness, and research functions together under one organization.

“I think the biggest takeaway that I have from having worked on this for the last six months or so, um, is in enterprise, it's, you know, y-you have to deal with the reality of the industry that you're in and go build around that.”

Claim: Based on about six months working on ChatGPT for Financial Services, the speaker concluded that enterprise products must be built around the realities of their industry.

“But if you ignore the state of the world that you're in when you start building and the forces that are at play within the industry to try... You know, it's either competitive forces, people have their own objectives, and if you just don't think about that, you're not gonna build the right product, right?”

Claim: Ignoring an industry's circumstances and forces, including competition and stakeholders' objectives, can lead to building the wrong product.

“And so we saw that with, um, ChatGPT for Financial Services, and we thought about that a lot. I'll give you one example. Um, data. Data is like a huge part of how we thought about, um, building the product.”

Claim: Data was a major consideration in building ChatGPT for Financial Services.

“On paper it looks bad. It's like you can... They might already have a license for it, so they might feel like they're paying twice for it. Um, and they might have an MCP. And so, like, why are you building this? And the reality is, we went and we tested all these MCPs, and what we found was the error rates were, were really, really high.”

Claim: OpenAI's testing found very high error rates in the MCP connectors it evaluated.

“Some, some of the MCPs that we connected, some big names, 40%-plus error rate on their MCPs, right? And so, um, MCPs is one of those things where it, it looks great on paper, but you have to go see, look at the data and see what's actually going on.”

Claim: Some MCP connectors from major providers that OpenAI tested had error rates exceeding 40%.

“And so for data, we're like, "Hey, look, this is really bad. The only way we can go about fixing this at scale is to bring the data in-house, like, just control that experience end to end." And the m- extra money that you pay on top to bundle that data is worth it, right?”

Claim: The team concluded that fixing the data problem at scale for ChatGPT for Financial Services required bringing data in-house and controlling the experience end to end.

“And so for data, we're like, "Hey, look, this is really bad. The only way we can go about fixing this at scale is to bring the data in-house, like, just control that experience end to end." And the m- extra money that you pay on top to bundle that data is worth it, right?”

Claim: The team considered the additional cost of bundling data into ChatGPT for Financial Services worthwhile.

“So that was a product decision that we had to make that was very industry specific. Um, and it's been resonating with customers. Customers love it. They, you know, the number one thing they call out is the data, and so, um, it worked.”

Claim: Customers have responded positively to ChatGPT for Financial Services, most often highlighting its data.

“But 95% of what we built was something that absolutely a startup could do, uh, another company could do. And we didn't really have to go leverage our ability to go and cha- make changes in the model to go about building fantastic product, right?”

Claim: The team did not need to modify the model to build a fantastic ChatGPT for Financial Services product.

“But 95% of what we built was something that absolutely a startup could do, uh, another company could do. And we didn't really have to go leverage our ability to go and cha- make changes in the model to go about building fantastic product, right?”

Claim: The speaker estimated that 95% of what the team built for ChatGPT for Financial Services could have been built by a startup or another company.

“And so I think it's a great call to kind of mention that piece. Um, what I would say is, like, uh, fewfold. One is most of the product that we built was built on top of the Codex, um, or I should say ChatGPT now, um, plugin architecture and pro- plugin primitives, and that is something that we've been investing a lot of time and energy in.”

Claim: Most of ChatGPT for Financial Services was built using the Codex (now ChatGPT) plugin architecture and primitives, an area in which OpenAI had invested substantial time and effort.

“Um, you know, O- OpenAI is like an interesting company where, you know, we ha- we kind of face the same problems internally that a startup would face, uh, if they were to build on top of OpenAI. So in very, in, in a lot of ways, like, my team is kind of like a startup that's building on top of OpenAI technology, right?”

Claim: OpenAI's internal team faced many of the same problems as a startup building on OpenAI technology.

“And so, um, even internally, we need to find scalable ways for teams like myself to stand on the shoulders of giants and build on top of Codex, right?”

Claim: The speaker's team needed scalable ways to build products on top of Codex.

“Codex is fantastic. Um, it's got great memory. It's got good skills, like there's a, you know, it's persistent.”

Claim: Codex has good memory and skills and is persistent.

“Um, there's a lot of things that are wonderful about Codex, but m- the thesis of my team was Codex was, Codex will get you about 80% of the way to a product that would be successful in the market.”

Claim: The speaker's team believed Codex could provide about 80% of what was needed for a market-successful product.

“But, you know, um, the 20% is missing, and so we need to find some way for my team to go build that 20% to actually go in. 'Cause the other half of the thesis was if you just took Codex and launched it as a financial product, um, it would fail.”

Claim: The speaker's team believed Codex lacked the final 20% needed for a successful financial product, and that launching Codex alone as a financial product would fail.

“And so that 20% is actually pivotal. And so what I would say is we, we went and we built on top of the plugin architecture. It's using the same primitives that are available to any of the builders that want to go build on top of, um, ChatGPT.”

Claim: The speaker considered the missing 20% of the product to be pivotal.

“And so that 20% is actually pivotal. And so what I would say is we, we went and we built on top of the plugin architecture. It's using the same primitives that are available to any of the builders that want to go build on top of, um, ChatGPT.”

Claim: The speaker's team built its product on the ChatGPT plugin architecture and used primitives available to other ChatGPT builders.

“And, uh, you know, I had a line in my, um, presentation, but I, I really do think that there's gonna be a billion-dollar business built on top of a, a ChatGPT plugin, right?”

Claim: The speaker believes a business worth a billion dollars will be built on top of a ChatGPT plugin.

“Like, it's, like you're gonna be able to leverage all of that horizontal functionality and just add your secret sauce on top, and that secret sauce might be data, it might be UI changes, um, it could be an integration.”

Claim: A business built on ChatGPT can use its horizontal functionality and add differentiated value through data, user-interface changes, or integrations.

“But, you know, that secret sauce is really valuable, and I think there's, there's a billion dollars worth of value to be built there.”

Claim: The speaker believes the differentiated value added on top of ChatGPT can be worth a billion dollars.

“So, uh, yeah, it's an interesting time for the industry because we're at a, a time now where orchestration is, uh, uh, is becoming a lot more automated. You know, uh, teams of agents going out and doing things.”

Claim: The speaker says orchestration is becoming more automated, including through teams of agents carrying out tasks.

“Yeah. Um, safety's super, super important. Um, I would say that, like, a- and it's only gonna get more important, uh, because the au- the, you know, the agents are already getting more autonomous and more independent, and I think that trend will be expected to continue, uh, into '27, into '28.”

Claim: The speaker says AI safety will grow in importance as agents become more autonomous and independent, a trend he expects to continue into 2027 and 2028.

“Um, I would say if, uh, for the, for the product that we built and launched last month, um, we, there was a bunch of paradigms that already existed, um, as it pertains to safety, um, and, you know, being thoughtful around auditing the work, making sure that the model's not making a mistake that's gonna impact business outcomes, and we were able to lean heavily into those.”

Claim: For the product launched the previous month, the speaker's team relied on established safety practices such as auditing work and checking that model errors do not affect business outcomes.

“And, and what I would say is the existing product that we launched is a productivity tool for knowledge workers working in the space. And so the idea behind it is there's this, a deep assumption that there's always a human in the loop, right?”

Claim: The speaker describes the team's product as a productivity tool for financial-sector knowledge workers that assumes a human is always involved.

“Like, so the idea there is, you know, uh, y- you're a financial analyst and you have a, a laundry list of things you want to get done that day, and, and just like how software engineers do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT, right? And it might get you 30% of the way there, it might get you 80% of the way there, depending on the task, but there's always this presumption of humans in the loop, uh, that are presiding over that work.”

Claim: For a financial analyst's task, ChatGPT may complete 30% to 80% of the work depending on the task, with a human presiding over the work.

“And, um, that made it very easy for us to, to build a product where we could kind of pass off the responsibility of auditing review, uh, i- into the people's hands and into people's responsibilities.”

Claim: The speaker says the human-in-the-loop approach allowed the team to assign auditing and review responsibilities to people.

“And that worked for us well, and it was easy to communicate. Um, so that was wonderful. I will say we're thinking very deeply about what comes next, and, and what comes next looks different, uh, than, than a productivity tool, um, that sits and, and is always being monitored by a human.”

Claim: The speaker says the team's future product direction will differ from a productivity tool continuously monitored by a human.

“And I think as we think about what comes next, there's kind of two things we're thinking about. One is, how do we kind of shape the security safety profile of the product into something that people understand intuitively?”

Claim: The speaker says the team is considering how to make the product's security and safety profile intuitive to users.

“And some- I would say something that people understand very intuitively is how to work with other people. Um, and so our thesis is if we make the product very similar to how you work with your f- your coworkers, your interns, people around you, that is something that people can pick up and learn very quickly, very easily, and we, and therefore, we need to shape the security profile of that to look like a person.”

Claim: The speaker's team believes users will quickly understand a product modeled on working with coworkers or interns, and therefore intends to shape its security profile to resemble a person's.

“And so there's a few things I'll call out there. One is there's this notion of read/write, uh, you know, even for people. Like, uh, you're a new software engineer at a company, you oftentimes, you know, when I was at Google, you need, you, you need something called readability. And so there's these kind of, like, checks and balances and systems in place before you can start mucking around and touching the system. And so there's already systems in place to monitor the work of people.”

Claim: The speaker says companies use access checks and balances for new employees and have systems for monitoring human work.

“So if mon- if models behave more like people, then you can leverage those systems, and that makes it very easy to kind of talk about and work through this, um, with these financial institutions. So that's something we're leaning into pretty heavily coming up.”

Claim: The speaker argues that if models behave more like people, existing human-work monitoring systems can be leveraged for models, facilitating discussions with financial institutions.

“And the very last thing I would say is, at least in the context of banking, which is where we kind of started, um, there's already this system in place for junior analysts doing a lot of the grunt work. Uh, their work gets passed off to associates. Associates are reviewing the work. Um, then the associates are kind of adding their secret sauce, they're adding their reasoning. That gets passed off to the VP. VP goes up to the director, managing directors.”

Claim: The speaker describes banking as having a review hierarchy in which junior analysts do much of the grunt work, associates review and add reasoning, and work proceeds to senior leaders.

“So there's this kind of like a hierarchical system where there's this implicit work checking that's happening, that's already been built out to stop-- uh, to try to prevent against, uh, mistakes that humans make, right?”

Claim: The speaker says banking's hierarchical review system includes implicit checks intended to prevent mistakes.

“Like mis- humans make just as many mistakes as models, and so those systems are built there. And so we're also leaning heavily in terms of those systems as it c- as it pertains to building financial products.”

Claim: The speaker claims humans make as many mistakes as models and says the team relies heavily on existing human review systems in building financial products.

“And they c- they start with different assumptions. Um, I think, I, I think the notion that like my career is over if you make one mistake is an incorrect notion, and I have a line of thought that like follows that.”

Claim: Joseph Kim considers the idea that one mistake will end a person's career to be incorrect.

“But let me, let's first assume that it's true, right? And, and in the case that it's true, um, what I would say is like not all mistakes are created equally.”

Claim: Even assuming that one mistake can end a career, mistakes differ in how consequential they are.

“And then not all model capabilities are created equally. And so when you think about the mistake that a model or a human can make that could be career-ending, um, you know, it's, it's usually once you get up to a certain level.”

Claim: Career-ending mistakes by models or humans tend to become relevant at a certain level of seniority.

“Like if you're a, you know, VP, the stakes are higher than when you're a junior analyst because there's a presumption of work having been checked up to that point. So if an MD, MD sees a mistake, that's a huge problem. A lot of the safety nets have been, you know, fa- have failed for that mistake to make it up to an MD, right?”

Claim: A mistake reaching a managing director is a serious problem, since multiple safety nets must have failed for it to get there.

“Like if you're a, you know, VP, the stakes are higher than when you're a junior analyst because there's a presumption of work having been checked up to that point. So if an MD, MD sees a mistake, that's a huge problem. A lot of the safety nets have been, you know, fa- have failed for that mistake to make it up to an MD, right?”

Claim: In the speaker's example, a vice president faces higher stakes than a junior analyst because the vice president's work is presumed to have been checked.

“Um, but what I would say is like, you know, the career-ending mistake is something where you get a piece of information factually incorrect, and that piece of factual information changes the conclusion. Like that is-”

Claim: A career-ending mistake can occur when incorrect factual information changes the conclusion.

“... obviously like the kind of the biggest type of mistake that you can make. And what I would say is we're seeing a lot of progress in, in, uh, attention to detail and, you know, the hallucination used to be a big term that used to come up in two thousand and twenty-two, two thousand twenty-three that doesn't come up nearly as often nowadays.”

Claim: The speaker says models are improving in attention to detail and that hallucinations are discussed much less often now than they were in 2022 and 2023.

“I would say that like we're starting to see huge improvements in the reliability of models as it pertains to factual mistakes, so the, the kind of egregious mistake that, uh, would get you fired, models are getting much better at and continue to get much better.”

Claim: Models are improving substantially, and continue to improve, in reliability on factual mistakes, including egregious errors that could get someone fired.

“And so I feel actually quite confident that, uh, a combination of that continual improvement plus the existing system checks in place are-- will make it such that, you know, that type of mistake, the chance of that propagating, uh, up to an MD is just very low probability, right?”

Claim: The speaker is quite confident that continued model improvement, together with existing system checks, will make the probability of a career-ending factual mistake reaching a managing director very low.

“Um, what, what I would say, uh, the models are lacking today, and they still need to improve is actually not the career-ending mistake, um, but it's the, uh, um, it's the, uh, bit that involves human judgment and taste.”

Claim: The speaker says models' main remaining shortcoming is human judgment and taste, not career-ending factual mistakes.

“So it's the, this slide looks ugly, right? And it's, it's not wrong, but it's formatted, like the color's a little bit off. Something about it looks sloppy and therefore, you know, this is not a good slide.”

Claim: A slide may be factually correct yet still be poor work if its formatting, colors, or overall appearance look sloppy.

“And, and that's, that's not a career-ending mistake in the way that, uh, in, in, uh, un- like a factual mistake is, um, but, um, it still matters. It still matters in terms of how useful that work is.”

Claim: Aesthetic mistakes are not career-ending in the same way factual mistakes can be, but they still affect the usefulness of the work.

“And so as we think about the progression going into twenty twenty-seven, what I would say is I expect the model to get much better at the human taste component, and that's what I was kind of alluding to in my keynote is, I think the model will get much better at human taste on things that are softer objective functions.”

Claim: The speaker expects models to improve substantially at human taste by 2027, particularly on tasks with softer objective functions.

“It's not is it right or wrong, it's comparative. Is, is there something that's better than this? And I think models are gonna get much better, uh, along that dimension moving forward.”

Claim: The speaker expects models to improve substantially at comparative judgments of taste in the future.

“It's not is it right or wrong, it's comparative. Is, is there something that's better than this? And I think models are gonna get much better, uh, along that dimension moving forward.”

Claim: Human-taste judgments are comparative assessments of whether one option is better than another, rather than determinations of right or wrong.

“Yeah. Uh, so it's funny. Like very early on in the, like early RLHF days, the only thing we trained models on w- were essentially taste, right? Like there, there was reinforcement learning through human feedback, and the human feedback was always a parallel signal. It was like, "Given two options, which one's better?" And so the early days of reinforcement learning were built on the notion of training on human taste.”

Claim: In the early days of reinforcement learning from human feedback, training primarily used human comparative judgments of which of two options was better.

“Uh, and then obviously things like verifiable rewards and environments and, you know, we built a lot of machinery to do other types of rewards and other types of RL. And so people kind of forgot that a lot, the early days of, of, of reinforcement learning were all around human taste.”

Claim: The development of verifiable rewards, environments, and other reinforcement-learning methods led people to forget that early reinforcement learning focused on human taste.

“And so how do you train on human taste? It's-- I think the technical answer is quite simple. I think the like practical answer obviously is more complicated. Um, the practical answer is you just train a reward model, right?”

Claim: The speaker considers the technical approach to training models on human taste simple, while its practical implementation is more complicated.

“And so how do you train on human taste? It's-- I think the technical answer is quite simple. I think the like practical answer obviously is more complicated. Um, the practical answer is you just train a reward model, right? You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. Um, so like, um, fun fact, like we did this for slides, right?”

Claim: A reward model can be trained for human taste by having people compare pairs of outputs and choose the better one.

“You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. Um, so like, um, fun fact, like we did this for slides, right?”

Claim: The speaker says pairwise human comparisons remain effective and that this method was used to improve slide generation.

“Like, I mean, how do you, how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic, um, judgment, right?”

Claim: Humans are very good at making aesthetic judgments.

“Like, I mean, how do you, how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic, um, judgment, right?”

Claim: Human pairwise judgments of which slide looks better can be used to improve models' presentation generation.

“Actually much better than models.”

Claim: Humans are much better than models at making aesthetic judgments.

“This is something that we discovered, is like a human can look at two slides and within seconds say, "This one looks better." And if you have a model, like a grading, grading model try to go about doing that, um, it's gonna spend a whole bunch of reasoning tokens, this pixel's this way, this thing's that, uh, and then maybe come to the wrong conclusion.”

Claim: Humans can judge which of two slides looks better within seconds, whereas a grading model may use many reasoning tokens and still reach the wrong conclusion.

“And so humans are still th- really valuable in making these models better. And so we rely, we will rely a lot on, uh, human preference, um, to imbue things like taste.”

Claim: Humans remain valuable for improving models, and the speaker expects human preferences to play a large role in giving models taste.

“So that's one. Um, the last thing I'll kind of add is there's precedent for this, which makes us confident that this strategy will work, and that precedent, I would say, is coding.”

Claim: The speaker considers progress in coding a precedent that makes them confident the strategy of combining verifiable rewards and human preference will work.

“Um, the models have gotten-- I would say coding is not fully solved, but like let's say we're eighty percent of the way there, and we're making progress on the, on the last twenty percent. Um, coding is a combination of verifiable rewards and human taste, right?”

Claim: Coding evaluation combines verifiable rewards with human judgments of taste.

“Um, the models have gotten-- I would say coding is not fully solved, but like let's say we're eighty percent of the way there, and we're making progress on the, on the last twenty percent. Um, coding is a combination of verifiable rewards and human taste, right?”

Claim: The speaker estimates that coding is about 80% solved, is not fully solved, and is progressing on the remaining 20%.

“And so the verifiable rewards are, you know, does a, does a code compile? Are there syntax errors? Um, is the output of the function expected? Um, and so there's these verifiable rewards that are in place.”

Claim: Verifiable coding checks include compilation, syntax correctness, and whether a function produces its expected output.

“But a lot of good software engineering is, um, is this clean system design? Did you build a system that's maintainable over time? Um, is it easy to unit test? Is it following paradigms that the vast majority of the community or the, um, industry thinks is the right paradigm?”

Claim: Good software engineering involves clean, maintainable, testable system design and adherence to paradigms widely considered appropriate by the community or industry.

“And if you look at it from that lens, um, financial Uh, a lot of financial work looks very similar. It's the... You know, have an Excel model, does it sum up? Are there reference errors?”

Claim: Some financial work resembles software engineering because Excel models have verifiable checks, such as whether totals are correct and whether reference errors exist.

“There's a whole bunch of verifiable things that we can check to make sure that the work is passing this, you know, first pass sanity check. But there's also a lot of, "Is this the way I would've built this model?"”

Claim: Financial models can be checked with verifiable first-pass tests, but evaluating them also requires judgment about whether the model was built appropriately.

“Um, something I learned about as I started working in finance is there's this notion of, like, the feel of the model, which is, you know, you, you kind of perturb the inputs, and you look at the outputs. And an experienced financial professional can kind of perturb the inputs, look at the outputs, and kind of understand, does the data flow through this model in a way that feels good, right?”

Claim: An experienced financial professional can assess a financial model's feel by perturbing its inputs, observing its outputs, and judging whether the data flows through it appropriately.

“And that is a human preference that looks a lot like, does this code look clean? Is it well-written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.”

Claim: The speaker's success in coding gives them confidence that many analogous evaluation problems in finance can be solved.

“And that is a human preference that looks a lot like, does this code look clean? Is it well-written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.”

Claim: Human judgment of a financial model's feel is analogous to human judgment of whether code is clean and well-written.

“Yeah. Um, I would say that, like, up until date, the vast majority of applications of AI in finance have been really focused on, I would say, like, the junior analysts' work.”

Claim: Up to the time of the conversation, most AI applications in finance had focused on junior analysts' work.

“Um, and I think that's a... And I think that's for two reasons. One is you have to build the product based on the, like, capabilities of the model of today.”

Claim: One reason AI products in finance have focused on junior analysts' work is that products must be designed around current model capabilities.

“And so if you look at the capabilities of the model today, they're really quite good at taking pretty concrete instructions of what to do, uh, and kind of executing, right? So build me a slide, I want it to look this way, um, you know, make it look good, use our template, these types of things.”

Claim: Current models are quite capable of executing concrete instructions, including creating slides to specified requirements and using a provided template.

“The model's quite capable of doing those things, and so therefore, we've built product experiences around that. Um, and the other thing I would call out is, um, there's a strong preference to make the model good at things that humans are not good at or don't like to do, right?”

Claim: There is a strong preference to develop models for tasks that humans are not good at or do not like doing.

“And so, uh, human-- You know, most humans I talk to, there's obviously exceptions, but a lot of the humans I talk to like, um, humans I talk to, people I talk to- ... um, uh, you know, they really don't like building Excel models.”

Claim: Many people the speaker talks to dislike building Excel models, although the speaker acknowledges exceptions.

“It's, it's a tedious kind of work, and it's very detail-oriented. It's easy to get wrong.”

Claim: Building Excel models is tedious, detail-oriented work that is easy to get wrong.

“And similarly with slides. Um, you know, you're, you're copying logos, you're placing them very precisely on a specific part of the page. That's very tedious work that a lot of people don't like to do. And so that's been part of the impetus for us to go and focus on some of these tasks, right?”

Claim: The fact that slide-production tasks are tedious and disliked by many people has helped motivate the speaker's team to focus on them.

“And similarly with slides. Um, you know, you're, you're copying logos, you're placing them very precisely on a specific part of the page. That's very tedious work that a lot of people don't like to do. And so that's been part of the impetus for us to go and focus on some of these tasks, right?”

Claim: Precisely copying and placing logos on slides is tedious work that many people dislike.

“It's a combination of two of those things. There's a, like, ex- the existing molecule is pretty good at, uh, certain types of tasks, and also human preferences of what they like to do and not like to do helps drive the direction of the product as well, right?”

Claim: Product direction is shaped both by the existing model's capabilities and by human preferences about which tasks people like or dislike doing.

“It's a combination of those two things. Um, I think that, like, we've solved a lot of the low-hanging fruit in those two things, and so going into 2027, we're gonna start to move up the stack, right? We're gonna go from the tasks that a junior analyst can do, to some of the tasks that an associate can do, to some of the tasks that an MD would do, and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier.”

Claim: The speaker expects AI systems to move beyond tasks performed by junior analysts toward associate- and managing-director-level tasks starting in 2027, because much of the low-hanging fruit in the two previously discussed areas has been solved.

“We're gonna go from the tasks that a junior analyst can do, to some of the tasks that an associate can do, to some of the tasks that an MD would do, and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier. Uh, it goes from, "This is clearly a good or bad slide," or something that would put in front of a client or not, to, "Well, I don't know if this investment idea is good." And it's, and it would take me, you know, three years of waiting to really be able to give you c- a clear concrete answer of, like, what the market has done.”

Claim: The speaker says objectives become less clear as AI systems move from junior-analyst tasks to higher-level financial work, and evaluating an investment idea may require three years of market outcomes for a clear answer.

“And so the objectives start to become fuzzier. And so the, the way you go about training the model and the way you think about how to make the model better at those things becomes more difficult. It becomes more difficult from a research perspective.”

Claim: The speaker says that fuzzier objectives make model training and improvement more difficult, particularly as a research problem.

“And so, um, but it's not impossible, is what I would say. And so what it takes is a lot of research focus. It takes a lot of wonderful data. It takes a lot of feedback from experts in the field.”

Claim: The speaker believes improving models for higher-level financial tasks is not insurmountable, but requires focused research, high-quality data, and expert feedback.

“We really wanna get the, the, you know, the P ninety-nine point nine type of feedback- ... uh, to make the model, you know, as good as humans. But I don't think it's an insurmountable problem. And I, and what it takes is continued scaling of the base model and focus.”

Claim: The speaker says their goal is to obtain P99.9-type feedback to make the model as good as humans.

“... uh, to make the model, you know, as good as humans. But I don't think it's an insurmountable problem. And I, and what it takes is continued scaling of the base model and focus.”

Claim: The speaker says continued base-model scaling and research focus are needed to improve models for these tasks.

“Focus from the research labs to think that this is an important problem to go solve. And one of the things that I'm viewing, uh, uh, on my team is making this a, an area of focus and priority for OpenAI.”

Claim: The speaker says their team is prioritizing the problem of improving models for higher-level financial tasks at OpenAI.

“And, and I think because of the combination of those two things, I feel confident we'll make a lot of strides in 2027.”

Claim: The speaker expects continued base-model scaling and research focus to produce substantial progress in 2027.

“Yeah, makes sense. Um, let's, yeah, let's talk a little about evals. Uh, evals is, yeah, something near and dear to my heart. Spent a lot of time working on evals. I would say that, um, the, like, um, the prior that I would share with people is building a good eval is incredibly hard.”

Claim: The speaker says building a good evaluation is extremely difficult.

“Peop-- You know, that's not obvious to people who haven't worked on an eval before. Um, to build an eval that actually, like, correlates with the thing that you're actually trying to measure, like the skill that you're trying to measure, capability you're trying to measure, very difficult.”

Claim: The speaker says it is very difficult to build an evaluation that correlates with the skill or capability it is intended to measure.

“Um, because whatever you put together is just a proxy for whatever that thing is, right? And it's just one dimension. So that's one. Two is it's very tedious to get the answer key or the way that you grade this correctly.”

Claim: The speaker says it is very tedious to create a correct answer key or grading method for an evaluation.

“Um, because whatever you put together is just a proxy for whatever that thing is, right? And it's just one dimension. So that's one. Two is it's very tedious to get the answer key or the way that you grade this correctly.”

Claim: The speaker says an evaluation is a proxy for the underlying capability and captures only one dimension of it.

“And if you don't get the way that you get, you grade the answers correctly, it's a eval that's very easy to hack, a very easy reward hack. And therefore, you know, you could score very highly on the eval, um, but actually do really poor on the, on the actual underlying skill or objective that you're trying to measure, right?”

Claim: The speaker says that incorrect evaluation grading makes reward hacking easy, allowing a model to score highly while performing poorly on the underlying skill or objective.

“And so the, the, the, like, prior that I will share is evals are really hard, um, not just in financial work, but just across the board. And, uh, what I would say is, uh, we don't think that there's a really good eval today, um, for financial, um, work.”

Claim: The speaker says evaluations are difficult across domains and that no really good evaluation for financial work exists today.

“Um, there's a bunch of existing prior work. I'll, I'll reference some of the work that we did, which was a GDP eval, um, which is, like, a little bit more broadly in knowledge work. Um, if you guys look at the Astra announcement, we don't talk about GDP eval. Uh, we, we know that the eval is completely saturated, uh, and that it's, uh, there's a, and the pieces that are not saturated are essentially broken, right? And so we actually don't report on that. We don't track it anymore.”

Claim: The speaker says the GDP eval is a broader knowledge-work evaluation that has become saturated, while its remaining unsaturated components are essentially broken; they no longer report or track it.

“And so one of the tricky things for, um, uh, GPT-6 Astra was, um, we didn't talk a whole lot about financial evals. We have some internally that we didn't talk too much about and didn't wanna share at the time yet.”

Claim: The speaker says OpenAI had internal financial evaluations for GPT-6 Astra that it did not discuss in detail or want to share at that time.

“They're coming. Um, but it was-- there was no good way for us to show progress, uh, on financial tasks on these evals. Um, there's a funny, like, funny things. Like, I think there's this bias in evals where if the numbers look good, the researchers assume that the model's gotten better, and they don't investigate it.”

Claim: The speaker says financial evaluations were forthcoming, but at that time there was no good way to demonstrate progress on financial tasks with them.

“They're coming. Um, but it was-- there was no good way for us to show progress, uh, on financial tasks on these evals. Um, there's a funny, like, funny things. Like, I think there's this bias in evals where if the numbers look good, the researchers assume that the model's gotten better, and they don't investigate it.”

Claim: The speaker says researchers may assume a model has improved when evaluation scores look good and fail to investigate further.

“If the model, uh, the, you've made a much more capable model and the eval numbers look bad or haven't increased in the, the scale that you want, then you go investigate, right? So there's this kinda like bias baked into, uh, investigation of these evals.”

Claim: The team investigates an evaluation when a more capable model scores poorly or its scores fail to improve as much as expected.

“If the model, uh, the, you've made a much more capable model and the eval numbers look bad or haven't increased in the, the scale that you want, then you go investigate, right? So there's this kinda like bias baked into, uh, investigation of these evals.”

Claim: The speaker says people investigate when a more capable model's evaluation scores are poor or do not increase as much as expected.

“Something like this happened with GPT-6 where we, we ran GPT-6 on a set of, um, financial evals, and the numbers didn't really change much. We're like, "Oh," like, "what's going on here? Let's take a look at this eval." We, we did a deep dive into these different evals, and lots of things were broken.”

Claim: The speaker says GPT-6's scores changed little on a set of financial evaluations, and investigation revealed that many parts of those evaluations were flawed.

“Something like this happened with GPT-6 where we, we ran GPT-6 on a set of, um, financial evals, and the numbers didn't really change much. We're like, "Oh," like, "what's going on here? Let's take a look at this eval." We, we did a deep dive into these different evals, and lots of things were broken.”

Claim: GPT-6's scores changed little when the team evaluated it on a set of financial tasks.

“Uh, we, like, the-- Basically, we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, um, poorly specified prompts, or just some other flaw with the eval.”

Claim: The speaker says the financial evaluations were saturated, and remaining wrong answers resulted from incorrect answer keys, poorly specified prompts, or other flaws in the evaluations.

“Uh, we, like, the-- Basically, we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, um, poorly specified prompts, or just some other flaw with the eval.”

Claim: The team concluded that its financial evaluations were saturated and that errors in answer keys, prompt specifications, or other evaluation flaws explained the questions the model missed.

“Um, and so all that to say, like, one of the things that my team is working on currently is to try to build a much better eval that actually predicts the ability to automate the work of that specific profession.”

Claim: The speaker says their team is developing an evaluation intended to predict how well AI can automate work in a particular profession.

“Um, and so all that to say, like, one of the things that my team is working on currently is to try to build a much better eval that actually predicts the ability to automate the work of that specific profession.”

Claim: The speaker's team is developing evaluations intended to predict how well a model can automate work in a particular profession.

“And so yeah, and so the idea is, like, we'll have something like investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst, uh, in investment banking.”

Claim: The speaker says a high score on an investment-banking evaluation could indicate that a model can probably automate much of a junior analyst's work.

“And so yeah, and so the idea is, like, we'll have something like investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst, uh, in investment banking.”

Claim: The speaker's team expects a high score on a proposed investment-banking evaluation to indicate that a model could probably automate much of a junior analyst's work in investment banking.

“Yeah. Wonderful. Well, I would say, like, first and foremost, like, please check out ChatGPT for Financial Services. Um, it is an enterprise product, but we're trying very hard to make it available to more, uh, prosumers on the pro plan.”

Claim: ChatGPT for Financial Services is an enterprise product, and OpenAI is working to expand access to prosumers on the Pro plan.

“And so hopefully we'll have some cool announcements there. Um, two, yes, my team is hiring. Uh, we're seeing incredible, uh, customer traction with the product, and, uh, because of that, we're hiring and, and scaling the team rapidly.”

Claim: The speaker's team is expanding its hiring and scaling rapidly in response to strong customer traction for its product.

“And so hopefully we'll have some cool announcements there. Um, two, yes, my team is hiring. Uh, we're seeing incredible, uh, customer traction with the product, and, uh, because of that, we're hiring and, and scaling the team rapidly.”

Claim: The speaker hopes OpenAI will announce further access for prosumers to ChatGPT for Financial Services.

“It's a very diverse team with lots of different roles, and so I just wanna call out a few of them very quickly. Um, there's obviously the, the standard, uh, front-end, back-end, uh, engineering. So, um, kind of traditional product building, so that's one.”

Claim: The speaker's team is hiring front-end and back-end engineers to build its product.

“Two, we're very much looking for wonderful product managers. I think in some ways, uh, great product management is the, the main bottleneck in trying to go win in this space. And so we're trying to hire wonderful product managers in investment banking, equity research, asset management.”

Claim: The speaker believes that strong product management is one of the main bottlenecks to succeeding in financial services AI.

“Two, we're very much looking for wonderful product managers. I think in some ways, uh, great product management is the, the main bottleneck in trying to go win in this space. And so we're trying to hire wonderful product managers in investment banking, equity research, asset management.”

Claim: The speaker's team is recruiting product managers with expertise in investment banking, equity research, and asset management.

“Uh, you know, across the board, we're try- we're trying to hire product managers. Uh, and then the other role that I wanna give a special shout-out to is we're hiring something called a subject matter expert.”

Claim: The speaker's team is recruiting subject matter experts.

“Um, and, you know, it's really important. Uh, one of the things that we did well in, in coding is, you know, the, the people that were training the models and making them better understood what good code looks like, right?”

Claim: The speaker says that coding-model development benefited substantially from trainers and other contributors who understood what constitutes good code.

“And that was a huge tailwind for us as we're making the models better at these things. We don't have that advantage when it comes to making the models, models better at finance. And so we're, we're building out and trying to hire a lot of folks that are able to bring that financial expertise and work with the engineers, work with the researchers, work with the product managers to make the model and the product better in your specific field.”

Claim: The speaker's team is recruiting financial experts to collaborate with engineers, researchers, and product managers on improving models and products for particular financial fields.

“And that was a huge tailwind for us as we're making the models better at these things. We don't have that advantage when it comes to making the models, models better at finance. And so we're, we're building out and trying to hire a lot of folks that are able to bring that financial expertise and work with the engineers, work with the researchers, work with the product managers to make the model and the product better in your specific field.”

Claim: Unlike in coding, the speaker's team lacks an established advantage from contributors who know what high-quality financial work looks like.

“And so if any of those resonates with you, like, I highly re- uh, you know, recommend that you reach out to me. I'm available on LinkedIn. We also have these, uh, job listings posted on our Careers page. Um, it's pretty easy to find.”

Claim: The speaker can be contacted on LinkedIn, and the team's open positions are listed on its Careers page.

“Yeah, we're working very hard to make it available to more people.”

Claim: OpenAI is working to expand access to ChatGPT for Financial Services beyond its current users.