Why Dwarkesh is Wrong about Computer Use + How OpenAI shipped its Jev competitor in 1 Week

“are thinking about we have dots which is the new sort of personal assistant product and that has some really exciting computer use features there's GPT 6.1 soul which is this amazing new model that I think is particularly great for computer use because of sort of the cost and speed advantages I think I think we shared that it's a fifth of the cost as Astra and a seventh of the cost”

Claim: OpenAI's new personal-assistant product, Dots, includes computer-use features.

“are thinking about we have dots which is the new sort of personal assistant product and that has some really exciting computer use features there's GPT 6.1 soul which is this amazing new model that I think is particularly great for computer use because of sort of the cost and speed advantages I think I think we shared that it's a fifth of the cost as Astra and a seventh of the cost”

Claim: The speaker considers GPT 6.1 Soul particularly good for computer use because of its cost and speed advantages, and said it costs one-fifth as much as Astra.

“Agents API which now has computer use in it, which is really cool because now developers can build on the same computer use uh that is is part of codecs um and and chat T_V_T_ Um and then there were some demos of our existing computer use features like app shots where you can take the context of something you're doing on your computer and bring it into codecs and chat T_V_T_ really fast.”

Claim: The Agents API includes computer-use capabilities also available in Codex and ChatGPT, which developers can build on.

“asked and um and then like native computer use on your mac where Roman had it taken screenshots of his app uh automatically and he could do other things on his computer while computer use was using his applications um so you have a really exciting keynote decision”

Claim: OpenAI demonstrated native Mac computer use that automatically took screenshots of Roman's applications while he continued using his computer for other tasks.

“So what's really cool about the decisions API is it has all these new capabilities. It does inference in parallel.”

Claim: The Decisions API performs inference in parallel.

“It doesn't have reasoning.”

Claim: The Decisions API does not have reasoning capabilities.

“It's a smaller model than the ones we use for computer use.”

Claim: The model behind the Decisions API is smaller than the models used for computer use.

“And so those capabilities make it really fast.”

Claim: The Decisions API's capabilities make it fast.

“They also make it a little bit less good at doing like long horizon sort of sophisticated tasks.”

Claim: The Decisions API's capabilities make it somewhat less capable at long-horizon, sophisticated tasks.

“And so I think it's I would say it's still an open area of research for how we like bring those approaches together.”

Claim: How to combine the Decisions API approach with the computer-use approach remains an open research question, in the speaker's view.

“That's a really cool product because each dot. It has access to its own Linux virtual computer in the cloud,”

Claim: Each Dot has access to a dedicated Linux virtual computer in the cloud.

“which is different from our other products. You know, traditionally we have access to a browser in the cloud or it has access to your own computer, but now you get your own entire Linux computer in the cloud.”

Claim: Unlike the company's other products, Dots provide their own full Linux computer in the cloud, rather than only a cloud browser or access to the user's computer.

“And so it can run full desktop applications and it can also use a web browser.”

Claim: A Dot's cloud computer can run full desktop applications and use a web browser.

“you know, I think the powerful thing about computer use and the reason why I think it's so exciting is because it makes it so that the agent can do anything. you as a person can do because all the software in the world was designed for humans and now agents can use that same software and you can delegate to the agent so yeah like anything that you would do on a computer you can ask a dot to do yeah I think what particularly useful is going to really depend on who the end user is and what's valuable in their life but yeah I would just start by thinking about like what are the things that you spend time on and how could you delegate those to an agent”

Claim: The speaker argues that computer use lets agents perform anything people can do on a computer because agents can operate the same software designed for humans.

“you as a person can do because all the software in the world was designed for humans and now agents can use that same software and you can delegate to the agent so yeah like anything that you would do on a computer you can ask a dot to do yeah I think what particularly useful is going to really depend on who the end user is and what's valuable in their life but yeah I would just start by thinking about like what are the things that you spend time on and how could you delegate those to an agent”

Claim: The speaker recommends identifying activities that consume one's time and considering whether to delegate them to an agent.

“you as a person can do because all the software in the world was designed for humans and now agents can use that same software and you can delegate to the agent so yeah like anything that you would do on a computer you can ask a dot to do yeah I think what particularly useful is going to really depend on who the end user is and what's valuable in their life but yeah I would just start by thinking about like what are the things that you spend time on and how could you delegate those to an agent”

Claim: The usefulness of computer-use tasks depends on the end user and what that person values in their life.

“and yeah about a flight booking and shopping and honestly even like playing a game or whatever right Totally.”

Claim: The speaker gives flight booking, shopping, and playing a game as examples of tasks an agent could do.

“I've subscribed to a meal prep service because I was trying to like eat healthy, you know, and I really like this meal prep service I found because it lets me customize the meals I order to like a high degree granularity so I can say like.”

Claim: The speaker subscribed to a meal-prep service while trying to eat healthily and valued its highly granular meal customization.

“But it was so complicated, it took me two hours to do an order.”

Claim: The speaker's meal-prep order took two hours because the ordering process was complicated.

“And I found that I could ask computer use to do it for me, and it did it in 15 minutes.”

Claim: Computer use completed the speaker's meal-prep order in 15 minutes after the speaker asked it to do the task.

“So I actually saved two hours. It both did it eight times faster than I could, and it saved me two hours on GPT-6.1 Sol.”

Claim: For the speaker's meal-prep order, computer use on GPT-6.1 Sol was eight times faster than the speaker and saved two hours.

“So those are the kinds of tasks that I feel like are really powerful.”

Claim: The speaker considers tasks like automating a complicated meal-prep order to be powerful uses of computer use.

“Oh, as a creator, I can tell you automatically, immediately, my number one use case is automating YouTube,”

Claim: The speaker's top computer-use case as a creator is automating YouTube tasks.

“because uh YouTube doesn't expose a lot of things via API, and you have to just put it in a VM and just like run it, uh for like let's say let's say they're A_V_ testing feature or making community posts, none of this is available by API, because they hate developers. Um anyway, so uh I”

Claim: YouTube's API does not expose many functions, including A/B testing features and community posts.

“and you have to just put it in a VM and just like run it, uh for like let's say let's say they're A_V_ testing feature or making community posts, none of this is available by API, because they hate developers. Um anyway, so uh I”

Claim: The speaker says YouTube tasks unavailable through the API can be performed by running them in a virtual machine.

“and you have to just put it in a VM and just like run it, uh for like let's say let's say they're A_V_ testing feature or making community posts, none of this is available by API, because they hate developers. Um anyway, so uh I”

Claim: The speaker attributes YouTube's omission of these functions from its API to the company's hostility toward developers.

“Yeah, you know, they said that a few months ago, I think, and I hope they have a different perspective now because computer use is like 180 degrees different”

Claim: The speaker hopes that the person who said computer use had not advanced in the previous two years now sees it differently, because the speaker says computer use is 180 degrees different from how it was.

“I guess I've always been excited about automation and helping people automate tasks because then you can like save time in your life and focus on things that are more important to you than like operating a computer very intricately.”

Claim: The speaker has long been interested in task automation because it can save people time and let them focus on more important activities instead of intricate computer operation.

“That was why we worked on some of those products that was at Apple before. We made a company called Sky.”

Claim: The speaker and colleagues worked on automation-related products at Apple before creating the company Sky.

“We ended up joining OpenAI,”

Claim: The speaker and colleagues eventually joined OpenAI.

“What was really interesting looking back at Sky is we were working on computer use there as well. And the models were so much less capable.”

Claim: When the team worked on computer use at Sky, the models were much less capable than they are now.

“And now the models just in the last one year have become extraordinarily capable of computer use. I think the biggest delta that I see is before they could like reliably start tasks, but then they would run into problems and now they're really good at debugging. They're really good at trying again,”

Claim: Models have become extraordinarily capable at computer use over the last year.

“And now the models just in the last one year have become extraordinarily capable of computer use. I think the biggest delta that I see is before they could like reliably start tasks, but then they would run into problems and now they're really good at debugging. They're really good at trying again, introspecting what is and isn't working.”

Claim: The speaker sees models' improved ability to debug, retry, and assess what is working as the biggest recent change in computer use: models previously started tasks reliably but often ran into problems.

“And I think we've also brought the computer use, the computer use field itself has moved forward. I think we're using more techniques.”

Claim: The speaker believes the computer-use field has advanced partly through the use of more techniques.

“Now computer use often writes code. So if you actually look at it in codecs and you expand the tool calls manually, you can see that it's not just doing one action at a time, it's actually writing JavaScript code that it executes, that the computer executes to perform sometimes many actions at once, which is a great, you know, speed up and great capability. We use more accessibility sort of multimodal interfaces. So the model may use screenshots,”

Claim: Computer-use systems can execute JavaScript that performs multiple actions at once, improving speed and capability compared with one-action-at-a-time operation.

“Now computer use often writes code. So if you actually look at it in codecs and you expand the tool calls manually, you can see that it's not just doing one action at a time, it's actually writing JavaScript code that it executes, that the computer executes to perform sometimes many actions at once, which is a great, you know, speed up and great capability. We use more accessibility sort of multimodal interfaces. So the model may use screenshots, it may use accessibility, it may use play. is playwright um it can use a lot of different mechanisms based on the task at hand um and then yeah the the model acceleration has been has been just amazing so um yeah what's different today I think we're making computers better all the time so I think just like one day's difference um is is probably a little bit less consequential than like even the past month or the past two months um but uh yeah I think the the computer using dot is really exciting as well as”

Claim: Computer-use systems can select among mechanisms such as screenshots, accessibility interfaces, and Playwright according to the task.

“is playwright um it can use a lot of different mechanisms based on the task at hand um and then yeah the the model acceleration has been has been just amazing so um yeah what's different today I think we're making computers better all the time so I think just like one day's difference um is is probably a little bit less consequential than like even the past month or the past two months um but uh yeah I think the the computer using dot is really exciting as well as”

Claim: The speaker considers recent model acceleration remarkable.

“is playwright um it can use a lot of different mechanisms based on the task at hand um and then yeah the the model acceleration has been has been just amazing so um yeah what's different today I think we're making computers better all the time so I think just like one day's difference um is is probably a little bit less consequential than like even the past month or the past two months um but uh yeah I think the the computer using dot is really exciting as well as this that the new model that we came out with.”

Claim: The speaker is excited about the computer-using dot and the newly released model.

“is playwright um it can use a lot of different mechanisms based on the task at hand um and then yeah the the model acceleration has been has been just amazing so um yeah what's different today I think we're making computers better all the time so I think just like one day's difference um is is probably a little bit less consequential than like even the past month or the past two months um but uh yeah I think the the computer using dot is really exciting as well as”

Claim: The speaker thinks a single day's computer-use improvement is probably less consequential than improvements accumulated over the past month or two.

“We actually have a bunch of different ways of measuring it, some of which are on different permutations and configurations of the harness.”

Claim: The team evaluates computer-use performance using multiple methods, including different harness permutations and configurations.

“It's a bit of a complicated story because our production products have some more safety checks and those are configured differently based on the needs of the task at hand.”

Claim: The team's production products include additional safety checks whose configuration varies with the task's requirements.

“So there's a lot of ways to measure it, but I think regardless of how we measure it, we find pretty consistent gains.”

Claim: The team observes fairly consistent computer-use performance gains across its different measurement methods.

“and those gains are sometimes in the harness and sometimes in the model and yeah I was really excited by this result that GPT 6.1 is even more cost-effective for computer use than its baseline cost improvement as compared to Astra it's like really cool to see”

Claim: The speaker says GPT 6.1's cost-effectiveness improvement for computer use exceeds its baseline cost improvement relative to Astra.

“and those gains are sometimes in the harness and sometimes in the model and yeah I was really excited by this result that GPT 6.1 is even more cost-effective for computer use than its baseline cost improvement as compared to Astra it's like really cool to see”

Claim: Computer-use performance gains can result from improvements to either the harness or the model.

“but I think like introducing more modalities has been really powerful.”

Claim: The speaker believes that adding more modalities has been powerful for computer use.

“One more specific example of that is in the past, I think we saw a lot of computer use products had to spend a lot of time like.”

Claim: Earlier computer-use products often wasted time by repeatedly scrolling, taking screenshots, and attempting actions to move through results.

“scrolling you know so it would like take a screenshot it would try to do something it would be like oh i gotta like scroll down to the next page of results and then it would take a screenshot and then we try to do something with scroll down again and so i think um with with accessibility and other and and direct access to the dom and other things like that now the language model can actually see like an entire page or an entire application it can write code that can do multiple steps at once and so i think those have been probably the”

Claim: The speaker considers whole-page or whole-application access and multi-step code execution among the biggest breakthroughs in computer use.

“scrolling you know so it would like take a screenshot it would try to do something it would be like oh i gotta like scroll down to the next page of results and then it would take a screenshot and then we try to do something with scroll down again and so i think um with with accessibility and other and and direct access to the dom and other things like that now the language model can actually see like an entire page or an entire application it can write code that can do multiple steps at once and so i think those have been probably the”

Claim: Accessibility information, direct DOM access, and related methods let a language model inspect an entire page or application and execute multiple computer-use steps through code.

“the biggest single aha moments there's like a lot of tiny ones that are less exciting in comparison but actually we do find also that a lot of speed improvements are driven by like a lot of little paper cuts that we got to go in and introspect yeah”

Claim: Many computer-use speed improvements result from identifying and fixing numerous small usability or engineering issues.

“dumping it out, but also making it token efficient, um uh doing it efficiently, there's like a bit of an art to it. And uh you know it turns out that the same technology that was invented for humans um you know who maybe have accessibility needs who wanna use the screen reader technology, that technology is really helpful for them to be able to use computers. It's also really helpful for L_M_s to be able to use computers, so that's been like really fun to get to work on.”

Claim: Screen-reader technology developed for people with accessibility needs can also help language models use computers.

“include where the link goes, it doesn't include you know maybe you take a screenshot of your calendar that c event ti titles are truncated, you know, but when you take an app shot it gives like the language model like full context about everything and um that lets it uh just sort of like do do much more.”

Claim: An app shot provides a language model with fuller context than a screenshot, including link destinations and untruncated calendar event titles.

“I mean, I think what's really crazy that I think, you know, the team's accomplished over the past couple. past couple months is that now computer use is like faster at accomplishing tasks than like the average human probably in most cases and I think that the next frontier is to have computer use be like literally superhuman in its performance where it actually is as fast or faster at using software than like expert computer users like us and I think that'll be really consequential and exciting when that happens because I think we'll be able to all of a sudden”

Claim: Over the past couple of months, computer-use systems have probably become faster than the average human at accomplishing tasks in most cases.

“past couple months is that now computer use is like faster at accomplishing tasks than like the average human probably in most cases and I think that the next frontier is to have computer use be like literally superhuman in its performance where it actually is as fast or faster at using software than like expert computer users like us and I think that'll be really consequential and exciting when that happens because I think we'll be able to all of a sudden”

Claim: The next frontier for computer use is performance that matches or exceeds expert computer users' speed at using software.

“past couple months is that now computer use is like faster at accomplishing tasks than like the average human probably in most cases and I think that the next frontier is to have computer use be like literally superhuman in its performance where it actually is as fast or faster at using software than like expert computer users like us and I think that'll be really consequential and exciting when that happens because I think we'll be able to all of a sudden”

Claim: The speaker expects superhuman computer-use performance to be consequential and exciting.

“and build products that provide just much more real-time experiences.”

Claim: The speaker expects improved computer use to enable products with much more real-time experiences.

“lowering the barrier to entry of, or the activation energy, I suppose, of using computers, I think will make us start to default to doing certain things in agents that we've become accustomed to doing manually.”

Claim: The speaker expects easier computer use to make people default to agents for some tasks they currently do manually.

“And I think that's exciting also because it'll save us a ton of time.”

Claim: The speaker expects agents to save people a lot of time by taking over tasks they currently do manually.

“different little paper cuts and bottlenecks that are sort of standing in the way of that I think that there is yeah there's things on the model side there's things on the inference side there's things on the harness side there's things in the in the representation you know we find that as computer use gets faster we're increasingly bottlenecked by just like the speed of doing an operation like for example you know a non-trivial amount of time in our benchmarks of computer use tasks is actually like”

Claim: As computer-use systems get faster, the latency of individual operations increasingly becomes a bottleneck.

“different little paper cuts and bottlenecks that are sort of standing in the way of that I think that there is yeah there's things on the model side there's things on the inference side there's things on the harness side there's things in the in the representation you know we find that as computer use gets faster we're increasingly bottlenecked by just like the speed of doing an operation like for example you know a non-trivial amount of time in our benchmarks of computer use tasks is actually like”

Claim: Computer-use performance faces bottlenecks involving models, inference, harnesses, and representation.

“like let's say you're automating a task on doordash.com. Like a lot of the time is actually waiting for doordash.com itself to load, you know?”

Claim: Computer-use benchmarks can spend a substantial portion of task time waiting for websites such as DoorDash to load.

“And you want to get, yeah, actually it's actually really important that you get that, like you want as little delay as possible between when it finally finishes loading and when you go and trigger the LM to do the next action,”

Claim: Computer-use systems should minimize the delay between a page finishing its load and triggering the language model's next action.

“which is actually itself a statistical science.”

Claim: The speaker characterizes deciding when to trigger a language model's next action as a statistical science.

“When possible, you want it to be event driven, JavaScript has load events for, the web browser has load events for web navigation, but there's other types of events that actually really can't be event driven. So there's a lot of complexity that”

Claim: JavaScript and web browsers expose load events for web navigation, but some other event types cannot readily be handled through event-driven triggers.

“I will also say, you know, like, you know, one milestone of computer use that we are at now is, you know, three, four years ago we were scared of hooking up LLMs to the web and to our devices,”

Claim: Three or four years ago, people were afraid to connect large language models to the web and their devices.

“and now I'm having it configure DNS for me. I'm having it pay my bills.”

Claim: The speaker now uses computer-use systems to configure DNS and pay bills.

“First of all, I'm just really excited that we brought computer use into the agents API. I think this is um really great because obviously a lot of developers are building applications that wanna be able to work with third party websites and and and and services and so computer use has this universality to it, it can work with anything. So now all of a sudden, developers can build using the same computer use implementation that we're building on.”

Claim: Computer use can work across third-party websites and services, giving it broad applicability for developers building applications that interact with those services.

“I think there's great work to be done if you wanna build your own computer use harness, but it's hard.”

Claim: Building a custom computer-use harness is possible but difficult.

“And also we train our models on our computer use harness.”

Claim: OpenAI trains its models using its computer-use harness.

“So there is an advantage to using the one that's in distribution for the model.”

Claim: Using the computer-use harness that a model was trained on gives an advantage over using a different harness.

“There actually might be a speed and cost and accuracy advantage.”

Claim: Using the in-distribution computer-use harness might improve speed, cost, and accuracy.

“you know, I think kind of to the point that you were making, like, I think we're all sort of still in the process and maybe like some of us are ahead of. many people in the world of like getting comfortable with this technology and trusting it and so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable by building the right kinds of safety checks by asking for the user's consent before doing something consequential like making a payment by asking you know maybe depending on the application making sure you're only letting it access the websites or applications that it actually needs for the task.”

Claim: People are still becoming comfortable with and learning to trust computer-use technology, with some people further along than others.

“many people in the world of like getting comfortable with this technology and trusting it and so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable by building the right kinds of safety checks by asking for the user's consent before doing something consequential like making a payment by asking you know maybe depending on the application making sure you're only letting it access the websites or applications that it actually needs for the task.”

Claim: Computer-use agents should be restricted to accessing only the websites or applications needed for their task, depending on the application.

“many people in the world of like getting comfortable with this technology and trusting it and so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable by building the right kinds of safety checks by asking for the user's consent before doing something consequential like making a payment by asking you know maybe depending on the application making sure you're only letting it access the websites or applications that it actually needs for the task.”

Claim: Computer-use applications should obtain user consent before taking consequential actions such as making a payment.

“many people in the world of like getting comfortable with this technology and trusting it and so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable by building the right kinds of safety checks by asking for the user's consent before doing something consequential like making a payment by asking you know maybe depending on the application making sure you're only letting it access the websites or applications that it actually needs for the task.”

Claim: Developers should build trust in computer-use technology over time by making reliable products and implementing appropriate safety checks.

“One of my favorite use cases for computer use, actually, and one that we see a lot in the wild, is computer use letting the agent actually test the software that the agent has built,”

Claim: A common computer-use application is having an AI agent test the software that the agent built.

“and then the codex builds it for you and then you have to test it and you are now like QA for the agent right so with computer use you can complete the develop the software development lifecycle where the agent can build software it can test it so I have a lot of fun you know building stuff having the agent test it by the time it comes to me it's already working I have extra fun because sometimes I'm like developing computer use itself and so now I have a computer”

Claim: Computer use can let an AI agent both build software and test it, covering more of the software-development lifecycle.

“So yeah, I I really I really think this is a super powerful class of use case.”

Claim: The speaker considers using computer use to test software built by an AI agent a very powerful use case.

“My name is Nick Grinch. I lead product for the API team. I've been here for roughly three years.”

Claim: The speaker says he leads product for OpenAI's API team and has worked at OpenAI for roughly three years.

“I've been working on launching models. I feel like that's just been like a thing, constant thing throughout my time here at OpenAI.”

Claim: The speaker says launching models has been a recurring part of his work at OpenAI.

“And with every new model, we try to like basically work super close. closely with the post-training team, the research team to figure out what's new in it, um and then we like expose those capabilities in the A_P_I_ uh”

Claim: When a new model is released, the API team collaborates closely with post-training and research teams to identify its new capabilities and make them available through the API.

“So that's like the the basic way of putting it. Um and if you just look at uh everything that's new with GPT-6, the the cool new capabilities that we launched were firstly async function calling.”

Claim: OpenAI launched asynchronous function calling as a new capability with GPT-6.

“Um so what you see with like um uh like a lot of the things that you're seeing in like Codex and Dots and everything is that tool calls take so long that you don't have to like pause the model's execution while uh the tool is running so you could just like kick off a tool call, keep running, keep reasoning and then check back in.”

Claim: Asynchronous tool calling allows a model to continue reasoning while a long-running tool call executes, then check back when it is ready.

“So we launched async tool calling,”

Claim: OpenAI launched asynchronous tool calling.

“we launched like mid-turn steering so now you can like inject messages while the model is reasoning in the middle.”

Claim: Mid-turn steering lets users inject messages while a model is reasoning, and OpenAI launched this capability.

“And then like our main goal in our main goal in the API is to like put things in the API once it's trained into the harness. And so we kind of wait for that moment until it's good enough.”

Claim: The team aims to add capabilities to the API harness once those capabilities have been trained and are good enough.

“And a lot of that is like actually being powered by WebSockets, which we launched a few, I want to say months ago. And so WebSockets just opens this like whole bidirectional like. like communication thing with the model. This is not uh the GPT live thing I'm just talking about, GPT-6. Uh and you can do all these like async tool calling, async reasoning, injecting messages.”

Claim: WebSockets enable bidirectional model communication that supports asynchronous tool calling, asynchronous reasoning, and message injection.

“like communication thing with the model. This is not uh the GPT live thing I'm just talking about, GPT-6. Uh and you can do all these like async tool calling,”

Claim: The speaker distinguishes the WebSockets discussion from the GPT Live feature and says it concerns GPT-6.

“So it's a really fun API to work on I think uh like really enjoying.”

Claim: The speaker finds working on the API really enjoyable.

“the most fun part of UltraFast has been just watching the inference team cook with Astra.”

Claim: The speaker’s favorite part of UltraFast has been watching the inference team work on Astra.

“Like, they're just like constantly having these like codex agents running, trying to like squeeze out more performance. And I would say like at least for a couple of months, a lot of it was focused on efficiency.”

Claim: The inference team uses Codex agents to seek further performance improvements.

“trying to like squeeze out more performance. And I would say like at least for a couple of months, a lot of it was focused on efficiency. and see and driving the cost down which is how we like were able to cut the lunar price by like 80 percent it was like a lot of that was driven by like uh all the the inference improvements they landed and then now they've like shifted gears towards like how can we make this run as fast as possible and so ultra fast has just been like amazing to see on a model like Astra like to go that fast has been really cool and yeah web sockets is like actually was like”

Claim: For at least a couple of months, the inference team focused much of its work on efficiency and cost reduction.

“and see and driving the cost down which is how we like were able to cut the lunar price by like 80 percent it was like a lot of that was driven by like uh all the the inference improvements they landed and then now they've like shifted gears towards like how can we make this run as fast as possible and so ultra fast has just been like amazing to see on a model like Astra like to go that fast has been really cool and yeah web sockets is like actually was like”

Claim: The inference team has shifted from efficiency and cost reduction toward maximizing runtime speed.

“and see and driving the cost down which is how we like were able to cut the lunar price by like 80 percent it was like a lot of that was driven by like uh all the the inference improvements they landed and then now they've like shifted gears towards like how can we make this run as fast as possible and so ultra fast has just been like amazing to see on a model like Astra like to go that fast has been really cool and yeah web sockets is like actually was like”

Claim: Inference improvements contributed to an approximately 80% reduction in Luna’s price.

“The first time we launched WebSockets, it was for a GBD 5.3 Codex Spark, which was, can't believe we named a model that, but you know, that's what we launched it for.”

Claim: The first WebSockets launch supported GBD 5.3 Codex Spark.

“And obviously it helps so much because like, you got to have the tool called, you got to like really reduce the overhead of going back and forth with tools. And so WebSockets is awesome for that.”

Claim: WebSockets reduce the overhead involved in repeated tool interactions.

“It was a great week. I mean, it was the first time we had frontier intelligence at extreme speeds.”

Claim: The speaker says that week marked their first experience offering frontier intelligence at extreme speeds.

“People really liked it.”

Claim: People responded positively to the release, according to the speaker.

“huge props to Diogo and like the Jeff team for like really inspiring the”

Claim: Diogo and the Jev team inspired the team’s work on the new effort.

“whole segment in the market like obviously Jeff comes out everyone's like losing their minds over it our users are like hitting us up but also like our internal teams are like we need like a much faster classification system we can I don't want to like get ahead of some of the the dots features that are going to come but you're going to see like some”

Claim: Jev’s launch prompted users to contact the team, while internal teams requested a substantially faster classification system.

“some cool like really snappy fast things built on top of the decisions API um but you know like yeah a props to Jeff for like inspiring this whole thing uh obviously a bunch of people at open AI get nerd sniped by that and they're like how can we like make this work we're not going to like”

Claim: The team plans to build fast, responsive features on top of the Decisions API.

“some cool like really snappy fast things built on top of the decisions API um but you know like yeah a props to Jeff for like inspiring this whole thing uh obviously a bunch of people at open AI get nerd sniped by that and they're like how can we like make this work we're not going to like train a new model”

Claim: The team did not plan to train a new model for this effort.

“I feel like OpenAI is such a strong like hacker culture and like people are just like they get excited about things and so a guy from inference this one awesome guy from the infra team are like this is amazing, we're going to like hack on it. They build a prototype,”

Claim: An inference engineer and an infrastructure engineer prototyped the idea, and the prototype worked.

“I feel like OpenAI is such a strong like hacker culture and like people are just like they get excited about things and so a guy from inference this one awesome guy from the infra team are like this is amazing,”

Claim: The speaker characterizes OpenAI’s culture as strongly hacker-oriented, with employees readily excited to experiment and build.

“it like works and now we're just like hill climbing on latency and trying to make this as fast as possible and we want to like launch it in the coming days.”

Claim: The team is reducing the prototype’s latency and aims to launch it in the coming days.

“So as soon as we hit our latency target, we'll try to get this out.”

Claim: The team intends to release the prototype after it meets its latency target.

“Yeah, I think the main use cases we've seen is like really fast classification.”

Claim: The team’s primary observed use case for the Decisions API is high-speed classification.

“All the computer use demos have been amazing and really cool.”

Claim: The speaker views the computer-use demonstrations very positively.

“I think there will be limitations, of course, in terms of, you know, having Astra like write like a JavaScript like script to control your computer versus having Luna pick like one action at a time.”

Claim: The speaker expects computer-use limitations when Astra writes a JavaScript control script, compared with Luna selecting actions one at a time.

“I think it's not going to be at the same intelligence level, but like maybe there's some computer use tasks that this is good enough for.”

Claim: The speaker expects this approach to fall short of the same intelligence level, while possibly being sufficient for some computer-use tasks.

“The other cool prototype I've seen internally is people hooking it up with GPD Live.”

Claim: The speaker has seen an internal prototype that connects the system to GPD Live.

“So GPD Live is like, you know, our bi-directional, like, real-time API.”

Claim: GPD Live is the company’s bidirectional real-time API.

“And it's built on this, like, model of front-end models and back-end models.”

Claim: GPT Live uses a front-end and back-end model architecture.

“super fast, yeah, thinker docker thing. So GPD Live is the docker, super fast, really good at delegation.”

Claim: GPT Live is very fast and effective at delegating tasks.

“And you have something like Astra sitting at the back. But tool calling has always felt like really slow in GPD life.”

Claim: The speaker has consistently experienced tool calling in GPT Live as slow.

“And so people have been like putting together these like tool calling demos of GPD live controlling a computer. It just feels like so much more snappy and natural.”

Claim: Demos of GPT Live controlling a computer through tool calls feel substantially more responsive and natural to the speaker.

“So I'm like kind of excited to see like what people do with live and with with with Luna on decisions API. So that'll be pretty exciting.”

Claim: The speaker is excited to see how people use Live and Luna with the Decisions API.

“because a lot of people have been putting out JF clones. There's been about 100 in the last two weeks.”

Claim: Approximately 100 JF clones have been released in the preceding two weeks.

“but like, they can clone a JF API, which is honestly structured outputs,”

Claim: The speaker characterizes cloning a JF API as essentially implementing structured outputs.

“OpenAI was first to, right?”

Claim: The speaker suggests that OpenAI was the first to offer the relevant structured-output capability.

“It is not just latency is not just structured. structured output right because i could just have luna it's the decision model is priced the same as luna right um have turned off reasoning and then have structured output do i have a jeff uh you know no right and that's that's the real There's yeah”

Claim: The speaker argues that a decision model should be evaluated on more than latency and structured output alone.

“I think the way that so we haven't trained like a new model for this. We're like building this purely on top of the same lunar weights that we have.”

Claim: The Decisions API implementation uses existing Luna weights rather than a newly trained model.

“like really just Luna and on top of that what you're doing is you're constraining so like structured output is a big part of it you're really optimizing the the inference stack to like get get very fast on TTFD and because you can have multiple questions what you do is like you basically run those in parallel like As a batch. all these yeah you run those in as a batch you all sorts of like inference techniques people are working on.”

Claim: The Decisions API builds on Luna by constraining outputs, optimizing inference for low time to first decision, and batching multiple questions in parallel.

“want to try to make it as fast as possible but I'd say like at least our implementation of it at the start and in this first version is like zero-shotting this on top of Luna uh to see how it goes and obviously you want to like put it out there like this is opening as like classic iterative deployment thing put it out there see what people think and then like we'll we'll we'll make more model improvements uh as needed um so yeah that's uh that's the system's API.”

Claim: The first version applies the Decisions API zero-shot on Luna and will be deployed iteratively, with later model improvements made if needed.

“Like we get it for free with Luna.”

Claim: The system includes vision capability at no additional cost because it uses Luna.

“I mean, I mean, you need to see how it pans out. You maybe maybe there's like going to be these are going to be like the key areas where we may have to like hill climb with the future model release.”

Claim: Confidence estimation and calibration may be important areas for improvement in future model releases.

“the other thing that's in the debate, obviously nobody knows because Jeff doesn't talk about it, but the two speculations are, one, maybe diffusion model instead of autoregressive. of but you you are able to achieve the parallel uh generation in in in your way and then the other one is some mechinterp type thing that that you're like analysing the activations and then just outputting weights which like you guys have all done the research on this people have speculated”

Claim: The speaker identifies two speculative architectural possibilities: using diffusion rather than autoregressive generation, or analyzing model activations to produce weights.

“it's just like what are you trying to aim for right because you can achieve the API everyone can achieve the API it's actually pretty trivial but like then there's the speed then there's the accuracy then there's the other calibration features I don't know what else”

Claim: The speaker says reproducing the API is relatively easy, while achieving speed, accuracy, and calibration features is the more substantive challenge.

“yeah yeah no totally it's uh it's so cool that this like whole space has been kicked off now and people are going to do so much cool stuff and everyone's going to learn from each other and And yeah, I'm excited about that.”

Claim: The speaker expects substantial activity and mutual learning among participants in the newly emerging space.

“The use cases internally have been pretty obvious, like the user ops team was like jumping on it, we were like, we got to classify all of our support tickets.”

Claim: OpenAI's user operations team quickly identified classifying support tickets as an internal use case for the Decisions API.

“So, you know, like this whole thing started like a week ago. So it's like very, very early and we're”

Claim: At the time of the conversation, the Decisions API effort had begun roughly one week earlier and was still at an early stage.

“And then with the agents API, we have, we're basically like having a bunch of first party products like at OpenAI built fully on top of it.”

Claim: OpenAI has multiple first-party products built entirely on the Agents API.

“We've had the codec security stuff that just went out that's fully built on top of the agents API.”

Claim: OpenAI's recently launched codec security product is built entirely on the Agents API.

“we have sort of we were having like a meetings type of thing launching today I think there was like a demo do you remember like the plug-in extensions when Sam was showing it there was like a demo for like you're in a calendar you can sort of like have your meeting notes jump into a zoom call like a granola type thing”

Claim: At the time of the conversation, OpenAI had a meeting-related feature expected to launch that day, demonstrated as handling calendar meeting notes and joining a Zoom call.

“So I think Agents API and Decisions API, they're like these are our newest products would love like any and all feedback on that to figure out where to take them.”

Claim: OpenAI is seeking feedback on its newest products, the Agents API and Decisions API, to help decide how to develop them.

“On Responses API, which is sort of like our workhorse over here, like really focused on performance right now.”

Claim: OpenAI’s Responses API is its workhorse, and the team is currently focused on improving its performance.

“and the performance comes in like two main ways. Uh first is just like latency, we've been like rewriting the whole responses API's track to like make it as fast as possible from a DTFD perspective, DVD perspective. So there's like that's like continues to be like a main area of focus for us. The second thing we've been trying to do is like really go deep on caching, particularly with these like personal agents that are you know like um uh basically like a single thread that just goes on and on for on forever.”

Claim: OpenAI is focusing on caching for personal agents that maintain a single thread over extended periods.

“and the performance comes in like two main ways. Uh first is just like latency, we've been like rewriting the whole responses API's track to like make it as fast as possible from a DTFD perspective, DVD perspective. So there's like that's like continues to be like a main area of focus for us. The second thing we've been trying to do is like really go deep on caching, particularly with these like personal agents that are you know like um uh basically like a single thread that just goes on and on for on forever.”

Claim: OpenAI is rewriting the Responses API stack to reduce latency and make the API as fast as possible.

“We've been trying to like really up our game on caching. We provide now guarantees of like cache hits within like 30 minutes. We're actually like we for one of our users we just launched like a much longer cache window.”

Claim: OpenAI offers a guarantee of cache hits within approximately 30 minutes.

“So we have like a 12 hour caching guarantee that we offer so that you you have like guaranteed cache hits for a long time.”

Claim: OpenAI offers a 12-hour cache-hit guarantee to at least one user.

“Not yet. That's in preview.”

Claim: OpenAI’s 12-hour caching guarantee is currently in preview and is not yet generally available.

“We're going to like try to get that out to everyone as soon as possible.”

Claim: OpenAI plans to roll out the 12-hour caching guarantee to everyone as soon as possible.

“just be a little bit more for the cash write and we like guarantee um like cash reads for like a much longer period. So even if like your instinct thread for example like y you just like do something on it and then come back to it like three to four hours later, you you still getting the caching performance out of it. You”

Claim: OpenAI’s longer cache window can preserve caching performance when a user returns to a thread three to four hours later.

“Yeah. Yeah. Just like building your apps with like to be very cache aware and sort of like use our prompt diagnostics or cache diagnostics tool to figure out like where things are dropping off and and so the caching part is like really important.”

Claim: Application developers should design their apps to be cache-aware and use prompt or cache diagnostics to find where cache continuity breaks.

“Yeah. Yeah. Just like building your apps with like to be very cache aware and sort of like use our prompt diagnostics or cache diagnostics tool to figure out like where things are dropping off and and so the caching part is like really important.”

Claim: Caching is an important consideration when building applications on the API.

“I also want to talk about pre warming. We have that in the API now. So like if you know that, hey, I'm going to get a cache. like i'm sorry i'm gonna get this prompt i just want to like pre-warm the cache pay like the cash right fee right now and then like have it sort of ready to go for the next 30 minutes for whenever it comes”

Claim: The API supports cache pre-warming, allowing developers to pay the cache-write fee in advance and keep a prompt’s cache ready for use during the following 30 minutes.

“um and so uh yeah like i'm very excited about getting feedback on like the low level performance things that we can keep making responses API the most performant and reliable way to like build on top of an LLM and then you you basically have our like new products where I'm just looking for like any and all feedback”

Claim: OpenAI is seeking feedback on low-level performance improvements to make the Responses API the most performant and reliable way to build on an LLM.

“So firstly, OpenAI has its own like proprietary compression, which is in compression. So in the agents API, it comes built into the harness.”

Claim: OpenAI’s proprietary compression is built into the Agents API harness.

“And if you're in responses API, there's like two ways of doing it. One is what we call server side compaction. Which is you basically tell responses API that if you ever hit this this threshold of tokens Just auto compacted and and and like go back or sorry like reduce the context being used and the second way is like slash compact Which is like if you want full control so you can like slash compact at any time have your own logic on when to like”

Claim: The Responses API supports two context-compaction approaches: server-side compaction and user-controlled compaction through the “/compact” command.

“Which is you basically tell responses API that if you ever hit this this threshold of tokens Just auto compacted and and and like go back or sorry like reduce the context being used and the second way is like slash compact Which is like if you want full control so you can like slash compact at any time have your own logic on when to like”

Claim: Responses API server-side compaction can automatically reduce the active context when a configured token threshold is reached.

“Just auto compacted and and and like go back or sorry like reduce the context being used and the second way is like slash compact Which is like if you want full control so you can like slash compact at any time have your own logic on when to like”

Claim: The Responses API’s “/compact” method lets developers trigger compaction at any time and choose when to do so using their own logic.

“Like uh I don't know but a lot of the the big coding agents like to do it manually I mean like if you look at the codex implementation of it in the codex open source codex harness you can see that they they use slash compact and do it um and uh there's also like new um by the way new compaction techniques that we are working on some of them you will be able to see in the codex harness like it's already implemented in the codex harness and so um uh they're like some file based based systems that we are like experimenting with.”

Claim: OpenAI is developing new context-compaction techniques, some already implemented in the Codex harness.

“Like uh I don't know but a lot of the the big coding agents like to do it manually I mean like if you look at the codex implementation of it in the codex open source codex harness you can see that they they use slash compact and do it um and uh there's also like new um by the way new compaction techniques that we are working on some of them you will be able to see in the codex harness like it's already implemented in the codex harness and so um uh they're like some file based”

Claim: Many large coding agents use manual context compaction, and the open-source Codex harness is an example that uses the “/compact” command.

“based systems that we are like experimenting with.”

Claim: OpenAI is experimenting with file-based systems for context compaction.

“We're obviously like very low level like I used to work at Stripe before this and at Stripe a lot of the game was like building these higher level primitives and products on top of like the core payments primitives and I'm always like curious about what the best way of doing that is in AI and I think we've had a couple of attempts at that we like had launched assistance API like way back in the day and like wasn't really the right fit.”

Claim: The speaker is interested in how to build higher-level primitives and products on top of core AI primitives, analogous to products built on Stripe's core payment primitives.

“We're obviously like very low level like I used to work at Stripe before this and at Stripe a lot of the game was like building these higher level primitives and products on top of like the core payments primitives and I'm always like curious about what the best way of doing that is in AI and I think we've had a couple of attempts at that we like had launched assistance API like way back in the day and like wasn't really the right fit.”

Claim: OpenAI previously launched the Assistants API, but the speaker says it was not really the right fit.

“and we're sort of like going off with this like agent's API and it gives you the codex harness but like where's like the the what's the right amount of flexibility to give in that that's like an open question like how it should be like have memory vaults and like all of these like higher level like API objects to to take away also like to to abstract away more like storage concepts like this is like a whole like”

Claim: OpenAI's Agents API provides the Codex harness.

“and we're sort of like going off with this like agent's API and it gives you the codex harness but like where's like the the what's the right amount of flexibility to give in that that's like an open question like how it should be like have memory vaults and like all of these like higher level like API objects to to take away also like to to abstract away more like storage concepts like this is like a whole like”

Claim: The speaker is considering whether the API should provide higher-level objects, such as memory vaults, and abstract away more storage concepts.

“and we're sort of like going off with this like agent's API and it gives you the codex harness but like where's like the the what's the right amount of flexibility to give in that that's like an open question like how it should be like have memory vaults and like all of these like higher level like API objects to to take away also like to to abstract away more like storage concepts like this is like a whole like”

Claim: The appropriate amount of flexibility to provide in the Agents API remains an open question for the speaker.

“very curious about figuring out how we design. I think a lot of things in AI are just have a low-level API primitive and see an example harness and go and have your coding agent implement that.”

Claim: The speaker says many AI offerings provide a low-level API primitive and example harness for users to implement with a coding agent.

“But how much of that do we build into the APIs, like a constant question that I'm thinking about.”

Claim: The speaker continually considers how much functionality should be built into APIs rather than left to users.