Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

“I've been doing research in generative models for like basically my entire career.”

Claim: Stefano Ermon has conducted research in generative models for essentially his entire career.

“I started at Stanford in 2014 as an assistant professor and I was working on building generative models. Back then, the research area was not particularly”

Claim: Stefano Ermon began working at Stanford as an assistant professor in 2014, where he worked on generative models.

“It's hot, you know, the models were not quite working well.”

Claim: When Ermon began at Stanford in 2014, generative models were not working very well.

“We were still like building little generative models over MNIST and it was like a big success if you could generate these grainy images of digits and, you know, it was even hard to publish papers back then on that topic and you had to kind of like justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because”

Claim: In the early days of generative modeling, producing grainy digit images from MNIST counted as a major success.

“We were still like building little generative models over MNIST and it was like a big success if you could generate these grainy images of digits and, you know, it was even hard to publish papers back then on that topic and you had to kind of like justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because because that was the thing that everybody cared about but then you know the things took over of course and so it was like a i was at the right place at the right time working on the right thing and so i've been doing research in in that space since since the beginning basically yeah”

Claim: In the early days of generative modeling, researchers often had to justify it as a way to learn features from unlabeled data that might improve supervised learning.

“We were still like building little generative models over MNIST and it was like a big success if you could generate these grainy images of digits and, you know, it was even hard to publish papers back then on that topic and you had to kind of like justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because”

Claim: In the early days of Ermon's generative-model research, publishing papers on the topic was difficult.

“i mean i always felt like that was going to be the that was the right way to think about about kind of like learning from unlabeled data that like building a generative model is really the right way to make sure you understand the structure in the data that was kind of like the way I was getting at.”

Claim: Ermon believed that generative modeling was the right approach to learning from unlabeled data and understanding the data's structure.

“I was not even dreaming about the kind of capabilities that these LLMs that we that we have today could could could do that but I was thinking more from I think world models perspective like I was I was working a lot on images and so thinking about”

Claim: While working on images, Ermon thought about generative modeling in terms of world models.

“I was not even dreaming about the kind of capabilities that these LLMs that we that we have today could could could do that but I was thinking more from I think world models perspective like I was I was working a lot on images and so thinking about”

Claim: Ermon did not anticipate the capabilities that today's large language models would achieve.

“like I have a world model, I can imagine what's going to happen if I were to stand up and walk out the door,”

Claim: A world model can represent possible outcomes of an action, such as standing up and walking out the door.

“like I can kind of like picture that in my mind and that's important to make decisions and kind of like model predictive control when having this kind of model of the world requires some generative capabilities.”

Claim: Using a world model for model predictive control requires generative capabilities.

“like I can kind of like picture that in my mind and that's important to make decisions and kind of like model predictive control when having this kind of model of the world requires some generative capabilities.”

Claim: Imagining possible outcomes is important for decision-making and model predictive control.

“And so I always felt like, okay, that's the right direction to work on.”

Claim: Ermon believed generative modeling was the right research direction.

“I felt like this is going to be very hard as a problem. So it's going to keep me busy for my whole career and it's a good problem.”

Claim: Ermon expected generative modeling to be difficult enough to occupy his entire career.

“problem to work on and then of course I was very wrong and things evolved much faster than I was expecting.”

Claim: Generative modeling advanced much faster than Ermon had expected.

“Yeah so I was working on generative models of images initially working on autoregressive models which were very slow and kind of like very blurry and then VAEs and then GANs.”

Claim: Ermon's early autoregressive image-generation models were very slow and produced blurry images.

“and back then we were very unhappy with the state of generative models for images like the GANs were they they worked but they were very unstable to train very hard to reproduce results and so we were trying to see is there a way to build something that is as good as a GAN but it's more principled and so we started working on score-based the generative models which are basically what eventually became”

Claim: Ermon's group found image generative models unsatisfactory because GANs were unstable to train and difficult to reproduce.

“and back then we were very unhappy with the state of generative models for images like the GANs were they they worked but they were very unstable to train very hard to reproduce results and so we were trying to see is there a way to build something that is as good as a GAN but it's more principled and so we started working on score-based the generative models which are basically what eventually became diffusion models back in 2019 with my PhD student Yang Song and so we kind of like came up with this idea of let's train a neural network to denoise images and if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers and that basically became the kind of like underlying technology of”

Claim: Stefano Ermon and his PhD student Yang Song began working on score-based generative models in 2019; this work eventually became diffusion models.

“and back then we were very unhappy with the state of generative models for images like the GANs were they they worked but they were very unstable to train very hard to reproduce results and so we were trying to see is there a way to build something that is as good as a GAN but it's more principled and so we started working on score-based the generative models which are basically what eventually became”

Claim: Ermon's group sought a more principled image-generation method that could perform as well as a GAN.

“diffusion models back in 2019 with my PhD student Yang Song and so we kind of like came up with this idea of let's train a neural network to denoise images and if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers and that basically became the kind of like underlying technology of”

Claim: Ermon and Yang Song proposed training neural networks to denoise images, on the premise that denoising would capture enough image structure to enable generation.

“diffusion models back in 2019 with my PhD student Yang Song and so we kind of like came up with this idea of let's train a neural network to denoise images and if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers and that basically became the kind of like underlying technology of”

Claim: Stefano Ermon's group proposed training neural networks to denoise images, reasoning that the ability to denoise an image would indicate sufficient understanding of its structure to build a generative procedure.

“and back then we were very unhappy with the state of generative models for images like the GANs were they they worked but they were very unstable to train very hard to reproduce results and so we were trying to see is there a way to build something that is as good as a GAN but it's more principled and so we started working on score-based the generative models which are basically what eventually became diffusion models back in 2019 with my PhD student Yang Song and so we kind of like came up with this idea of let's train a neural network to denoise images and if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers and that basically became the kind of like underlying technology of”

Claim: Ermon and his PhD student Yang Song began developing score-based generative models in 2019, which became diffusion models.

“diffusion models back in 2019 with my PhD student Yang Song and so we kind of like came up with this idea of let's train a neural network to denoise images and if you can denoise an image then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers and that basically became the kind of like underlying technology of shift modern diffusion models where instead of generating images you know left to right one pixel at a time you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end and that started yeah back in 2019 in my lab and then it kind of like took over the space and even today the best models for image generation video generation music to some extent a lot of the protein stuff they are based on diffusion.”

Claim: Diffusion models underpin modern image-generation systems, and the best models for image and video generation, along with many protein applications, are based on diffusion.

“shift modern diffusion models where instead of generating images you know left to right one pixel at a time you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end and that started yeah back in 2019 in my lab and then it kind of like took over the space and even today the best models for image generation video generation music to some extent a lot of the protein stuff they are based on diffusion.”

Claim: At the time of the conversation, many leading image- and video-generation models, some music-generation models, and much protein-related work were based on diffusion.

“shift modern diffusion models where instead of generating images you know left to right one pixel at a time you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end and that started yeah back in 2019 in my lab and then it kind of like took over the space and even today the best models for image generation video generation music to some extent a lot of the protein stuff”

Claim: Diffusion models generate images by iteratively refining pure noise into an image, rather than generating pixels sequentially from left to right.

“shift modern diffusion models where instead of generating images you know left to right one pixel at a time you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end and that started yeah back in 2019 in my lab and then it kind of like took over the space and even today the best models for image generation video generation music to some extent a lot of the protein stuff”

Claim: Diffusion models generate images by starting with noise and iteratively refining the result, rather than generating pixels sequentially from left to right.

“And so my group has worked a lot on various kinds of diffusion models, technique for accelerating them to generate samples very quickly, to improve the quality of these models.”

Claim: Stefano Ermon's group has researched multiple diffusion models and techniques for faster sampling and improved model quality.

“And so my group has worked a lot on various kinds of diffusion models, technique for accelerating them to generate samples very quickly,”

Claim: Ermon's group developed techniques to accelerate diffusion models and generate samples more quickly.

“to improve the quality of these models.”

Claim: Ermon's group worked to improve diffusion-model quality.

“And so since we were able to get them to work on images, I started thinking about how do we get diffusion models to work on text or code generation and discrete objects. Like, is there a way to move beyond autoregressive models? goes to something that it's more parallel, more with built-in error correction.”

Claim: After applying diffusion models to images, Stefano Ermon explored using them for text, code, and other discrete objects as a potentially more parallel alternative to autoregressive generation, with built-in error correction.

“And so since we were able to get them to work on images, I started thinking about how do we get diffusion models to work on text or code generation and discrete objects.”

Claim: After developing diffusion models for images, Ermon explored applying them to text, code, and other discrete objects.

“Like, is there a way to move beyond autoregressive models? goes to something that it's more parallel, more with built-in error correction.”

Claim: Ermon explored whether diffusion models could replace autoregressive models with a more parallel generation approach that includes built-in error correction.

“And so I've been doing a bunch of research at Stanford on getting diffusion models to work on text and code generation.”

Claim: Ermon conducted research at Stanford on using diffusion models for text and code generation.

“We had a breakthrough in 2024.”

Claim: Ermon's group achieved a breakthrough in diffusion-model research in 2024.

“We had a breakthrough in 2024. We published a paper basically showing that for the first time it was possible to match the quality of an autoregressive model at the GPT-2 scale. So less than a billion parameters,”

Claim: In 2024, Stefano Ermon's team reported that a diffusion model with fewer than one billion parameters could match an autoregressive model's quality at the GPT-2 scale.

“We published a paper basically showing that for the first time it was possible to match the quality of an autoregressive model at the GPT-2 scale.”

Claim: A 2024 paper by Ermon's group showed that a diffusion model could match an autoregressive model's quality at approximately GPT-2 scale.

“So less than a billion parameters,”

Claim: The diffusion model in Ermon's GPT-2-scale work had fewer than one billion parameters.

“still fairly academic,”

Claim: Ermon described the GPT-2-scale diffusion-model work as still fairly academic.

“but we were able to train basically still a transformer model as a diffusion model on the same data,”

Claim: Ermon's group trained a transformer-based diffusion model on the same data used for the autoregressive comparison.

“but we were able to train basically still a transformer model as a diffusion model on the same data, we were able to match the quality, like the same perplexity, and you were fitting the data just as well as an autoregressive model with the same number of parameters,”

Claim: In the reported comparison, a transformer trained as a diffusion model matched an autoregressive model's perplexity using the same data and number of parameters.

“we were able to match the quality, like the same perplexity, and you were fitting the data just as well as an autoregressive model with the same number of parameters,”

Claim: In Ermon's comparison, the diffusion model matched the autoregressive model's perplexity and data fit using the same number of parameters.

“but the diffusion model was significantly faster.”

Claim: In Ermon's comparison, the diffusion model generated significantly faster than the autoregressive model.

“but the diffusion model was significantly faster. Because it's diffusion, because you're outputting many tokens at the same time, we were able to generate text.”

Claim: In the reported comparison, the diffusion model generated text significantly faster than the autoregressive model by outputting many tokens simultaneously.

“Because it's diffusion, because you're outputting many tokens at the same time, we were able to generate text.”

Claim: The diffusion model generated text by outputting many tokens simultaneously.

“Then X-Faster compared to the auto-aggressive model and so that felt very very exciting and I really wanted to see what happens if you scale up, if you train bigger models and so I started Inception,”

Claim: The faster text generation result motivated Stefano Ermon to investigate scaling diffusion models to larger sizes.

“Then X-Faster compared to the auto-aggressive model and so that felt very very exciting and I really wanted to see what happens if you scale up, if you train bigger models and so I started Inception, a company to basically scale up the technology and then try to build commercial scale diffusion based language models.”

Claim: Stefano Ermon started Inception to scale diffusion technology and develop commercial-scale diffusion-based language models.

“Then X-Faster compared to the auto-aggressive model and so that felt very very exciting and I really wanted to see what happens if you scale up, if you train bigger models and so I started Inception,”

Claim: Ermon was excited by the result and wanted to test how diffusion language models would perform at larger scales.

“Then X-Faster compared to the auto-aggressive model and so that felt very very exciting and I really wanted to see what happens if you scale up, if you train bigger models and so I started Inception, a company to basically scale up the technology and then try to build commercial scale diffusion based language models.”

Claim: Ermon started Inception to scale diffusion technology and develop commercial-scale diffusion-based language models.

“the ways of building generative models, there is autoregressive, where you kind of like have a model that predicts the next token or the next pixel, and then you generate left to right one token at a time, and then there's diffusion, which is a coarse-to-fine generation, kind of like iterative denoising kind of generation,”

Claim: Autoregressive generative models predict and emit one token or pixel at a time from left to right, whereas diffusion models generate through iterative coarse-to-fine denoising.

“and as you said, like we have continuous modalities where diffusion dominate, there is discrete modalities, text and code.”

Claim: Diffusion models dominate continuous modalities; text and code are discrete modalities.

“primarily like all the big labs are kind of like betting on the same architecture autoregressive models and as we move towards more and more like multimodal models and kind of like there is this idea that maybe we'll have a model that can handle all modalities and we'll know everything about the world”

Claim: Major AI labs are primarily betting on autoregressive models as the field moves toward increasingly multimodal models.

“What architecture would that be? Like will it be another regressive model? Will it be a diffusion model? Nobody knows. I think that I think the jury is still out there.”

Claim: It remains uncertain whether a future model capable of handling all modalities will use an autoregressive architecture or a diffusion architecture.

“At inception, we're betting on diffusion models because we believe that what matters eventually will be inference time scaling. And there are fundamental reasons for why diffusion models are better than autoregressive models at inference time.”

Claim: Inception is betting on diffusion models because it believes inference-time scaling will eventually be important; Ermon argues that diffusion models have fundamental advantages over autoregressive models at inference time.

“There was an inflection point in 2017 when people switched from RNNs to transformers, right? The problem was that RNNs had to essentially process tokens sequentially one at a time, and training was very slow.”

Claim: The 2017 shift from RNNs to transformers was driven by RNNs' sequential token processing and slow training.

“And so people came up with this idea of let's have an architecture that allows you to process many tokens at the same time in parallel. And that was a transformer. And that was the thing that scaled better for training and that enabled a lot of the successes behind LLMs.”

Claim: Transformers process many tokens in parallel, scale better for training than sequential RNNs, and enabled many of the successes behind large language models.

“Auto-regressive models are still sequential. The computation is one left to right one token at a time. You cannot generate the 10th token until you've generated everything that comes before it.”

Claim: During inference, autoregressive models generate tokens sequentially from left to right, so a token cannot be generated until all preceding tokens have been generated.

“That kind of workload does not map well to GPUs. That kind of workload is extremely memory bound. You're spending most of your time moving around weights across the memory hierarchy and you're doing very little. a little arithmetic, and that's a fundamental problem of autoregressive models.”

Claim: Autoregressive inference maps poorly to GPUs because it is extremely memory-bound: most time is spent moving weights through the memory hierarchy and little on arithmetic.

“the equivalent at inference time is a diffusion base, is a diffusion model, because a diffusion model is built to have at inference time a workload where you process many tokens at the same time.”

Claim: Diffusion models are designed to process many tokens in parallel during inference.

“And so the workload that we have at inference time in a diffusion model It's basically very, very similar to the workload you have for training, where you're processing many tokens at the same time in parallel.”

Claim: Diffusion-model inference has a workload similar to training because it processes many tokens in parallel.

“And so it's built to essentially have an inference workload that maps really, really well too. Mac models, but the kind of things GPUs do really, really well.”

Claim: Diffusion models have inference workloads that map well to operations GPUs perform well.

“And so we bet on trying to build the architecture and trying to build the kind of models that will scale best at inference time because, you know, economics are dominated by, you know, the kind of intelligence per watt, the intelligence per dollar that you're able to get from the models. If you think about a lot of the advances with the reasoning models,”

Claim: The speaker says the company chose to build architectures and models intended to scale best at inference time because economics are dominated by the intelligence per watt and per dollar those models deliver.

“the intelligence per dollar that you're able to get from the models. If you think about a lot of the advances with the reasoning models, a lot of it is scaling test and compute,”

Claim: The speaker says that many advances in reasoning models come from scaling test-time compute.

“So being able to scale better along that axis will also matter.”

Claim: The speaker expects the ability to scale test-time compute more effectively to matter.

“And even if you think about RL post-training, a lot of the bottleneck is generating rollouts, like letting the model explore.”

Claim: Generating rollouts that let a model explore is a major bottleneck in reinforcement-learning post-training.

“And so inference is again a key bottleneck for RL push training.”

Claim: Inference is a key bottleneck in reinforcement-learning post-training.

“And so if you have a model that scales better at inference time, then automatically you're going to get better scaling during RL push training.”

Claim: A model that scales better at inference time will, according to the speaker, also scale better during reinforcement-learning post-training.

“And so that's why we decided to bet on a diffusion-based LLM because it's inherently more parallel and the bitter lesson is that the more parallel solution is the one. And is the one that is eventually going to win.”

Claim: The speaker says the company chose a diffusion-based LLM because it is inherently more parallel, and predicts that the more parallel solution will eventually win.

“Yeah, so there was a lot of research that went into figuring out how to apply a technology that was inherently very tied to kind of like continuous structure in the data. So if you think about a diffusion model, it's learning how to denoise images and it kind of like makes sense for continuous data because if you think about... even two pixel colors you can kind of like interpolate between them and it will still make sense but if you think about two words there is not necessarily something in between them right it's all discrete and so it required a lot of R&D and a new science that had to be developed to figure out how to how to extend those kind of ideas to discrete spaces.”

Claim: Diffusion models naturally fit continuous data such as images, where pixel colors can be interpolated meaningfully; discrete words do not necessarily have meaningful intermediate values.

“even two pixel colors you can kind of like interpolate between them and it will still make sense but if you think about two words there is not necessarily something in between them right it's all discrete and so it required a lot of R&D and a new science that had to be developed to figure out how to how to extend those kind of ideas to discrete spaces.”

Claim: Applying diffusion-model ideas to discrete spaces required substantial research and development and new scientific methods.

“We think it works really well so we've been able to to train train diffusion-based LLMs that are comparable in quality with the speed optimized models for frontier labs.”

Claim: The speaker says the company has trained diffusion-based LLMs whose quality is comparable to speed-optimized models from frontier labs.

“So our Mercury models are on par with the Heiku models, Flash models, meaning HANA models from OpenAI, if you look at benchmarks, while being significantly faster.”

Claim: The speaker claims that Inception's Mercury models match the benchmark performance of the referenced frontier models while running significantly faster.

“actually use the like we are serving these models in production today we did all the work of figuring out how to even just like build a serving engine right you cannot run this diffusion-based LLMs on the LLM or SG language you have we have to build our own serving engine and we can handle a lot of the complexity of like real production workloads and we've solved all these challenges and we can deliver this kind of new experience yes end to end to to real customers today yeah”

Claim: Inception serves its diffusion-based LLMs to real customers in production and built a custom serving engine because existing LLM serving engines could not run them.

“so it's we are about two years old around 50 people spending a lot of time still on r d kind of like figuring out what's the right way to train these models how to accelerate inference like it's not obvious how you”

Claim: Inception is still devoting substantial effort to researching how to train its models and accelerate inference.

“so it's we are about two years old around 50 people spending a lot of time still on r d kind of like figuring out what's the right way to train these models how to accelerate inference like it's not obvious how you”

Claim: At the time of the conversation, Inception was about two years old and had around 50 people.

“you need to think about an autoregressive model it's pretty clear there's not a lot of things you can do there in terms of like okay you generate one token at a time and that's it in a diffusion based model we know that there is a lot of different possibilities for training compute for quality at inference time like even if you think about image diffusion models or video diffusion models there is a lot of techniques that you can use to kind of like accelerate sampling of distillation or like fancy differential equation solving techniques that allow you to sample very very quickly from these models and so there is a lot of research on the training on the inference and then engineering like just”

Claim: Diffusion-based models allow multiple approaches to allocating training compute and improving inference-time quality, including distillation and differential-equation-solving techniques to accelerate sampling.

“you need to think about an autoregressive model it's pretty clear there's not a lot of things you can do there in terms of like okay you generate one token at a time and that's it in a diffusion based model we know that there is a lot of different”

Claim: Autoregressive models generate one token at a time, leaving relatively few opportunities to alter that generation process.

“possibilities for training compute for quality at inference time like even if you think about image diffusion models or video diffusion models there is a lot of techniques that you can use to kind of like accelerate sampling of distillation or like fancy differential equation solving techniques that allow you to sample very very quickly from these models and so there is a lot of research on the training on the inference and then engineering like just”

Claim: Distillation and specialized differential-equation-solving techniques can substantially accelerate sampling from image and video diffusion models.

“possibilities for training compute for quality at inference time like even if you think about image diffusion models or video diffusion models there is a lot of techniques that you can use to kind of like accelerate sampling of distillation or like fancy differential equation solving techniques that allow you to sample very very quickly from these models and so there is a lot of research on the training on the inference and then engineering like just”

Claim: Diffusion models offer multiple ways to use training compute, improve inference-time quality, and accelerate sampling.

“possibilities for training compute for quality at inference time like even if you think about image diffusion models or video diffusion models there is a lot of techniques that you can use to kind of like accelerate sampling of distillation or like fancy differential equation solving techniques that allow you to sample very very quickly from these models and so there is a lot of research on the training on the inference and then engineering like just Just like thinking about data mixes, evals, RL, post-training infrastructure, like there is a lot of work that needs to happen to figure out how to build the recipes that work for this new model.”

Claim: Developing effective recipes for diffusion models requires substantial work on training, inference, engineering, data mixes, evaluations, reinforcement learning, and post-training infrastructure.

“possibilities for training compute for quality at inference time like even if you think about image diffusion models or video diffusion models there is a lot of techniques that you can use to kind of like accelerate sampling of distillation or like fancy differential equation solving techniques that allow you to sample very very quickly from these models and so there is a lot of research on the training on the inference and then engineering like just Just like thinking about data mixes, evals, RL, post-training infrastructure, like there is a lot of work that needs to happen to figure out how to build the recipes that work for this new model.”

Claim: Developing effective training and deployment recipes for diffusion-based language models requires work across data, evaluation, reinforcement learning, post-training infrastructure, training, inference, and engineering.

“And we try to leverage existing things as much as possible. uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be”

Claim: Inception tries to use existing resources, including public datasets, evaluations, and benchmarks, as much as possible.

“And we try to leverage existing things as much as possible.”

Claim: Inception aims to reuse existing tools and work wherever possible.

“it's still a transformer-based model, so you don't have to throw away a lot of the work that has been done on good architectures. We still use attention.”

Claim: Diffusion-based language models can remain transformer-based and use attention, allowing them to benefit from existing architectural work.

“it's still a transformer-based model, so you don't have to throw away a lot of the work that has been done on good architectures.”

Claim: Inception can build on much of the existing work on effective architectures because its model is transformer-based.

“it's still a transformer-based model, so you don't have to throw away a lot of the work that has been done on good architectures. We still use attention.”

Claim: Inception's diffusion-based language model is transformer-based and uses attention.

“uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be”

Claim: Inception focuses on work where it can have the greatest impact and differentiation, and the speaker says its current differentiator is speed.

“uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be”

Claim: Inception reuses publicly available datasets, evaluations, and benchmarks, while concentrating on work where it can have the greatest impact and differentiation.

“uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be”

Claim: At the time of the conversation, Inception's primary area of differentiation was speed.

“uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: A diffusion-based language model might eventually be significantly more intelligent than an autoregressive model.

“uh we still use a lot of the public data sets that people have created and evals and benchmarks so you know we're a startup we try to be scrappy we try to use existing things as much as possible and kind of like focus on the on the the aspects where we can have the highest impact and then where we can be the most differentiated and right now it's speed in the future who knows like it's possible that a diffusion based language model will be be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: The speaker says it is possible, but unknown, whether a diffusion-based language model will eventually be significantly more intelligent than an autoregressive model.

“be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: It is not known whether diffusion-based language models will become more intelligent than autoregressive models.

“be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: Efficiency will eventually be very important in AI systems.

“be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: The speaker expects efficiency to become very important for AI systems.

“be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: The speaker doubts that the best way to build powerful AI systems has been discovered and believes alternative model-building approaches remain to be found.

“be maybe significantly more intelligent than an autoregressive one like nobody knows that that's why i think this is very exciting because we're developing these really powerful ai systems but it's all very fresh it's all very new uh i doubt we've discovered the best way of building these systems there's got to be alternative there's got to be other ways of building these models and and eventually yeah efficiency will be very important like if you think about the ai factor”

Claim: The speaker doubts that the best way to build AI systems has been discovered and believes alternative approaches remain to be found.

“factory like how is that going to work I think nobody really knows and just being able to play in that space and thinking about alternative ways of creating intelligence I think it's exciting”

Claim: The speaker says it is not yet clear how the AI factory will work.

“factory like how is that going to work I think nobody really knows and just being able to play in that space and thinking about alternative ways of creating intelligence I think it's exciting”

Claim: The speaker believes that how an AI factory will work is not yet well understood.

“factory like how is that going to work I think nobody really knows and just being able to play in that space and thinking about alternative ways of creating intelligence I think it's exciting”

Claim: The speaker considers exploring alternative ways to create intelligence exciting.

“I just think if you if you fundamentally believe that we can use all of the power we have and there will be economic reasons to do so, then I think the focus on efficiency is going to go dramatically up.”

Claim: If people believe that all available power can be used and that economic incentives will support its use, then attention to efficiency will increase dramatically.

“You know, I think many people now argue like one of the most important decisions for a company in the AI space is like, how do we use the power we have between, you know, training and the most valuable use cases for any watt on that topic like here in.”

Claim: Many people argue that a key decision for an AI company is how to allocate available compute power between training and the use cases that provide the greatest value per watt.

“Yeah, it is basically applications where latency matters.”

Claim: Speed is especially valuable in applications where latency matters.

“I think everyone cares about speed in the sense that if you can give me the same quality but faster, people will always pick the faster solution.”

Claim: People will choose a faster solution when it provides the same quality as a slower one.

“we're seeing it with these like faster versions of even the models from Frontier Labs people are willing to pay more to get access to faster models right and I think once you get used to a fast model it's hard to go back it's kind of like broadband right and then you get faster and faster and if you were able to try even the people cannot go back once you once you get the fast model”

Claim: Customers are willing to pay more for access to faster versions of models from frontier AI labs.

“we're seeing it with these like faster versions of even the models from Frontier Labs people are willing to pay more to get access to faster models right and I think once you get used to a fast model it's hard to go back it's kind of like broadband right and then you get faster and faster and if you were able to try even the people cannot go back once you once you get the fast model”

Claim: Users who become accustomed to a fast model find it difficult to return to a slower model.

“There are a few that we can mention, like in the voice space, for example, open call is an example. They are building like voice agents.”

Claim: OpenCall is building voice agents.

“of course, the pipeline is like you have an ASR model, you have an LLM that it's kind of like doing all the tool calls and figuring out what to say next. And then there is a text to speech. component at the end.”

Claim: A typical voice-agent pipeline uses an ASR model, an LLM for tool calls and response decisions, and a text-to-speech component.

“It has to be a reasoning LLM typically to have the highest level of quality.”

Claim: A voice-agent pipeline typically needs a reasoning LLM to achieve the highest quality.

“Speed matters a lot to them.”

Claim: Speed is highly important to OpenCall.

“They were previously using serving their LLMs on Cerebras. So they were using custom chips to get to the kind of speed that they need to deliver the best experience to their customers.”

Claim: OpenCall previously used Cerebras custom chips to serve its LLMs at the speed it needed for a good customer experience.

“And then they switched over to our diffusion-based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model. on custom hardware.”

Claim: OpenCall switched to Inception's diffusion-based LLMs because they could provide approximately the speed of an autoregressive model running on custom hardware.

“If you have a diffusion-based LLM that it's built to be parallel, it's accelerating at the software level,”

Claim: Designing a diffusion-based LLM for parallelism can increase inference speed through software-level acceleration.

“then you can get the same speed on NVIDIA GPUs, which means much more availability. I mean, GPUs are scarce, but there's more of that than custom chips and lower cost, higher quality.”

Claim: Software-level acceleration can achieve comparable speed on NVIDIA GPUs, which are more available and lower-cost than custom chips, while providing higher quality.

“And I think hardware is one way to accelerate things.”

Claim: Hardware can be used to accelerate AI systems.

“exactly. And especially they are complementary. That's the exciting piece is that to some extent, the gains that you get from the software, they are multiplicative. applicative with the gains you get from the hardware for”

Claim: Software and hardware acceleration can complement each other, with their performance gains multiplying.

“Yeah, it's a very valid point and something that is also like top of mind for us.”

Claim: Stefano Ermon considers it a valid, top-of-mind concern that large players with the resources to scale compute may absorb advances in AI architectures or methods.

“I think initially for us, for sure, like the moat is sort of like the IP, the trade secrets, like the ideas that we and our researchers have to build these models and make them better.”

Claim: Initially, Inception sees its intellectual property, trade secrets, and researchers' ideas for building and improving models as its competitive moat.

“we've developed a product and we have real customers and we are getting feedback on on the models from from the real world is that by doing that we are also like developing components that are also very important to to deploy these models for example like a serving engine you know if you don't have the serving engine you can't really serve these models in production and so by forcing ourselves from the very beginning to go out”

Claim: Inception has developed a product, serves real customers, and receives real-world feedback on its models.

“and deploy something end-to-end we're learning a lot about how to serve these models and how to build software that it's kind of like needed to run these models and that again becomes IP like even if you train the diffusion-based LLM if you don't have the serving engine if you don't have the VLLM equivalent to serve it you're still stuck and you still cannot use it.”

Claim: By deploying models end-to-end, Inception learns how to serve them and build the software needed to run them, creating intellectual property.

“and deploy something end-to-end we're learning a lot about how to serve these models and how to build software that it's kind of like needed to run these models and that again becomes IP like even if you train the diffusion-based LLM if you don't have the serving engine if you don't have the VLLM equivalent to serve it you're still stuck and you still cannot use it.”

Claim: A diffusion-based LLM cannot be used in production without a serving engine, such as a vLLM equivalent.

“On the same line, like along the same lines, we are working with real customers and we're getting feedback on the models. We figure out what works, what doesn't. We collect data sometimes from them.”

Claim: Inception uses customer relationships to gather model feedback, identify what works and does not, and sometimes collect customer data.

“We figure out what works, what doesn't. We collect data sometimes from them. create evals based on what they're seeing and so that again becomes part of the of the technical mode because you know of course those things are a little bit harder to replicate”

Claim: Inception considers customer-informed evaluations part of its technical moat because they are somewhat harder to replicate.

“Yeah, it's a good question. And fundamentally, if you think about whenever you train a generative model, what you're doing is whether it's an autoregressive model or to some extent even a diffusion model is. you are trying to identify structure in the data by essentially building a compression scheme.”

Claim: Generative models identify structure in data by learning a compression scheme; this applies to autoregressive models and, to some extent, diffusion models.

“but whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data.”

Claim: Generative models seek common structure in data by learning an efficient compression scheme.

“And so the more you can compress the data, the more structure, the more patterns you're identifying, and that's how these models work,”

Claim: In the speaker's account, greater data compressibility corresponds to identifying more structure and patterns in the data.

“which is the amazing thing. It's just like by predicting the next word. you are learning something about the structure of the data.”

Claim: Next-word prediction enables a model to learn aspects of the data's structure.

“And that's the same whether you're using a diffusion model or you're using an autoregressive model. Both methods are essentially trying to learn a compression scheme.”

Claim: Both diffusion and autoregressive models attempt to learn a compression scheme.

“And when I mentioned the original 2024 paper, when we showed that we are achieving parity with autoregressive models, the metric that we're using is basically perplexity, which is a notion of motion of how much structure have you identified in the data and so even though it might not seem obvious we were actually able to identify at the GPT-2 scale”

Claim: In its original 2024 paper, Inception reported parity with autoregressive models on perplexity, which Ermon described as a measure of how much structure a model identifies in data.

“motion of how much structure have you identified in the data and so even though it might not seem obvious we were actually able to identify at the GPT-2 scale the same amount of structure as an autoregressive model yeah”

Claim: At GPT-2 scale, Inception reported that its diffusion model identified as much data structure as an autoregressive model.

“there is noise in everything”

Claim: All data contains noise.

“and so to the extent you know the numbers don't lie to the extent that you are able to drive different But if it flexes down, then it means that you can actually build like a compression scheme and that will get you that sort of like level of compression. And so the structure must be there and the model must have been able to uncover it.”

Claim: Ermon argues conditionally that if a model's perplexity decreases, it can achieve the corresponding compression level, indicating that structure exists in the data and the model has uncovered it.

“And then it's more a question of an inductive bias, like, you know, is a transformer a better way of identifying those patterns or something else?”

Claim: Once the relevant structure is present, selecting an inductive bias involves determining empirically whether transformers or other approaches identify the patterns better.

“Is next token prediction the right modeling framework or is it more like. The noising and that's that's very much an empirical question that I think at the moment we don't have a tools even to understand”

Claim: Ermon says it remains an empirical question whether next-token prediction or a noising-based framework is preferable, and that current tools are insufficient to understand the question.

“Yes, that's a key value proposition. And one of the things that they always look into is, you know, yeah, to what extent a lot of the value they provide is like the harness and making sure that the models indeed are doing the right thing.”

Claim: Ermon identifies the model harness and its role in ensuring models behave appropriately as a key value proposition for AI products.

“And the interesting thing about a diffusion-based LLM is that we've built everything to be backwards compatible. So it's still like the API is the same. It's the same, it's still OpenAI compatible, text in, text out.”

Claim: Inception's diffusion-based LLM is designed to be backwards compatible and retain an OpenAI-compatible API with text input and text output.

“And it so happens that the models we've trained are good at following instructions, they're good at outputting, you know, if you're using JSONs, like structure outputs, they can handle all of those things.”

Claim: Inception says its trained models follow instructions well and can produce structured outputs, including JSON.

“And it was good enough, it was better in fact that the models they were using before.”

Claim: Ermon says Inception's models were good enough for the customers' needs and better than the models those customers had previously used.

“And so they are still able to provide the kind of like level of service to their customers by using. thing mercury but”

Claim: Ermon says customers can maintain their level of service to end users by using Inception's model.

“it could be that that I think that's actually a very interesting point is that we know that diffusion models are typically easier to control compared to autoregressive models and the reason is that if you think about an autoregressive model you kind of like have to wait until you've generated the full object to know whether or not it satisfies let's say a constraint or whether or not whether it's aligned or whether or not whatever you know it's on brand whatever it is that the objective function that you care about maybe you're generating a molecule and you care about solubility and then and then you kind of like have to wait until you have the full object to be able to score it with some reward function but a diffusion model it's more course to find generation so from the very beginning you know kind of like is this object the kind of thing I want or not and you can steer this”

Claim: Diffusion models are typically easier to control than autoregressive models because they can be evaluated and steered progressively during generation, while autoregressive outputs generally must be completed before they can be scored against constraints or objectives.

“clear the generation in the direction provided by an external reward function or some set of constraints and so there is a lot of evidence in the academic literature at least that diffusion models are easier to control and there are different ways of steering them that are just not possible without the regressor models so that would be a different interface for the model that maybe might not be even available for other regressor models i think that's something that”

Claim: The speaker says academic literature provides substantial evidence that diffusion models are easier to control and support steering methods unavailable to autoregressive models.

“that's the thing we don't we don't know right and that's the that's”

Claim: The speaker does not know what capabilities Inception's models may have at scale beyond performance.

“the exciting it's emergent yes yes like right now the wedge speed oh we know they are much faster that was the initial bet because that was easier to test it was like easy to you know, to measure and it's obviously valuable,”

Claim: The speaker says diffusion models are much faster, and that speed was the initial bet because it was easier to test and measure and was valuable.

“But as we, that's why I find it so exciting is that as we learn more about how to train these models, you know, we don't know what we're going to find.”

Claim: The speaker says it is not yet known what they will discover as they learn more about training diffusion models.

“And there is a decent amount of evidence in the academic literature, for example, that diffusion-based models are more data efficient compared to autoregressive models.”

Claim: Academic literature provides a decent amount of evidence that diffusion-based models are more data-efficient than autoregressive models.

“you're learning by denoising. You start with an image, you add noise, and then you learn how to remove the noise.”

Claim: A diffusion model is trained by learning to denoise data, such as adding noise to an image and learning to remove that noise.

“And so it's effectively doing data augmentation in the sense that the same image is augmented by many noisy views.”

Claim: Diffusion training effectively performs data augmentation by creating many noisy views of the same image.

“And so they tend to be a little bit more data efficient.”

Claim: Diffusion models tend to be somewhat more data-efficient.

“And so if that, you know, holds up at scale and you believe that maybe we'll get into tasks, where data becomes the bottleneck, then it becomes more interesting,”

Claim: If diffusion models' data-efficiency advantage holds at scale, it could be especially valuable for tasks where data is a bottleneck.

“That's why it's so exciting because things are changing and then this technology is so important and so valuable that having something differentiated, I think will create value.”

Claim: The speaker thinks that because this technology is important and valuable, a differentiated offering will create value.

“group in an open router has this very nice way of kind of like looking at all the different use cases and you can kind of see you know the research and conversational and coding and software engineering and law processing law like they have like a nice taxonomy basically of tasks and I was doing some estimates and I think there is like Between 20 and 30 percent where latency is really, really important.”

Claim: The speaker estimates that latency is very important for 20–30 percent of workloads.

“And so at the very least, as a lower bound, I think it could be addressable by models that are within a given latency budget, they would give you the highest possible quality.”

Claim: The speaker estimates that at least some latency-sensitive workloads could be served by models that maximize quality within a specified latency budget.

“it's a different stack.”

Claim: Diffusion models require a different technology stack from the one the speaker is contrasting them with.

“until one of the challenges that we had to build are a lot of things in-house and there is not a mature sort of like ecosystem of anything about the serving engine or like kernels like a lot of the things had to be developed in-house and”

Claim: The speaker says a challenge in working with diffusion models was building many components in-house because no mature ecosystem existed for serving engines or kernels.

“so there is not really anything open source or there are some open source models but they're not particularly good”

Claim: The speaker says that although some open-source diffusion models exist, they are not particularly good.

“good and so that makes it a little bit more difficult to you know deploy to get customers to try things they're not used to it so that's that's been one of the challenges”

Claim: The speaker says limited open-source options and customer unfamiliarity make deployment and customer trials more difficult.

“So we have to build our own stack for doing a... SFT if we're doing RLHF, doing RL, I mean that becomes IP to some extent so it's one of the reasons we decided not to open source everything was really to keep the IP a little bit closer to us and not opening it but then there are downsides like there is less less an opportunity for the community to contribute it's harder to adopt it's hard to do on-prem kind of like deployments and so on.”

Claim: The speaker says the company must build its own stack for supervised fine-tuning and reinforcement-learning workflows, including RLHF.

“SFT if we're doing RLHF, doing RL, I mean that becomes IP to some extent so it's one of the reasons we decided not to open source everything was really to keep the IP a little bit closer to us and not opening it but then there are downsides like there is less less an opportunity for the community to contribute it's harder to adopt it's hard to do on-prem kind of like deployments and so on.”

Claim: The speaker says the company chose not to open-source everything in part to retain closer control over its intellectual property.

“SFT if we're doing RLHF, doing RL, I mean that becomes IP to some extent so it's one of the reasons we decided not to open source everything was really to keep the IP a little bit closer to us and not opening it but then there are downsides like there is less less an opportunity for the community to contribute it's harder to adopt it's hard to do on-prem kind of like deployments and so on.”

Claim: The speaker says keeping the technology closed reduces community contribution opportunities and makes adoption and on-premises deployment more difficult.

“there are pros and cons with the two choices yeah”

Claim: The speaker says both open-sourcing and keeping the technology closed have advantages and disadvantages.

“so we're not able to share much about the the training the size of the models or the flop flops or all of that it's kind of like a trade secret but we are continuing to push the frontier and inception is a great place to be if you want to be have an opportunity to shape the direction of the field like it's still a relatively small field there is a lot to be invented and so a lot of the people that decided to come to inception instead of joining one of the other labs is really that they want to have ownership”

Claim: Inception treats details about its training, model sizes, and FLOPs as trade secrets and cannot share much about them.

“flops or all of that it's kind of like a trade secret but we are continuing to push the frontier and inception is a great place to be if you want to be have an opportunity to shape the direction of the field like it's still a relatively small field there is a lot to be invented and so a lot of the people that decided to come to inception instead of joining one of the other labs is really that they want to have ownership”

Claim: Inception is continuing to push the frontier of AI research and development.

“flops or all of that it's kind of like a trade secret but we are continuing to push the frontier and inception is a great place to be if you want to be have an opportunity to shape the direction of the field like it's still a relatively small field there is a lot to be invented and so a lot of the people that decided to come to inception instead of joining one of the other labs is really that they want to have ownership and they like to invent new things,”

Claim: Many people who joined Inception instead of another lab were motivated by wanting ownership and opportunities to invent new things.

“flops or all of that it's kind of like a trade secret but we are continuing to push the frontier and inception is a great place to be if you want to be have an opportunity to shape the direction of the field like it's still a relatively small field there is a lot to be invented and so a lot of the people that decided to come to inception instead of joining one of the other labs is really that they want to have ownership”

Claim: The field Inception works in is still relatively small, with many things left to be invented.

“they like to be in a space where there's more of a green field and more opportunities to try things that is less that it's known or available out there, it's a little bit more open-ended and so we tend to attract those kind of people.”

Claim: Inception tends to attract people seeking an open-ended, greenfield environment with opportunities to try less-established ideas.

“oh yeah, yeah, I think we're not there yet.”

Claim: The speaker believes his team is not yet at a point where models can recursively improve research enough to make its work unnecessary.

“Maybe we don't have access to the models that other folks have,”

Claim: The speaker may lack access to models available to other people.

“I mean, we use models a lot, of course, and it has accelerated. accelerated the speed at which we can iterate, try ideas, and we use models from Frontier Labs,”

Claim: Using models extensively has accelerated the speaker's team's iteration and experimentation.

“try ideas, and we use models from Frontier Labs,”

Claim: The speaker's team uses models from frontier AI labs.

“It has accelerated our development process a lot.”

Claim: Using models has substantially accelerated the speaker's team's development process.

“at least that right now i don't know what it's going to be in six months or a year but right now the the human ingenuity is still like super important and the ability to come up with the the right ideas and kind of like prune the space and kind of like identify directions that are more promising has been really important to us”

Claim: The speaker is uncertain how important human ingenuity will be to the team's work six months to a year later.

“at least that right now i don't know what it's going to be in six months or a year but right now the the human ingenuity is still like super important and the ability to come up with the the right ideas and kind of like prune the space and kind of like identify directions that are more promising has been really important to us”

Claim: At the time of the conversation, human ingenuity remains highly important to the speaker's team's work.

“at least that right now i don't know what it's going to be in six months or a year but right now the the human ingenuity is still like super important and the ability to come up with the the right ideas and kind of like prune the space and kind of like identify directions that are more promising has been really important to us”

Claim: The team's ability to generate good ideas, prune the search space, and identify promising research directions has been important to its progress.

“It's a small team, but everyone is very talented and they work very hard and we have access to agents that are making us a lot more productive. And so I think the numbers are,”

Claim: The speaker describes the team as small, talented, and hardworking, and says agents make it substantially more productive.

“but everyone is very talented and they work very hard and we have access to agents that are making us a lot more productive. And so I think the numbers are, are sufficient to do to do a lot and in fact often the i feel like the the botanic is more compute than than than people uh but yeah the team is organized like there is a there is a product team effectively that is handling the platform and working with customers to to make them successful with our models and then so there is basically a team that it's serving the current”

Claim: The speaker believes the team's staffing is sufficient for substantial work and that compute, rather than headcount, is often the bottleneck.

“are sufficient to do to do a lot and in fact often the i feel like the the botanic is more compute than than than people uh but yeah the team is organized like there is a there is a product team effectively that is handling the platform and working with customers to to make them successful with our models and then so there is basically a team that it's serving the current current best version of the model and then there is a team that it's building the next version of the model and that includes training”

Claim: Inception separates work between a team serving the current best model and a team developing the next model version.

“are sufficient to do to do a lot and in fact often the i feel like the the botanic is more compute than than than people uh but yeah the team is organized like there is a there is a product team effectively that is handling the platform and working with customers to to make them successful with our models and then so there is basically a team that it's serving the current”

Claim: Inception's product team operates the platform and supports customers in using its models successfully.

“current best version of the model and then there is a team that it's building the next version of the model and that includes training RL inference and so that's more research.”

Claim: Inception's next-model team handles training, reinforcement learning, and inference and is more research-oriented.

“Like I was treated as co-advisor, so I worked on flash attention, for example. That's another thing that.”

Claim: FlashAttention originated in academia and later had a major impact in industry.

“that came out from academia that then eventually had a huge impact in industry right or another example is dpo that was another project started out as a rotation a project in my group uh it's an algorithm that is used to align you know llms and diffusion models and everywhere right and that's again something that that was developed entirely in academia”

Claim: DPO is used as an alignment algorithm for large language models and diffusion models.

“that came out from academia that then eventually had a huge impact in industry right or another example is dpo that was another project started out as a rotation a project in my group uh it's an algorithm that is used to align you know llms and diffusion models and everywhere right and that's again something that that was developed entirely in academia”

Claim: DPO originated as a rotation project in the speaker's academic group and was developed entirely in academia.

“And it was like based on a clever insight, like some interesting mathematical structure that you have in that problem that allows you to come up with a very different and more efficient way of post-training and aligning these models,”

Claim: DPO exploits mathematical structure in its problem to provide a different, more efficient method for post-training and aligning models.

“Lots of opportunities for finding new and better ways of solving important problems.”

Claim: The speaker believes there are many opportunities to discover improved solutions to important problems.

“One of the nice things about academia is that it allows you to take these contrarian bets.”

Claim: Academia allows researchers to pursue contrarian research bets.

“I mean, there is the challenge that maybe we don't have enough resources and there is never enough resources and if we had more compute,”

Claim: The speaker says academia may lack sufficient resources and that resources are never enough.

“I mean, there is the challenge that maybe we don't have enough resources and there is never enough resources and if we had more compute, we could be more efficient.”

Claim: The speaker believes academic researchers could work more efficiently if they had more compute.

“you know, you have access to amazing students and everyone is kind of like trying to develop a new thing.”

Claim: The speaker says academia provides access to excellent students who are working to develop new ideas.

“People are not scared about taking bets.”

Claim: The speaker says academic researchers are not afraid to pursue risky research bets.

“People are not scared about taking bets. And that's why academia has been so impactful, I think, even in the AI space.”

Claim: The speaker attributes academia's substantial impact, including in AI, in part to researchers' willingness to take bets.

“A lot of the important ideas have roots or even were created in academia.”

Claim: The speaker says many important AI ideas originated in or have roots in academia.