Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics

“so last month we released our world action model. model called Dyna2.”

Claim: Dyna Robotics released its world action model Dyna2 the month before the conversation.

“So it's the first word action model trained on more than 1 million hours of data.”

Claim: Dyna2 was the first world action model trained on more than one million hours of data.

“word action model is a model that's able to predict the future video, the future state of the world,”

Claim: A world action model predicts future video or the future state of the world.

“you could also use that future generation to also produce actions robots can execute in the physical world to control the robot.”

Claim: A world action model can generate actions for robots to execute in the physical world, using its prediction of the future.

“It's a very powerful architecture.”

Claim: Jason Ma considers the world action model architecture very powerful.

“Because if you think about how humans act in the physical world, in our brain, we're implicitly thinking about how the world will evolve based on our actions.”

Claim: Humans implicitly anticipate how the physical world will evolve in response to their actions.

“So now we have a model architecture that's able to do that to guide robot actions.”

Claim: The world action model architecture models how the world evolves to guide robot actions.

“So what we find is that by doing so, the model is much more dexterous, much more robust,”

Claim: According to Jason Ma, using a world action model makes the robot model more dexterous and robust.

“and much more capable of generalizing to environments to scenarios it's never seen before. Yeah.”

Claim: According to Jason Ma, the world action model approach improves generalization to previously unseen environments and scenarios.

“the most important input is video, You know, you can think about videos that the robot is capturing with its cameras, So video or images of the, you know, past many seconds, that's the input.”

Claim: The most important input to the robotics model is video or images captured by the robot's cameras over the preceding several seconds.

“But for robots, what's really interesting is that robot also has a lot of actuator, a lot of joint encoders on itself. So robot also knows, you know, the angles of its joints. So those can also be inputs.”

Claim: Robots can use actuator and joint-encoder information, including their joint angles, as model inputs.

“to this world model that's something that's very different from just a pure video model that's only looking at videos right so that's the input and additionally for robots you want to be able to talk to it to instruct it to do different tasks right so you can also feed it like language instruction so those are what i consider the basic set of inputs that go into any robotics models including world action models and once we have that the output”

Claim: Robotics models, including world action models, can take language instructions as inputs.

“At the basic level, it needs to output actions the robots can execute on the hardware.”

Claim: A robotics model must output actions executable by the robot's hardware.

“But in the case of word action model, what's really interesting is that in addition to the actions, it can also output future video,”

Claim: A world action model can output future video as well as robot actions.

“it can also output future video, So basically you can almost output the future video or some representation of it first and then use that to guide robot actions. So that's the key difference between world action models outputs versus other kind of models, robot models outputs.”

Claim: World action models differ from other robotics models by producing future-video predictions or representations in addition to actions.

“So basically you can almost output the future video or some representation of it first and then use that to guide robot actions.”

Claim: A world action model can generate future video or a representation of it before using that prediction to guide robot actions.

“So the way I think about it is like this is kind of analogous to you know, chain of thought, you know, basically the robots, the models reasoning in the visual, the physics space and then outputting actions.”

Claim: Jason Ma describes a world action model as analogous to chain-of-thought reasoning in visual and physics space before producing actions.

“So the mental framework to think about how do we build models for robotics is that there are essentially a couple of things to look at. One is data. And the second is like, how do we make a model learn as much as possible from that data?”

Claim: Jason Ma's framework for building robotics models centers on the data and maximizing what the model learns from it.

“right? So the blog post we released is our Dynatube word action model. model so under the hood it's a diffusion transformer so transformer is essentially like the language model you know uh you can sequence a sequence math instance coder”

Claim: Dyna Robotics' world action model uses a diffusion transformer architecture.

“but what's really interesting is that in visual domains right if you think about generating images generating videos and in this case robots looking at images to generate actions people have found the diffusion kind of architecture to work really well”

Claim: Diffusion architectures have worked well for visual generation and for generating robot actions from images, according to Jason Ma.

“A diffusion model is a model that takes some random noise and through iterative computation can go from noise to something that's really high dimension or that's something really highly like defined and have very fine grained feature.”

Claim: A diffusion model iteratively transforms random noise into a high-dimensional output with detailed, fine-grained features.

“and for robotics, even though you're not generating image, you're generating actions that needs to be very precise. very like smooth right so it has kind of the same kind of”

Claim: Robot actions generated by robotics models need to be precise and smooth.

“our consideration of the sitarata is generating image so that's why from first principle you know the whole field not just us have been using this kind of diffusion transformer architecture to train robotics foundation models you need the transformer aspect because transformer is the thing that can just ingest the most amount of data to learn then diffusion helps you to refine the outcome to get to the smooth precise fine-grained motion you need”

Claim: In robotics foundation models, transformers ingest large amounts of data, while diffusion refines outputs into smooth, precise, fine-grained motions.

“It is mostly an efficiency speed up. Like let's say normally you would take 40 to 100 diffusion steps. You can reduce it to 4.”

Claim: Flow matching can reduce diffusion inference from a typical 40–100 steps to four steps, primarily improving efficiency and speed.

“tuning or converting the initial noise input into the final output. So flow matching is kind of a special case of diffusion that makes the transformation much simpler.”

Claim: Flow matching is a special case of diffusion that simplifies the transformation from initial noise to final output.

“In practice, you don't have to tune as much and it works just as well.”

Claim: In practice, flow matching requires less tuning than diffusion and works just as well.

“That is why I think about two to three years ago, most generative modeling for image and video has shifted from a diffusion model to a full matching model but the concept is the same and full matching is just a special simpler to tune a variant of diffusion yeah yeah”

Claim: The speaker says that about two to three years before the conversation, most generative modeling for images and video shifted from diffusion models to flow-matching models.

“Yeah, so the way we think about it is that for robotics, we need to build a very general strong base model,”

Claim: The speaker believes robotics requires a very general, strong base model.

“how a human interacts and behaves in the physical world, We have very strong base knowledge. We can just readily do a lot of tasks.”

Claim: Humans have strong foundational knowledge that enables them to readily perform many physical-world tasks.

“But if you want to become an expert in any given task, there's some amount of training that needs to happen.”

Claim: A system needs some task-specific training to become an expert at a particular task.

“So our approach is to first train a general model that's training on as much data as we can find.”

Claim: The speaker's team first trains a general model on as much available data as it can find.

“So that is when our data to report our model is trained over a million hours of human data as a starting point.”

Claim: The speaker's model is trained on more than one million hours of human data as its starting point.

“And then given that strong foundation, what we found is we can quickly make the robot or make the model do many, many different tasks with a very minimal amount of fine tuning to just tell the model,”

Claim: The speaker's team can quickly adapt its foundation model to many tasks with very little fine-tuning.

“here's the task I wanted to do. And here's very few amount of demonstrations and the model can readily do a bunch of the tasks.”

Claim: The speaker's model can perform many tasks after receiving a task specification and a small number of demonstrations.

“So I think data is the most important thing in robotics today.”

Claim: The speaker considers data the most important factor in robotics today.

“Because unlike language model, there's no internet,”

Claim: Robotics lacks an internet-scale data source comparable to the one available for language models.

“right? So every company that's building model building hardware has to think about data so there's roughly couple of different ways of operating here one is that you can collect a lot of data yourself which is what we are doing as”

Claim: Every company building robotics models or hardware must address the question of data.

“company that's building model building hardware has to think about data so there's roughly couple of different ways of operating here one is that you can collect a lot of data yourself which is what we are doing as”

Claim: The speaker's company collects a large amount of robotics data in-house.

“For robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient,”

Claim: No individual company can collect enough data on its own to train a robotics foundation model.

“It's kind of like, you know, you're trying to build an internet by just having Reddit or just having Facebook. They're a great website, a lot of users, but it's not enough,”

Claim: The speaker compares relying on one company's data to trying to build an internet using only Reddit or Facebook, which he says would be insufficient despite their large user bases.

“right? So when we think about... Data, there has to be some sort of data acquisition happening where you buy data, you look at data that's already out there.”

Claim: Robotics data acquisition should include purchasing data and evaluating data that already exists.

“You mentioned data quality. I think data quality in robotics is a non-stationary problem in the sense that what is high quality data for your model changes over time. And when your model...”

Claim: In robotics, the data quality a model needs changes over time.

“doesn't know anything like when it's zero any data is useful right you can use a lot of video data uh you know in the case of egocentric data i think it's actually a very good way of pre-training robots because human data is so diverse like it covers every single physical activity you can possibly imagine and that gives robot the model such a strong prior on how things how the world can evolve around”

Claim: At the earliest stage, when a model has no knowledge, any data can be useful.

“doesn't know anything like when it's zero any data is useful right you can use a lot of video data uh you know in the case of egocentric data i think it's actually a very good way of pre-training robots because human data is so diverse like it covers every single physical activity you can possibly imagine and that gives robot the model such a strong prior on how things how the world can evolve around”

Claim: Human data gives a robot model a strong prior about how the physical world changes around a human actor.

“doesn't know anything like when it's zero any data is useful right you can use a lot of video data uh you know in the case of egocentric data i think it's actually a very good way of pre-training robots because human data is so diverse like it covers every single physical activity you can possibly imagine and that gives robot the model such a strong prior on how things how the world can evolve around”

Claim: Egocentric human data is an effective source for robot pretraining because its diversity spans a very broad range of physical activities.

“embodied actor human but the issue with human data is that before your robot looks exactly like a human there's going to be some gap between human actions and the robot actions right so just by training a human data today it's not very likely you can just execute robot actions directly so you still need some amount of adaptation using robots own data right but the robots own data”

Claim: A gap remains between human actions and robot actions until the robot's form matches a human's exactly.

“embodied actor human but the issue with human data is that before your robot looks exactly like a human there's going to be some gap between human actions and the robot actions right so just by training a human data today it's not very likely you can just execute robot actions directly so you still need some amount of adaptation using robots own data right but the robots own data”

Claim: Training a robot only on human data is unlikely to enable direct execution of robot actions.

“embodied actor human but the issue with human data is that before your robot looks exactly like a human there's going to be some gap between human actions and the robot actions right so just by training a human data today it's not very likely you can just execute robot actions directly so you still need some amount of adaptation using robots own data right but the robots own data”

Claim: Robots require some adaptation using robot-generated data to bridge the gap from human data.

“Collecting our robot is going to be much more expensive and much harder to collect.”

Claim: Robot-generated data is much more expensive and difficult to collect than human data.

“So the kind of recipe we're thinking about is like, how do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to make the robot master whatever task we're interested in. So data quality...”

Claim: The speaker's proposed training recipe combines a large source of human egocentric data, which provides broad physical-manipulation priors, with a small amount of robot data for task mastery.

“And getting data in robotics is really, I think it's a fascinating multifaceted problem. But the reality is that there's not a single winning strategy like, oh, here's one kind of data. I'm only going to collect on that.”

Claim: The speaker believes no single data-collection strategy is sufficient to win in robotics.

“which will give you a strong foundation model you really need to think about this human data covers this part of robot cognition I need robot data covers low-level reflex dexterity the robot also needs and put them into one model and I think that's what gives a really adaptable really strong model at the end yeah yeah”

Claim: The speaker believes human data supplies some aspects of robot cognition, robot data supplies low-level reflexes and dexterity, and combining both in one model yields a strong, adaptable model.

“so in our Dynatube blog at the end, we actually have an email address called datapartnerships at Dynat.co.”

Claim: Dyna Robotics lists datapartnerships@dynat.co at the end of its Dynatube blog as a contact for data partnerships.

“I mean, if you really want to nail a task, having data from that task will certainly help you a lot, right?”

Claim: Having data from a specific task will certainly help substantially when trying to perform that task well.

“So for us, it's like, we still think, you know, just a million hours of data is not enough. It's just the beginning.”

Claim: Dyna Robotics considers one million hours of data insufficient and only a starting point.

“We show the first model on a million hours scale, but we need more than that, right?”

Claim: Dyna Robotics has shown its first model using data at the one-million-hour scale and says it needs more than that.

“So we're still looking at a lot of just raw diverse data.”

Claim: Dyna Robotics continues to seek large quantities of raw, diverse data for its robotics work.

“that we can get but we're also thinking about how do we get more custom data either collected in-house or by our partner for like really you know specific niche tasks because i think you still need that to get to you know excellence that you know any model needs to actually be useful in the physical world right it's kind of like humans you know you uh you know if you know sean or me you know we go work at rosewood hotel tomorrow we're not going to be able”

Claim: Jason Ma believes models need custom, task-specific data to reach the performance required for usefulness in the physical world.

“that we can get but we're also thinking about how do we get more custom data either collected in-house or by our partner for like really you know specific niche tasks because i think you still need that to get to you know excellence that you know any model needs to actually be useful in the physical world right it's kind of like humans you know you uh you know if you know sean or me you know we go work at rosewood hotel tomorrow we're not going to be able”

Claim: Dyna Robotics is considering collecting custom data internally or through partners for specific niche robotics tasks.

“that we can get but we're also thinking about how do we get more custom data either collected in-house or by our partner for like really you know specific niche tasks because i think you still need that to get to you know excellence that you know any model needs to actually be useful in the physical world right it's kind of like humans you know you uh you know if you know sean or me you know we go work at rosewood hotel tomorrow we're not going to be able and be able to make the bed the way they want right we need a lot of training so that kind of data like special data it's kind of like what we're close doing today for language model i think will be very very valuable for robotics as well”

Claim: Jason Ma argues by analogy that people need substantial training to make beds to a particular hotel’s standards, just as robots need task-specific training.

“and be able to make the bed the way they want right we need a lot of training so that kind of data like special data it's kind of like what we're close doing today for language model i think will be very very valuable for robotics as well”

Claim: Jason Ma believes specialized training data similar to data used in language-model development will be highly valuable for robotics.

“Yeah. So I think generally in synthetic environments, synthetic data, simulation data for robotics, it is going to be one of the next frontiers.”

Claim: Jason Ma expects synthetic environments, synthetic data, and simulation data to become a major next frontier in robotics.

“right because today people have looked at egocentric data robot data you know those are scaling but i think since that static data will be a new dimension of scaling so going back to your question like yeah we have been exploring like using frontier models to generate synthetic environments for us on our data on our environments that we care about and surprisingly i think now is at a point where this technology is actually”

Claim: Jason Ma says egocentric and robot data are scaling and expects static data to add a new dimension to robotics scaling.

“right because today people have looked at egocentric data robot data you know those are scaling but i think since that static data will be a new dimension of scaling so going back to your question like yeah we have been exploring like using frontier models to generate synthetic environments for us on our data on our environments that we care about and surprisingly i think now is at a point where this technology is actually”

Claim: Dyna Robotics has explored using frontier models to generate synthetic environments tailored to its data and target environments.

“very much worth like going really really deep into so for some context so in my phd i primarily focused on simulation-based training for robotics back then simulation wasn't ready but we had this paper called eureka which was the first paper taking like a large language model at the time is gpt4 we just had it like self generate environment generate reward functions or reinforcement learning just like”

Claim: Jason Ma says the Eureka paper used GPT-4 to generate simulation environments and reinforcement-learning reward functions.

“very much worth like going really really deep into so for some context so in my phd i primarily focused on simulation-based training for robotics back then simulation wasn't ready but we had this paper called eureka which was the first paper taking like a large language model at the time is gpt4 we just had it like self generate environment generate reward functions or reinforcement learning just like”

Claim: Jason Ma’s PhD research primarily focused on simulation-based training for robotics.

“just like do the things in simulation and at the end i think we had this really nice demo of the robot spinning pencils right that was really cool at the time but”

Claim: Jason Ma says the Eureka work included a demonstration of a robot spinning pencils.

“that kind of thing was still very limited at the time because back then the model didn't really have good vision capability didn't have an innate 3d understanding right so it was basically just generating code and like using very limited feedback from text to try to come up with new simulation environments but today like the vision capability of these frontier models are crazy they can do like real to sim they're catting there's 3d reconstruction all of that”

Claim: Jason Ma says simulation-based robotics training was limited at the time because models had weak vision capabilities and lacked innate 3D understanding.

“that kind of thing was still very limited at the time because back then the model didn't really have good vision capability didn't have an innate 3d understanding right so it was basically just generating code and like using very limited feedback from text to try to come up with new simulation environments but today like the vision capability of these frontier models are crazy they can do like real to sim they're catting there's 3d reconstruction all of that”

Claim: Jason Ma says current frontier models have strong vision capabilities that support real-to-simulation workflows and 3D reconstruction.

“now it's at a tipping point where. this technology could really accelerate robotics and for us right we think about that as like a way to accelerate our internal progress right because we're building models we're building hardware so if a frontier model can give us more data that way like generate really tailored data for ourselves that's extremely useful for us and another thing we've been thinking about is instead of using these models to generate like”

Claim: Jason Ma believes frontier-model technology may now be at a tipping point and could substantially accelerate robotics.

“this technology could really accelerate robotics and for us right we think about that as like a way to accelerate our internal progress right because we're building models we're building hardware so if a frontier model can give us more data that way like generate really tailored data for ourselves that's extremely useful for us and another thing we've been thinking about is instead of using these models to generate like”

Claim: Dyna Robotics believes frontier models can accelerate its internal model and hardware development by generating tailored data.

“3D environments, we have also used our own world model, right, the world action model I was telling you about because the model not only generates action, can also generate videos the model can also generate its own training data so that becomes a, you know, self-fulfilling like flywheel that I think will also be powerful”

Claim: Dyna Robotics says its world action model can generate both actions and videos, enabling it to generate training data for itself.

“the world action model I was telling you about because the model not only generates action, can also generate videos the model can also generate its own training data so that becomes a, you know, self-fulfilling like flywheel that I think will also be powerful”

Claim: Jason Ma believes a model generating its own training data could create a powerful self-reinforcing flywheel.

“I think even though we have scale to a million hours of data, we are deploying a bunch of robots, the robots are actually, you know, already starting to do very cool things. We're just at the...”

Claim: Dyna Robotics says it has scaled to one million hours of data and is deploying robots that are already beginning to perform notable tasks.

“beginning just the tip of iceberg for like what is there to come yeah yeah”

Claim: Jason Ma believes current robotics progress is only the beginning of what is yet to come.

“yeah no that's a great question so i think what's really different about robots versus language model and other applications is that for coding agents language model it's okay if it's it has latency right like you know today whenever i ask chat gpt it takes like a couple seconds before it spits out anything but for robots like the inference latency has to be like very short like the robot needs to act like you can't just hold a knife and not and not do anything for three seconds that's going to freak people out right”

Claim: Jason Ma says robots require very low inference latency because a robot holding a knife cannot remain inactive for three seconds without alarming people.

“yeah no that's a great question so i think what's really different about robots versus language model and other applications is that for coding agents language model it's okay if it's it has latency right like you know today whenever i ask chat gpt it takes like a couple seconds before it spits out anything but for robots like the inference latency has to be like very short like the robot needs to act like you can't just hold a knife and not and not do anything for three seconds that's going to freak people out right”

Claim: Robots require very short inference latency; a robot pausing for several seconds while holding a knife would alarm people.

“our model is pretty smart so that's okay but uh in general like if you think about very dynamic tasks like for example you want your robot to like you know hold a i don't know a tennis racket and you know juggle the ball right it's very dynamic very reactive you can't just like the physics gravity doesn't wait for you right so for that reason robotics model you though it needs to be able to inference very fast and the fastest way to do that is by serving on-prem like on device right because if you call a model by”

Claim: Robotics models need very fast inference, and on-premises or on-device serving is the fastest way to achieve it.

“our model is pretty smart so that's okay but uh in general like if you think about very dynamic tasks like for example you want your robot to like you know hold a i don't know a tennis racket and you know juggle the ball right it's very dynamic very reactive you can't just like the physics gravity doesn't wait for you right so for that reason robotics model you”

Claim: Dynamic, reactive robotic tasks require fast responses because physical processes such as gravity continue while the robot is acting.

“no guarantee if you drive a car there's no guarantee right like so for that reason the model that gets deployed eventually has to be of reasonable size that can be served locally but that doesn't mean that the initial model you train also has to be small like in language model you”

Claim: A robotics model deployed in the field must be small enough for local serving, but the initial training model need not be small.

“have a large people do distillation right like the best way to make a small model very smart is by doing distillation right you train the biggest model then you distill right so i think in robotics we're getting there but i think that's what people are going to cover shot yeah”

Claim: Robotics is beginning to adopt training large models and distilling them into smaller models, which the speaker expects people to do.

“have a large people do distillation right like the best way to make a small model very smart is by doing distillation right you train the biggest model then you distill right so i think in robotics we're getting there but i think that's what people are going to cover shot yeah”

Claim: Training a large model and distilling it is, in the speaker's view, the best way to make a small model highly capable.

“Yeah, so we build our own inference stack for whatever model we deploy because I think like I was saying like latency is so important so only the whole stack is extremely important for robotics, right? You want to optimize like the inference speed for the kind of model you're running and there's a lot of work in there. So that's what we do.”

Claim: Dyna Robotics builds its own inference stack for each deployed model because low latency is crucial in robotics.

“So the way we think about what we need to own versus what we can buy is like if the input output and off a component is very very clear and like there's already extremely good mature solution then there's no point to build in-house for example on this robot you see the lidars right so those black circle those are lidars to help the robot detect obstacle navigate surroundings so that's a very mature technology that people have been using for now decades it's on self-driving car”

Claim: Dyna Robotics prefers buying components when their input-output function is clear and mature, high-quality solutions already exist, rather than building them in-house.

“input output and off a component is very very clear and like there's already extremely good mature solution then there's no point to build in-house for example on this robot you see the lidars right so those black circle those are lidars to help the robot detect obstacle navigate surroundings so that's a very mature technology that people have been using for now decades it's on self-driving car”

Claim: LiDAR is a mature technology used for decades, including in self-driving cars, and can help robots detect obstacles and navigate.

“So we don't need to build LiDAR from scratch,”

Claim: Dyna Robotics does not need to develop LiDAR from scratch.

“right? Same goes with some cameras, but for anything that we think is extremely crucial for the whole system's performance at the end, our vision is to be able to own that and build that ourselves.”

Claim: Dyna Robotics aims to develop in-house the components it considers critical to overall system performance.

“right? Same goes with some cameras, but for anything that we think is extremely crucial for the whole system's performance at the end, our vision is to be able to own that and build that ourselves.”

Claim: Dyna Robotics buys or uses some existing cameras instead of building all cameras in-house.

“So one thing is you mentioned actuator. So that's something we're working on and we're going to own ourselves.”

Claim: Dyna Robotics is developing actuators in-house and intends to own them.

“like the quality and how you design actually really matters for how well you can model the robot right actually there's what drives the robot to build a model your robot very well in simulation or to do any kind of modeling you need to select actually it's very good but you know can't actually be easily modelable so that's what we care about eventually so we're going to own that same thing with you know the robot's proportion you know this robot's proportion is like”

Claim: Actuator quality and design influence how accurately a robot can be modeled, including in simulation.

“we measure against real people why do we do that because we're pre-training on human data that's transfer right so there's a lot of these kind of decision that goes into hardware that's very much tied to like because eventually you know it's hardware that carries intelligence intelligence is the extremely important thing we want to make sure the hardware serves”

Claim: Dyna Robotics matches robot proportions to real people because its pretraining uses human data to support transfer.

“transfer right so there's a lot of these kind of decision that goes into hardware that's very much tied to like because eventually you know it's hardware that carries intelligence intelligence is the extremely important thing we want to make sure the hardware serves”

Claim: Dyna Robotics designs its hardware to support the robot's intelligence.

“i mean i mean it's a good question right i think there's a lot of exciting discussions on burning manufacturing back to the u.s but you know for robots especially that's actually much harder than”

Claim: Rebuilding robotics manufacturing in the United States is more difficult than it may appear.

“uh than what it seems and i would say even now there isn't a company in the u.s that's truly building everything from scratch in the u.s right like even when they say they assemble robot they build whatever actually doing in the u.s they're still sourcing the raw materials from other places in the world and i think that's something to really be aware of as a robot builder right like i think uh”

Claim: As of the conversation, no U.S. company builds a robot entirely from scratch using only domestic inputs; U.S. robot makers still source raw materials internationally.

“Like as a full stack robotics company, we realize just how long the supply chain is and how long the whole problem is is actually extremely hard.”

Claim: Building robotics as a full-stack company reveals that the supply chain is extensive and the overall challenge is extremely difficult.

“And I think these days, a lot of discussion centers around a robot foundation model, model, data, but I think hardware and just how hard it is to build good hardware is very under emphasized. But I think”

Claim: Current discussion of robot foundation models underemphasizes hardware and the difficulty of building high-quality hardware.

“That's going to be extremely important.”

Claim: Hardware will be extremely important to robotics.

“yeah so we are currently exploring many different options right because with global supply chain then there is some you know geopolitical things right shipping also shipping delays and all that stuff”

Claim: Dyna Robotics is exploring multiple supply-chain options, citing geopolitical concerns and shipping delays associated with global supply chains.

“yeah and uh in the u.s you know it's just that going from zero to one will take some time right so we're trying to think about a solution that can both allow us to build more in the US while leveraging global supply chain right but I do think going back to your previous question on like some companies choose to just build model we build everything I think I think it's very important to get this right I feel that for robotics the hard thing is”

Claim: Dyna Robotics expects establishing U.S. manufacturing from zero to take time and is considering a strategy that combines more U.S. production with use of the global supply chain.

“solution that can both allow us to build more in the US while leveraging global supply chain right but I do think going back to your previous question on like some companies choose to just build model we build everything I think I think it's very important to get this right I feel that for robotics the hard thing is”

Claim: Dyna Robotics builds the complete robotics system rather than only the model, and the speaker considers this approach very important to get right.

“uh like building the whole system or the necessary thing is to build a whole system like previously i was i did mine i was at nvidia primarily just building models and i saw the upper bound of the system when you don't control the hardware, when you don't get to optimize that,”

Claim: The speaker says that while primarily building models at NVIDIA, he observed the limitations of a system whose hardware he could not control or optimize.

“Because if you think about the value creation for robots, eventually robots are here to deliver service to solve some physical tasks.”

Claim: The speaker describes robots' purpose as delivering services by solving physical tasks.

“So if you want to solve physical tasks, you want to be able to optimize everything. And that's important.”

Claim: The speaker argues that solving physical tasks requires being able to optimize the robot's entire system.

“so we think of this as generally like a semi humanoid so basically the upper body is you know like a humanoid it's got you know of course shoulder got arms it also got like four degrees of freedom on the waist so you can basically do all the upper body motion that a human can do but it's a semi humanoid because the lower body is wheels wheels.”

Claim: Dyna Robotics' robot is semi-humanoid, with a humanoid upper body, shoulders and arms, four degrees of freedom at the waist, and a wheeled lower body.

“Because I think there is a difference. There's optimal like practicality versus performance trade-off. I think with legs, the energy consumption will be a lot higher.”

Claim: The speaker sees a practicality-performance trade-off in robot locomotion and expects legged robots to consume substantially more energy.

“It would be a lot more unstable. You know, safety will be extremely, extremely hard to guarantee.”

Claim: The speaker expects legged robots to be substantially less stable and says guaranteeing their safety would be extremely difficult.

“So right now for legged robots, there's actually no guarantee that the robot will stay upright.”

Claim: The speaker says there is currently no guarantee that a legged robot will stay upright.

“i'm sure you guys have that experience too right then the next few days like you just couldn't move or like you pull it or you slept in the wrong position so your neck is very stiff we have all experienced that and then whenever you experience that you kind of realize wow so many muscles so many degrees of freedom that we use in our daily life that we take for granted but when it's disabled like it's very awkward for you to do some tasks that you feel are very natural right”

Claim: People often take the many muscles and degrees of freedom used in everyday movement for granted; when those abilities are impaired, ordinary tasks can become awkward.

“right so that's also how we think about like when we design robots when you have like a this kind of design which is this purely elevator could only go up and down we realize a lot of the like human torso movement like we use our like you know waist back a lot for like day-to-day physical activity right even like if i want to get this bottle like i'm not just moving my arms it's actually very strange for me if i”

Claim: Human waist and back movement contributes substantially to everyday physical activity.

“way that humans move their body that I think it would be equally useful if robot has a degree of freedom and if you further think about okay we want to learn from human data like humans are moving their torso all the time you want to be able to like ingest that into the model to control the robot but if you have a form factor that doesn't allow for that then the transfer becomes less and The robot becomes less dexterous and it's just more awkward.”

Claim: When a robot's form factor cannot accommodate human torso movements, transferring human movement data to its control is less effective, making the robot less dexterous and more awkward.

“way that humans move their body that I think it would be equally useful if robot has a degree of freedom and if you further think about okay we want to learn from human data like humans are moving their torso all the time you want to be able to like ingest that into the model to control the robot but if you have a form factor that doesn't allow for that then the transfer becomes less and”

Claim: The speaker believes robots would benefit from degrees of freedom that allow them to reproduce human body movements.

“oh yeah yeah it generates very well so for all the demos that i have shown you on the new robot because the new robot we you know it got ready not so long ago so we quite literally did not have a lot of data compared to the stationary arms we showed you which had much more data right but a new robot”

Claim: Dyna Robotics' new robot had relatively little data because it had only recently become ready, whereas the stationary arms shown had substantially more data.

“the model has not seen this in pre-training in some other pre-trained model it's able to learn the task that you saw like very quickly and i credit a lot of that to like you know like even before the hardware is ready we're already committed to pre-training on humans because we knew at some point our robots will be like human you can get ahead on the training side while hardware is getting ready and i think now we're seeing that like the transfer is actually there and is actually very helpful”

Claim: The speaker says the model learned the demonstrated task quickly even though it had not encountered that task during pre-training.

“the model has not seen this in pre-training in some other pre-trained model it's able to learn the task that you saw like very quickly and i credit a lot of that to like you know like even before the hardware is ready we're already committed to pre-training on humans because we knew at some point our robots will be like human you can get ahead on the training side while hardware is getting ready and i think now we're seeing that like the transfer is actually there and is actually very helpful”

Claim: The speaker attributes some of the model's rapid task learning to pre-training on human data before the robot hardware was ready.

“the model has not seen this in pre-training in some other pre-trained model it's able to learn the task that you saw like very quickly and i credit a lot of that to like you know like even before the hardware is ready we're already committed to pre-training on humans because we knew at some point our robots will be like human you can get ahead on the training side while hardware is getting ready and i think now we're seeing that like the transfer is actually there and is actually very helpful”

Claim: The speaker says transferring learning from human-data pre-training to the robot is working and is very helpful.

“so if we want to learn a new task, you know, we basically do a little bit of robot data collection, a little small amount of data collected train the model then we just deploy the model in the real world and then we look at the performance then yeah”

Claim: Dyna Robotics’ process for learning a new task is to collect a small amount of robot data, train a model, deploy it in the real world, and evaluate its performance.

“yeah, like a couple hours, couple hours, yeah, we're in some of the tasks like just tens of minutes,”

Claim: Robot data collection for a new task can take a couple of hours, though for some tasks it takes only tens of minutes.

“metrics you can look at right just like I've trained your model it converges nicely now after that for robotics today we just run the model in the real world right on the you know same setup that the data is collected to see how the model does sometimes the model just works great out of the box right you don't really need to do much more but when the models you know like maybe like 70% 80% there then use”

Claim: Dyna Robotics evaluates trained robotics models in the real world using the same setup on which the training data was collected.

“metrics you can look at right just like I've trained your model it converges nicely now after that for robotics today we just run the model in the real world right on the you know same setup that the data is collected to see how the model does sometimes the model just works great out of the box right you don't really need to do much more but when the models you know like maybe like 70% 80% there then use”

Claim: Some robotics models perform well immediately after training and need little further iteration.

“metrics you can look at right just like I've trained your model it converges nicely now after that for robotics today we just run the model in the real world right on the you know same setup that the data is collected to see how the model does sometimes the model just works great out of the box right you don't really need to do much more but when the models you know like maybe like 70% 80% there then use see how the model behaves you figure out what it's not good at then you can like collect a bit more data then iterate right i think of this as like the robotics version of like reinforcement learning from human feedback right the robot rolls out you can score it good bad somewhat good and then that data the robot's only experience can be used to train the model and you can also just collect more data yourself to train a model right eventually you”

Claim: When a robotics model is roughly 70–80% successful, Dyna Robotics identifies its weaknesses through real-world behavior, collects additional data, and iterates on the model.

“see how the model behaves you figure out what it's not good at then you can like collect a bit more data then iterate right i think of this as like the robotics version of like reinforcement learning from human feedback right the robot rolls out you can score it good bad somewhat good and then that data the robot's only experience can be used to train the model and you can also just collect more data yourself to train a model right eventually you”

Claim: The speaker characterizes robot rollouts scored by people as good, bad, or somewhat good as a robotics analogue of reinforcement learning from human feedback, with the resulting experience usable to train the model.

“see how the model behaves you figure out what it's not good at then you can like collect a bit more data then iterate right i think of this as like the robotics version of like reinforcement learning from human feedback right the robot rolls out you can score it good bad somewhat good and then that data the robot's only experience can be used to train the model and you can also just collect more data yourself to train a model right eventually you”

Claim: Dyna Robotics can train its models using both robot-generated experience and additional data collected directly by the team.

“You know, our goal is like, you know, this robot is going to be our platform. Anyone should be able to train tasks on it.”

Claim: Dyna Robotics aims to make its robot a platform that anyone can use to train tasks.

“one hour to finish a whole yield price so the longest duration that a robot can do like a full job is already in the hour range”

Claim: At the time of the conversation, Dyna Robotics had demonstrated a robot completing a full job lasting about an hour.

“right but you know to be able to do that for any task you know I think the bottleneck is coming back to data and hardware right so for some of the tasks you know for example setting up the space”

Claim: The speaker identifies data and hardware as bottlenecks to enabling robots to perform any task.

“you know you might need more than like a you know the parallel jaw gripper right you might need something more dexterous like a hand like a human hand right but then if you want to train a model on the hand then your bottleneck by the hardware stability like hand today the best hands dexterous hands are still not durable enough they overheat pretty quickly right so that becomes a bottleneck right like you we you know you quite literally have to put a”

Claim: Training robotics models with dexterous hands is limited by hardware stability because the best available dexterous hands are not durable enough and overheat quickly.

“you know you might need more than like a you know the parallel jaw gripper right you might need something more dexterous like a hand like a human hand right but then if you want to train a model on the hand then your bottleneck by the hardware stability like hand today the best hands dexterous hands are still not durable enough they overheat pretty quickly right so that becomes a bottleneck right like you we you know you quite literally have to put a”

Claim: Some robotic tasks may require a dexterous, human-like hand rather than a parallel-jaw gripper.

“Because you can imagine like today most of the data is like vision based, right? It's like video streams and like, you know, some encoding of your hand motion.”

Claim: The speaker says that current physical-task data is mostly vision-based, consisting of video streams and some encoding of hand motion.

“where the robots are joints but humans have a lot more sensors you know we hear things we feel things smell yeah you smell things right so we subconsciously use all of those to help us be so good at manipulating the physical world but today like if we're not capturing that at all then even when we scale to millions of hours you might just be missing critical dimensions of the tasks so you'll never be able to solve a task right”

Claim: Humans use sensory inputs such as hearing, touch, and smell to help them manipulate the physical world effectively.

“where the robots are joints but humans have a lot more sensors you know we hear things we feel things smell yeah you smell things right so we subconsciously use all of those to help us be so good at manipulating the physical world but today like if we're not capturing that at all then even when we scale to millions of hours you might just be missing critical dimensions of the tasks so you'll never be able to solve a task right”

Claim: The speaker warns that without capturing human sensory dimensions beyond current data, scaling to millions of hours might still omit critical task information and prevent robots from solving some tasks.

“So I think those are like what I consider like, you know, what's dragging on the timeline,”

Claim: The speaker considers hardware limitations and data-collection limitations to be slowing progress on the timeline for physical-agent development.

“but for tasks or for workflows that you can collect data today to high fidelity, then I don't necessarily think there's a fundamental bottleneck, right?”

Claim: The speaker does not consider there to be necessarily a fundamental bottleneck for tasks whose data can already be collected at high fidelity.

“Just like what we are demonstrating to the public this week, you know, we have a physical agent that can do the entire laundry workflow, as a human would,”

Claim: The speaker says the company is publicly demonstrating a physical agent capable of completing an entire laundry workflow in a human-like manner.

“We don't really have like, oh, you actually need more than a parallel jogger, but you actually need some sensing modality that's just not available today, right?”

Claim: The speaker says that completing the laundry task does not require a sensing modality that is unavailable today.

“uh you know we have like a one video for all the customers we're working with it's a really really nice video shows what we're doing and uh yeah i think it's great but yeah we have customers uh you know best western that's in the hotel space we fold a lot of laundry like uh you know towels for them every hotel needs every gym needs every fitness center needs it yeah”

Claim: The speaker says Best Western is a hotel-sector customer for whom the company folds laundry such as towels.

“but uh in dintai phong like if you go to dintai phong or other like you know somewhat upscale restaurant there's napkins on your table right so robots robots today fold thousands of napkins for on many many different locations many different dentaiphone locations in the united states so that's actually our biggest scaled customer right so i want to make a distinction here”

Claim: The speaker says robots fold thousands of napkins at multiple Din Tai Fung locations in the United States and identifies Din Tai Fung as the company's largest scaled customer.

“Today, there's a lot of robotics companies that's talking about deployments, talking about pilots, but what's really hard is not putting your robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites.”

Claim: The speaker argues that the main challenge in robotics deployment is demonstrating enough customer value to secure deployment across many sites, rather than placing a robot at one site or conducting a pilot.

“So that's what's happened for us with DintaiFong, which to the best of my knowledge hasn't really happened for other.”

Claim: The speaker says Din Tai Fung has deployed the company's robots across multiple sites, and he is not aware of similar multi-site deployments by other companies developing general-purpose robots.

“companies doing similar like a general purpose robots so I didn't type on our robots fold thousands of napkins per day it's the quality standard is extremely high you know because they want like okay uh the second forward just seam of the napkin is only like a couple centimeters below the uh i don't even know how you call it the pocket so you can stick the chopsticks in so it's actually very fine grain high precision has i can show you yes”

Claim: The speaker says the company's robots fold thousands of napkins per day for Din Tai Fung and must meet an extremely high, fine-grained precision standard.

“But the hard thing about manipulating deformable object, right? Like napkin, towels, everything is deformable. There's just infinitely many possible configuration. So that's really hard for a specialized robot,”

Claim: The speaker says specialized robots struggle to manipulate deformable objects such as napkins and towels because those objects can take infinitely many configurations.

“And what we find is that by developing a general purpose model and robot and adapting it to these use cases, you can actually get to production quality faster.”

Claim: The speaker says that developing a general-purpose model and robot and adapting them to particular use cases can achieve production quality faster.

“And what's really nice is that now that Gen Typhoon has scaled, has signed a much bigger contract with us,”

Claim: The speaker says that after Din Tai Fung's deployment scaled, it signed a substantially larger contract with the company.

“they not only want our robots to fold napkins, now they want our robots to cut, that's why we have the example to cut food. That's one of the use cases they're thinking.”

Claim: The speaker says Din Tai Fung is considering food cutting as an additional use case for the company's robots, beyond napkin folding.

“One is that robotics is going to scale just like any other domains that machine learning has solved,”

Claim: Robotics will scale in the same way as other domains solved by machine learning.

“the one thing that's a lot harder is like, what is the right data is much harder.”

Claim: Identifying the right data for robotics is much harder than identifying the right data in domains such as language or video games.

“to figure out the other domains right i mean language is text uh in video games just video game data but for robotics like what sensors you choose determine what kind of data you have right but if you have the right set of sensors and just scale that data i think you'll get predictable return model performance so in that way robotics isn't so different so that's the better lesson as the corollary of that is that”

Claim: If robotics uses an appropriate set of sensors and scales the resulting data, model performance will improve predictably.

“to figure out the other domains right i mean language is text uh in video games just video game data but for robotics like what sensors you choose determine what kind of data you have right but if you have the right set of sensors and just scale that data i think you'll get predictable return model performance so in that way robotics isn't so different so that's the better lesson as the corollary of that is that”

Claim: The sensors selected for a robot determine the kind of data available for robotics learning.

“Like, there's a lot of research in robotics that's coming out with very specific priors.”

Claim: A substantial amount of robotics research uses highly specific priors.

“But I think the better lesson here is that just keep the algorithm simpler and scale the right kind of data, then generalization, better behavior will emerge.”

Claim: In robotics, keeping algorithms simple and scaling appropriate data can lead to emergent generalization and better behavior.

“which is like at the end of the day, scaling will work.”

Claim: Scaling will ultimately work in robotics.

“But in robotics to scale and how to scale just much harder because there is no internet.”

Claim: Scaling robotics is much harder than in internet-based domains because robotics lacks an internet source of data.

“You have to come up with your hardware, come up with the sensors, get all of those right and then scale.”

Claim: Scaling robotics requires developing suitable hardware and sensors before scaling the resulting system or data.

“Yeah. So the way I think about it is that, yes, eventually the robot should, if we want to build a general purpose robot, like a machine that can be adapted to as many use cases as possible, I think human form factor will be the best one just because that's what we have been doing. for ages,”

Claim: The speaker thinks a human form factor will be best for a general-purpose robot adaptable to many use cases, because the world has long been shaped around humans.

“But I think the best way to actually build a human robot is to take an iterative approach because robotic is so hard.”

Claim: The speaker believes an iterative development approach is the best way to build a human-form robot because robotics is difficult.

“The whole problem, the whole stack is so long.”

Claim: Building a robot involves a long and complex end-to-end technology stack.

“It's really hard to know a priori what's the right decision to make,”

Claim: Because robotics is complex, it is difficult to determine in advance which design decisions are correct.

“And because hardware decision is much more expensive and much more costly just because hardware iteration is a lot slower, you actually want to take an incremental iterative approach.”

Claim: Robotics development should proceed incrementally and iteratively because hardware decisions are costly and hardware iteration is slow.

“And because hardware decision is much more expensive and much more costly just because hardware iteration is a lot slower,”

Claim: Hardware decisions are costly, and hardware development cycles are much slower than other forms of iteration.

“not going all the way to the final form factor because the chances that the thing you build actually doesn't last it's not good enough so you just end up scratching it anyway you know like it's public news like you know optimists you know already gone through many many iterations and they had to scratch their designs right and i think by iterating with a semi-human form factor and actually putting them out to real customers or and just getting models redeployed”

Claim: Using a semi-human robot form factor with real-customer deployment and model redeployment creates a feedback loop for development.

“not going all the way to the final form factor because the chances that the thing you build actually doesn't last it's not good enough so you just end up scratching it anyway you know like it's public news like you know optimists you know already gone through many many iterations and they had to scratch their designs right and i think by iterating with a semi-human form factor and actually putting them out to real customers or and just getting models redeployed”

Claim: Starting with a robot's final form factor risks producing an inadequate or short-lived design that must be discarded.

“I think the iterative approach will actually get us closer to the final robot faster.”

Claim: An iterative robotics-development approach will reach a final robot sooner than a non-iterative approach.

“a cohort yeah yeah yeah and that's actually a very hard problem because if you just think about like the model you know it's outputting action there's the actions you know dimension right the bigger the dimension is the harder to fit the model to the data but here the action space is quite large but uh”

Claim: A larger action-space dimension makes it harder to fit a robotics model to its data; the robot discussed has a large action space.

“uh we did very good job to make sure the model fits well and gives you a smooth”

Claim: The speaker says the model was designed to fit the data well and generate smooth motion despite the robot's large action space.

“yeah so it's in the orders of like uh you know about a hundred action dimensions”

Claim: The robot's action space has approximately 100 dimensions.

“right so the model is just basically predicting at very high frequency you know like tends like at least 50 dimensional vector like 10 20 times a second to get that continuous motion yeah”

Claim: The robotics model predicts a vector of at least 50 dimensions 10–20 times per second to generate continuous motion.

“context is a very important thing in robotics right because otherwise if you're just making decision based on the current observation sometimes it's really hard to tell like what to do next right for example like if you think about opening a microwave like in the middle part”

Claim: Robotics systems need context because a current observation alone may not reveal the appropriate next action.

“yeah yeah so i mean eventually we will have our own sdk but uh the way the way we think about is we want a general like harness layer for robotics foundation model today that is our own model right the way i think about is like for any like”

Claim: Dyna Robotics aims to build a general-purpose harness layer for robotics foundation models and currently uses its own model.

“real task long task in the physical world there's a lot of like a high level decision making you have to do right it's not just like robot you know only reasoning about the physical world in the action continuous trajectories right because that's a very like low level the thing you're actually executing the physical world but that's not the intent that you know like when we are doing this inter making we think about steps of a recipe we think about oh like i finished this i need to do that well”

Claim: Long-horizon physical-world tasks require high-level decision-making in addition to low-level reasoning about physical actions and continuous trajectories.

“while I'm waiting for this part to finish I need to go somewhere else to do something there's a lot of high-level cognition decision making that's happening and I think for robots to handle even you know the laundry tasks we're showing you it has to have that capability because you know it needs to reason okay the washer is running so I should go do something else while the washer is finishing but now the washer is done I should take I should take the towels from the wash and put it in dryer immediately,”

Claim: Robots performing laundry workflows need high-level cognition to schedule other work while a washer runs and transfer laundry to the dryer when the washer finishes.

“Because if not, then that becomes a bottleneck in the whole workflow,”

Claim: In a laundry workflow, delaying the transfer of laundry from the washer to the dryer can create a bottleneck.

“you know try to learn that with a you know low-level model that's outputting actions but it'd be extremely inefficient so that is by having a hierarchical system like putting a harness like a agentic system around our word to action model is what allowed the model to compose different capabilities different skills to solve a much longer horizon task it's very much like in language model right like two years ago three years ago and the model was only good at autocomplete”

Claim: A hierarchical agentic harness around Dyna Robotics’ word-to-action model enables the model to combine capabilities and skills for longer-horizon tasks.

“you know try to learn that with a you know low-level model that's outputting actions but it'd be extremely inefficient so that is by having a hierarchical system like putting a harness like a agentic system around our word to action model is what allowed the model to compose different capabilities different skills to solve a much longer horizon task it's very much like in language model right like two years ago three years ago and the model was only good at autocomplete”

Claim: Learning high-level task reasoning solely with a low-level action-output model would be extremely inefficient.

“complete like humans have to like tap tap tap to get the model to keep spitting out code but today just like an agentic system just carries the end-to-end task and that's and i think that's important because i think today in robotics people are still thinking about autonomy”

Claim: Compared with earlier language models that required repeated human prompting to continue generating code, agentic systems can now carry out end-to-end tasks.

“complete like humans have to like tap tap tap to get the model to keep spitting out code but today just like an agentic system just carries the end-to-end task and that's and i think that's important because i think today in robotics people are still thinking about autonomy At the very task level, like the one minute, 30 second task,”

Claim: The speaker says robotics autonomy is still often considered at the level of short tasks, such as one-minute or 30-second tasks.

“but really the value is about tasks or like workflows that go on like unsupervised for hours and that's what we're building towards and that's why I think thinking about the robotics problem, not just as a modeling problem, but rather as a system problem,”

Claim: Robotics should be approached as a system problem rather than solely as a modeling problem.

“but really the value is about tasks or like workflows that go on like unsupervised for hours and that's what we're building towards and that's why I think thinking about the robotics problem, not just as a modeling problem,”

Claim: The speaker sees the value of robotics as enabling tasks or workflows to run unsupervised for hours.

“you need good agent, good harness engineer, good model, good hardware, everything.”

Claim: A robotics system requires a capable agent, harness engineering, model, and hardware.

“Like, it affects us very deeply in many ways.”

Claim: Advances in language models affect Dyna Robotics in many significant ways.

“like, we're extremely AI-native company. right like coding agents all of that is a very fundamental part of our work and we're very welcoming of that like whenever there's a new model we're the first one to test it right like we being like when gpt6 came out we had like a company-wide hackathon the same week”

Claim: Dyna Robotics describes itself as extremely AI-native, with coding agents forming a fundamental part of its work.

“right like coding agents all of that is a very fundamental part of our work and we're very welcoming of that like whenever there's a new model we're the first one to test it right like we being like when gpt6 came out we had like a company-wide hackathon the same week”

Claim: Dyna Robotics says it tests new models promptly and held a company-wide hackathon in the week GPT-6 was released.

“So we really like using coding agents to accelerate our work.”

Claim: Dyna Robotics uses coding agents to accelerate its work.

“I think it's about the ability for it to like construct virtual environments, right?”

Claim: The speaker found GPT-6’s ability to construct virtual environments notable.

“Yeah, I mean, in the six months to one year frame, our goal is definitely like scaling up our robots and our model and truly getting them to deploy.”

Claim: Dyna Robotics aims, over the next six months to one year, to scale up its robots and model and deploy them.

“right? Because our goal is to build a general model or general platform.”

Claim: Dyna Robotics aims to build a general-purpose model or platform.

“And we think of each different task as like validation of our platform as a way, as a wedge for us to. figure out how to build everything better.”

Claim: Dyna Robotics uses individual tasks to validate its platform and learn how to improve it.

“So today we are doing a lot of that in laundry, in restaurants.”

Claim: Dyna Robotics currently focuses substantial work on laundry and restaurant applications.

“We have already deployed tens of robots,”

Claim: Dyna Robotics has deployed tens of robots to date.

“but our goal next year is that overall just deploying hundreds of robots.”

Claim: Dyna Robotics aims to deploy hundreds of robots in total next year.

“So I think robotics is the most exciting space today and there's just so many.”

Claim: The speaker considers robotics the most exciting field today.

“And I think what we are seeing today is that there's still Many different approaches to robotics.”

Claim: The speaker says that robotics currently has many different approaches.

“We believe that hardware needs to be built in-house.”

Claim: The speaker's company believes robotics hardware needs to be built in-house.

“We need to own the whole chain.”

Claim: The speaker's company believes it needs to own the entire robotics development and deployment chain.

“We believe that scaling human data is extremely important,”

Claim: The speaker's company considers scaling human data extremely important for robotics.

“but there are companies that's betting on just scaling robots data or scaling some sort of intermediate like data collection devices, right?”

Claim: Some robotics companies are pursuing scale by collecting more robot data or data from intermediate collection devices.

“All of them are in play and I think it's different approaches. We'll just see what happens. I think right now. It's still a bit early to tell,”

Claim: The speaker says multiple robotics approaches remain viable and it is too early to know which will succeed.

“but we feel pretty good about the way we're building things just because we as a company culture-wise, mission-wise, we're not here to just build a model. We're here to truly get robots deployed.”

Claim: The speaker says the company is confident in its approach because its mission is to deploy robots rather than merely build a model.

“So we deploy as much as possible, work on the hard problem so we identify problems much earlier on.”

Claim: The speaker says extensive robot deployment and tackling difficult problems help the company identify problems earlier.

“So then I think that really helps us identify what is the true bottleneck and then solve it. And I think”

Claim: The speaker says identifying problems early helps the company find and address the true bottleneck.

“For folks, for people who are only working on one part of the problem, it's kind of like fog of war. You don't really see the whole picture and you might be building something that's extremely cool technologically, but may not be what people need or want at the end, right?”

Claim: The speaker argues that teams focused on only one part of robotics may lack a full view and build technically impressive systems that users do not need or want.

“I think there's a whole graveyard of robotics company and I think all of them had very shiny, very cool robots, but you know, just figuring out. what people actually want is actually it's very easy to say you know like build something people want you know everyone says that but it's actually really hard to execute a robotics company yeah”

Claim: The speaker says many robotics companies have built impressive-looking robots, while determining and delivering what people actually want is difficult.

“yeah i would say that's also not standard so we're building world action model on top of world model so there's many companies building world model you know you mentioned runway there”

Claim: The speaker says world action models are not an industry standard and that the company builds its world action model on top of a world model.

“also getting to adapt video world models. There are also companies building vision language action model. So that's like taking vision language models and adapting them to robotics.”

Claim: The speaker says robotics companies pursue different model approaches, including adapting video world models and adapting vision-language models into vision-language-action models.

“So I think there's many different ways to go about it. But I think at the end of the day, like what we call the models is less important than what we believe the model needs to be able to do at a first principle level.”

Claim: The speaker believes model labels matter less than the capabilities robotics models need to provide.

“the most important thing is that you need to be able to predict the future,”

Claim: The speaker considers future prediction the most important capability for robotics.

“By robotics, any physical system is fundamentally about based on the information you have at the present moment and before, what should you do next? So it's really about predicting what comes next. Predicting robot actions like predicting what the robot should do next.”

Claim: The speaker describes robotics as predicting what happens next and what action a robot should take, using present and prior information.

“But if you think about it, robot is just a part of the world, right? Robot influences the world, but the world also influences how the robot should behave in that environment.”

Claim: The speaker says robots both affect their surroundings and are affected by the world when determining how to behave.

“the most first principle thing you can do is just predict as much of the world that you can possibly predict,”

Claim: The speaker argues that a first-principles robotics approach should predict as much of the world as possible.

“So that's why we believe predicting the future is extremely important.”

Claim: The speaker's company considers future prediction extremely important for robotics.

“the instantiation is world action model.”

Claim: The speaker says the company's current implementation of its robotics approach is a world action model.

“And VLA only predicts robot actions or some more abstract representation of the world.”

Claim: The speaker says a vision-language-action model predicts robot actions or an abstract representation of the world.

“But whenever you're predicting abstract representation of the world, you're encoding your own bias into what's important to predict.”

Claim: The speaker argues that choosing an abstract world representation embeds the modeler's biases about which aspects matter for prediction.

“And the bitter lesson tells you. Let's remove human biases. Let's just take the most raw representation, the camera inputs and just predict all of that, right?”

Claim: The speaker invokes the bitter lesson to argue for predicting raw camera inputs rather than selecting a human-designed abstract representation.

“Like if you're trying a world model that's purely based on videos, I think it will be a very powerful world model, but it may not be a sufficiently rich enough world model.”

Claim: The speaker thinks a video-only world model could be powerful yet may lack sufficient richness for robotics.

“because you're missing critical aspects of the world like sound like touch right so i think what where we could be wrong is like okay i have the best world model on vision vision-based world model but tomorrow someone else comes out with a vision plus tactile world model that's just like a hundred x more data efficient just because you know vision is a very”

Claim: The speaker gives the hypothetical that a vision-and-tactile world model could achieve 100× the data efficiency of a vision-only model.

“because you're missing critical aspects of the world like sound like touch right so i think what where we could be wrong is like okay i have the best world model on vision vision-based world model but tomorrow someone else comes out with a vision plus tactile world model that's just like a hundred x more data efficient just because you know vision is a very”

Claim: The speaker says video-only world models may omit important sensory information, including sound and touch.

“high information or like highly redundant. you know, input because, you know, every other pixel, they don't change that much.”

Claim: The speaker characterizes visual input as information-rich but highly redundant, with adjacent pixels often changing little.

“You can see, you can understand the world without perfect vision,”

Claim: The speaker says perfect vision is not necessary to understand the world.

“but tactile, you know, other modality might be much denser in information,”

Claim: The speaker suggests tactile sensing or another modality may carry denser information than vision.

“So maybe having that, predicting that would just allow you to use much less vision so you can get to the same performance with much less compute, with much less data,”

Claim: The speaker hypothesizes that predicting tactile information could reduce the vision, compute, and data needed to reach the same performance.

“So the reason I bring this up is because I think if vision tactile world model or vision tactile audio, whatever world model is the final thing that solves robotics, then I think robotics companies will be able to do that better than.”

Claim: The speaker believes robotics companies would outperform video-generation companies pivoting to robotics at developing the eventual solution if it requires a vision-tactile or vision-tactile-audio world model.

“a video generation companies is trying to pivot to robotics the reason is like to understand sensing is actually very complicated like we've been building that since the company started so we know the bottleneck we know how to combine them but for if you're just trying to adapt the video model to robotics action then like you know if we were wrong”

Claim: The speaker says sensing is complicated and that the company has developed relevant know-how since its founding, including knowledge of bottlenecks and how to combine modalities.

“a video generation companies is trying to pivot to robotics the reason is like to understand sensing is actually very complicated like we've been building that since the company started so we know the bottleneck we know how to combine them but for if you're just trying to adapt the video model to robotics action then like you know if we were wrong If, you know, video model is not enough that like you don't really have the know-how to get tactile to work very well, but we have that.”

Claim: The speaker argues that if video models prove insufficient for robotics, video-model companies may lack tactile-sensing expertise that the speaker's company has.

“it could be another color another input to the model.”

Claim: The speaker says infrared could be treated as another color and as an additional model input.

“yeah maybe it's useful to talk about how i got into ai in the beginning right so uh as a kid so i spent time in japan and china i grew up playing go you know going alpha go so when alpha go came out i was a senior in high school and i was mind blown because i've been playing this game for many many years it's so complicated like i think before alpha go came out people were like okay go is one of the game the ai will never solve the search space is like”

Claim: AlphaGo’s release astonished the speaker because he had played Go for many years and understood its complexity.

“yeah maybe it's useful to talk about how i got into ai in the beginning right so uh as a kid so i spent time in japan and china i grew up playing go you know going alpha go so when alpha go came out i was a senior in high school and i was mind blown because i've been playing this game for many many years it's so complicated like i think before alpha go came out people were like okay go is one of the game the ai will never solve the search space is like larger than the total number of atoms in the world how could you possibly search for solutions in that then like you know RL reinforcement learning just like cracked it right”

Claim: Before AlphaGo, people believed AI would never solve Go because its search space was larger than the number of atoms in the world.

“larger than the total number of atoms in the world how could you possibly search for solutions in that then like you know RL reinforcement learning just like cracked it right”

Claim: Reinforcement learning enabled AI to overcome the challenge of Go.

“And I think that was how I got into AI initially. I was just really interested in models or systems, agents that can self-improve.”

Claim: The speaker’s initial interest in AI centered on models, systems, and agents capable of self-improvement.

“So when I got to Penn as a PhD student, I just really wanted to work on reinforcement learning.”

Claim: While pursuing his PhD at Penn, the speaker wanted to work on reinforcement learning.

“So I'm co-advised. Both my advisors were interested in RLL at the time. One of my advisors was more in robotics.”

Claim: At the time, both of the speaker’s PhD advisors were interested in reinforcement learning, and one focused more on robotics.

“He did a postdoc at Berkeley Robotics Lab.”

Claim: One of the speaker’s PhD advisors did a postdoctoral fellowship at Berkeley Robotics Lab.

“So that's how I got into robots.”

Claim: The speaker says his advisor’s robotics background led him to enter robotics.

“you know, we have GrassLab, I think one of the best places to do robotics.”

Claim: The speaker considers GrassLab one of the best places to conduct robotics research.

“And so that's how I got into doing RL for robotics in my PhD.”

Claim: The speaker’s PhD research involved applying reinforcement learning to robotics.

“And then, yeah, I did a bunch of research thinking about how to connect reinforcement learning with human data,”

Claim: The speaker’s PhD research included investigating ways to connect reinforcement learning with human data.

“how to do reinforcement learning theory in a very rigorous way.”

Claim: The speaker’s research included developing or studying reinforcement learning theory in a rigorous way.

“And then I went to NVIDIA, that's when GPT-4 came out.”

Claim: GPT-4 had just been released when the speaker joined NVIDIA.

“At the time, we were very interested in just solving a pen spinning task.”

Claim: While at NVIDIA, the speaker’s team focused on trying to solve a pen-spinning task.

“let's try to do dexterous pen spinning.”

Claim: The speaker’s team attempted to develop dexterous pen-spinning capability.

“The initial idea was like learning it from human videos, We were trying to watch humans.”

Claim: The initial approach considered for pen spinning was to learn from videos of humans performing the task.

“do it but then I realized okay today we don't have a dexterous hand that can spin pencil at all so how would I even be able to evaluate the algorithm even if learning from human worked at all so I thought okay we need to first try this in simulation then as soon as I tried to set up a simulated environment for a robot hand with a pen I immediately saw what the issue is the issue is like”

Claim: At the time, no available dexterous hand could spin a pencil, preventing the speaker from evaluating the algorithm on a physical hand.

“do it but then I realized okay today we don't have a dexterous hand that can spin pencil at all so how would I even be able to evaluate the algorithm even if learning from human worked at all so I thought okay we need to first try this in simulation then as soon as I tried to set up a simulated environment for a robot hand with a pen I immediately saw what the issue is the issue is like”

Claim: Because a suitable physical hand was unavailable for evaluation, the speaker decided to test the pen-spinning task in simulation first.

“Like, if I had to come up with this simulation environment, the reward function for reinforcement learning to solve pen spinning, that's impossible. Like, I wouldn't be able to come up with all the terms that you need to get RL to learn the right behavior.”

Claim: The speaker believed he could not devise all the reward-function terms needed to train reinforcement learning to perform pen spinning correctly.

“And then I was just thinking, what if a model that can write code can do that?”

Claim: The speaker considered whether a code-writing model could generate the solution needed for the pen-spinning task.

“That's what became the Eureka project,”

Claim: The idea of using a code-writing model to devise the solution became the Eureka project.

“Like in my opinion, we have the first like recursive self-improving robotic agent in simulation because we really like use that to solve a new task that no one has demonstrated before.”

Claim: The speaker considers Eureka the first recursive self-improving robotic agent in simulation, claiming it solved a task that had not previously been demonstrated.

“And that allowed me to see the potential of using language model, using coding models to really accelerate robotics.”

Claim: The speaker says Eureka demonstrated the potential of language models and coding models to accelerate robotics.

“So then I went to Google DeepMind because at the time, I mean, even today they have one of the frontier models in Gemini, right? So I was looking at how to use Gemini for robotics.”

Claim: The speaker joined Google DeepMind to investigate applications of Gemini in robotics.

“So then I went to Google DeepMind because at the time, I mean, even today they have one of the frontier models in Gemini, right?”

Claim: The speaker says Google DeepMind had one of the frontier models in Gemini when he joined and still has one today.

“But eventually I realized like I had a lot of fun doing academic research. But there was always a question in the back of my mind, like, does any of this matter?”

Claim: The speaker enjoyed academic research but questioned whether his laboratory work had practical significance.

“You know, like when you're working on this kind of thing in the lab, like, you know, you asked me, how do we come up with so many interesting fun tasks? you're basically doing your own eval you're grading your own homework like i wanted something more real i wanted something that could actually be used by millions of people in the world and i felt 2024 was when we started dining like end of 2024 that was the right time that we can take some of these technology to actually putting them out in the real world so that's why”

Claim: The speaker wanted to move beyond self-evaluated laboratory research toward technology that could be used by millions of people.

“you're basically doing your own eval you're grading your own homework like i wanted something more real i wanted something that could actually be used by millions of people in the world and i felt 2024 was when we started dining like end of 2024 that was the right time that we can take some of these technology to actually putting them out in the real world so that's why”

Claim: The speaker believed late 2024 was the right time to deploy some of the technology in the real world.

“i started i know with linux new york and uh fast forward two years we build our platform our model our robots are in jensai phone go check it out and uh”

Claim: The speaker says he started a company in New York and that, two years later, its platform, model, and robots were deployed at a location transcribed as “Jensai Phone.”

“like some of a lot of ideas that I temporarily abandoned when we started dynamic because you know to build a company have to be very practical like I think simulation wasn't ready in 2024 but now with you know like the new frontier models and what we're seeing internally like this might change and that's very exciting”

Claim: The speaker temporarily set aside some ideas when starting the company because he believed company-building required practicality.

“like some of a lot of ideas that I temporarily abandoned when we started dynamic because you know to build a company have to be very practical like I think simulation wasn't ready in 2024 but now with you know like the new frontier models and what we're seeing internally like this might change and that's very exciting”

Claim: The speaker believed simulation was not ready in 2024, though new frontier models and internal developments might change the situation.

“yeah i think culture yeah so let's first talk about you know how we run a company right like we think of ourselves as like a like a like eventually like a research plus product company it's kind of open ai where you do frontier research but eventually all of them make their way into codecs to chat gpt that's how we think about it too we're not here to just do open-ended like whatever thing that interests us at the time we're always like product driven like we want to deploy our robots our platform to a new customer what are the technical problems we have to solve right we have to solve generalization we have to solve”

Claim: Dyna Robotics views itself as a research-and-product company that conducts frontier research intended to feed into its products.

“like eventually like a research plus product company it's kind of open ai where you do frontier research but eventually all of them make their way into codecs to chat gpt that's how we think about it too we're not here to just do open-ended like whatever thing that interests us at the time we're always like product driven like we want to deploy our robots our platform to a new customer what are the technical problems we have to solve right we have to solve generalization we have to solve”

Claim: Dyna Robotics prioritizes product-driven work over open-ended projects chosen solely by current research interest.

“like eventually like a research plus product company it's kind of open ai where you do frontier research but eventually all of them make their way into codecs to chat gpt that's how we think about it too we're not here to just do open-ended like whatever thing that interests us at the time we're always like product driven like we want to deploy our robots our platform to a new customer what are the technical problems we have to solve right we have to solve generalization we have to solve of whole body control we have to solve how to adapt quickly we have to solve how to make the robots follow language we have to solve dexterity so once all those problems are very clear then like i mean our engineers are very capable they're very smart so they just create instances of the task they create tasks that would stress test those capabilities right like the grabbing and drink task is just how we test language following right you can just”

Claim: Dyna Robotics identifies generalization, whole-body control, rapid adaptation, language instruction following, and dexterity as technical problems to solve for its robots.

“of whole body control we have to solve how to adapt quickly we have to solve how to make the robots follow language we have to solve dexterity so once all those problems are very clear then like i mean our engineers are very capable they're very smart so they just create instances of the task they create tasks that would stress test those capabilities right like the grabbing and drink task is just how we test language following right you can just”

Claim: Dyna Robotics uses a drink-grabbing task to evaluate its robots’ ability to follow language instructions.

“just specify what you want if the model follows intent follows instruction it should be able to do it so that's how we think about like at the top level we identify what are the things we have to solve then we have a lot of bottoms-up project to like actually drive execution to solve them and you know you hire the right people and everyone here is super passionate about robots you know they've been doing this for a long time they love it and we just create a culture where it just naturally happens”

Claim: Dyna Robotics sets technical priorities at the top level and uses bottom-up projects to execute against those priorities.

“you know we recently grew so much that I'm the perfect number but I would say on the hardware side we have I would say probably like 30% 40% of hardware just because hardware you need a lot more people and then like you know we have like around 10 people working on models and then we have like software engineers robotic engineer we have a lot of very talented people on the business side you know the most fun thing about building a robotics company is like you see everything.”

Claim: The hardware side probably makes up 30% to 40% of Dyna Robotics, and the speaker attributes its relatively large staffing need to hardware requiring more people.

“I would say probably like 30% 40% of hardware just because hardware you need a lot more people and then like you know we have like around 10 people working on models and then we have like software engineers robotic engineer we have a lot of very talented people on the business side you know the most fun thing about building a robotics company is like you see everything.”

Claim: Dyna Robotics has around 10 people working on models.

“Yeah, I think our vision is to truly, the high-level mission is truly bring like, you know, robots and intelligence right into the physical world like actually that means like actually seeing our robots deployed like thousands of customers even more millions of robots that's where we want to get to yeah”

Claim: Dyna Robotics aims to bring robots and intelligence into the physical world, ultimately deploying robots to thousands of customers and reaching millions of robots.

“but internal different options but uh how the name came to be was there are two things one is uh so uh touring award winner richard sudden was like the pioneer reinforcement when they wrote a paper called dina many many many years ago it's about basically an agent that self-improves in a world model and i felt even when we started the company like We weren't doing world model yet, but I felt that's how robotics will get solved.”

Claim: When Dyna Robotics began, it was not working on world models, although the speaker believed world models would be the way to solve robotics.

“but internal different options but uh how the name came to be was there are two things one is uh so uh touring award winner richard sudden was like the pioneer reinforcement when they wrote a paper called dina many many many years ago it's about basically an agent that self-improves in a world model and i felt even when we started the company like”

Claim: The speaker says Dyna Robotics was partly named after a paper called “Dyna,” which describes an agent that self-improves in a world model.

“The second thing is that it's a bit more personal, like I grew up in Japan and, you know, the year I was born, there was just like a. cartoon show i don't know if you guys have seen like ultraman yeah yeah yeah the ultraman that year was called diana”

Claim: The speaker says Dyna Robotics’ name also draws on Ultraman’s Diana, a character associated with the year he was born.

“calls to action yeah i think um i do feel like as an industry robotics is still very demo oriented like i have done a lot of demos in the past in the lab you know even here like we had to show some demos but i think”

Claim: The robotics industry remains highly focused on demonstrations.

“We need more people that's truly mission focused on doing whatever it takes to get robots deployed,”

Claim: The robotics industry needs more mission-focused people committed to getting robots deployed.

“get robotic foundation models truly integrated with hardware and seeing them in real customer sites.”

Claim: Robotic foundation models should be integrated with hardware and deployed at real customer sites.

“I think we are a pioneer on this regard and I want to see more in this field. And for the audience that's watching, if this machine inspires you,”

Claim: The speaker considers their company a pioneer in integrating robotic foundation models with hardware and deploying them at customer sites, and wants to see more work in the field.

“makes you wake up every day and you want to contribute you know we're always hiring and uh yeah looking forward to that but i think just bay area anywhere else uh yes just bay area to that spirit but you know as company grows never say never to other locations”

Claim: The company is hiring and currently focused on the Bay Area, but may consider other locations as it grows.

“but that's what I want to say and I wanted to see more robots in the world.”

Claim: The speaker wants more robots to be deployed in the world.

“Yeah, yeah, I think for robotics gotta do it on site.”

Claim: Robotics work should be done on site.