Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick

“In fact, the information can be very expensive. And in fact, I would say most information is very very expensive. If you completely say, well constraints don't matter uh I I don't need any prior information, I don't want to encode anything in there, we still have the problem of now needing to train for you re-learn everything from scratch. He believes that you can figure out like a a suitable reward function for every task. That is enough. Like I think that energy-based models are not really useful.”

Claim: An unnamed person believes that finding a suitable reward function for every task is sufficient.

“In fact, the information can be very expensive. And in fact, I would say most information is very very expensive. If you completely say, well constraints don't matter uh I I don't need any prior information, I don't want to encode anything in there, we still have the problem of now needing to train for you re-learn everything from scratch. He believes that you can figure out like a a suitable reward function for every task. That is enough. Like I think that energy-based models are not really useful.”

Claim: If a system uses no constraints or prior information, it still must be trained to relearn everything from scratch.

“In fact, the information can be very expensive. And in fact, I would say most information is very very expensive. If you completely say, well constraints don't matter uh I I don't need any prior information, I don't want to encode anything in there, we still have the problem of now needing to train for you re-learn everything from scratch. He believes that you can figure out like a a suitable reward function for every task. That is enough. Like I think that energy-based models are not really useful.”

Claim: The speaker thinks energy-based models are not really useful.

“In fact, the information can be very expensive. And in fact, I would say most information is very very expensive. If you completely say, well constraints don't matter uh I I don't need any prior information, I don't want to encode anything in there, we still have the problem of now needing to train for you re-learn everything from scratch. He believes that you can figure out like a a suitable reward function for every task. That is enough. Like I think that energy-based models are not really useful.”

Claim: Most information is very expensive to obtain or use.

“Oh, at least not useful anymore. I think in world models and so both japa and world model are ultimately branding. There jap there were japa like models before japa and there were world models before world models. Right now all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff, how do you kno like how many times did it fail to do something.”

Claim: The speaker says current methods are designed to look good in demonstrations and perform some impressive tasks.

“Oh, at least not useful anymore. I think in world models and so both japa and world model are ultimately branding. There jap there were japa like models before japa and there were world models before world models. Right now all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff, how do you kno like how many times did it fail to do something.”

Claim: The speaker raises the concern that demonstrations may not reveal how often current methods fail.

“Oh, at least not useful anymore. I think in world models and so both japa and world model are ultimately branding. There jap there were japa like models before japa and there were world models before world models. Right now all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff, how do you kno like how many times did it fail to do something.”

Claim: Models resembling JEPA existed before JEPA, and world models existed before they were called world models.

“Oh, at least not useful anymore. I think in world models and so both japa and world model are ultimately branding. There jap there were japa like models before japa and there were world models before world models. Right now all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff, how do you kno like how many times did it fail to do something.”

Claim: The speaker clarifies that energy-based models are at least no longer useful.

“Oh, at least not useful anymore. I think in world models and so both japa and world model are ultimately branding. There jap there were japa like models before japa and there were world models before world models. Right now all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff, how do you kno like how many times did it fail to do something.”

Claim: The speaker considers both JEPA and world models to be ultimately branding terms.

“Yeah. So my name is Alexander Magic. I've been I so I first did my bachelor's at FAU in Erlangen in Germany. Then I also did my master's there. So my bachelor's was in computer science, my master's is in artificial intelligence. And then I went over to Fraunhofer, which is kind of one of the biggest like applied research institutes. And now I'm also dual affiliated with the Technical University of Nuremberg.”

Claim: Alexander Mattick says he went to Fraunhofer, which he describes as one of the largest applied research institutes.

“Yeah. So my name is Alexander Magic. I've been I so I first did my bachelor's at FAU in Erlangen in Germany. Then I also did my master's there. So my bachelor's was in computer science, my master's is in artificial intelligence. And then I went over to Fraunhofer, which is kind of one of the biggest like applied research institutes. And now I'm also dual affiliated with the Technical University of Nuremberg.”

Claim: Alexander Mattick says he is dual-affiliated with the Technical University of Nuremberg.

“Yeah. So my name is Alexander Magic. I've been I so I first did my bachelor's at FAU in Erlangen in Germany. Then I also did my master's there. So my bachelor's was in computer science, my master's is in artificial intelligence. And then I went over to Fraunhofer, which is kind of one of the biggest like applied research institutes. And now I'm also dual affiliated with the Technical University of Nuremberg.”

Claim: Alexander Mattick says he earned a bachelor's degree in computer science and a master's degree in artificial intelligence at FAU in Erlangen, Germany.

“There is kind of an still under construction kind of in the very much in the build up of um becoming a real thing. So it's really exciting to to to actually be basically one of the first people who actually does like its P_H_D_ there.”

Claim: The speaker says the Technical University of Nuremberg is still being built up and is not yet fully established.

“There is kind of an still under construction kind of in the very much in the build up of um becoming a real thing. So it's really exciting to to to actually be basically one of the first people who actually does like its P_H_D_ there.”

Claim: Alexander Mattick says he is among the first people pursuing a PhD at the Technical University of Nuremberg.

“Let us plick a group of functions which is always gonna be normalised. And this gives us kind of the like very traditional variation in inference uh problem. Effectively we have a normal distribution with unknown parameters and now we try to fit them onto data.”

Claim: Traditional variational inference fits data using a parameterized distribution family whose members are guaranteed to be normalized.

“The issue with this is, and people notice is very quickly, is that now you are very constrained by the actual density. So let's say your data is not normally distributed. Let's say it has let's say you it's bimodal. Now, your actual density no fits, right? So basically what would happen is we if you imagine you have like a a mode here and we have a mode here, then a non then just fitting a normal will put uh all the mass basically in the middle and kind of try to split the difference.”

Claim: A restricted density family can poorly fit data with a different distribution; specifically, fitting a normal distribution to bimodal data places probability mass between the modes.

“The first idea which kind of came up more or less in the same time was doing something like rejection sampling. So rejection sampling something that you may have already seen like in school where you try where you did like you you uh painted a square, you had a circle inside of it and they did like random points. And then the fraction of random points that is in the circle, you can use that to compute pi. Exactly the same idea you can do with any kind of density as long as you have something we just just kind of above it. Right, let's say we have uh like a just a box which is I have which is essentially an uh a uniform distribution, then you scale it up to be basically around your density, and then you randomly pepper with points. That's rec rejection sampling, and it works pretty well assuming you have uh a relatively close envelope over your function. So it doesn't end up scaling very well, especially with high dimensional data.”

Claim: Rejection sampling works reasonably well when its envelope closely bounds the target density, but scales poorly, particularly for high-dimensional data.

“data. Right, because you again it's one of the uh quintessential examples of like this uh curse of dimensionality.”

Claim: Rejection sampling illustrates the curse of dimensionality.

“So then people went and said, well, we can do a little bit better by framing this as basically as the limiting process of a Markov uh process. And this gives us something, M_C_M_C_ what basically the idea is that we may not be able to create the dentist density, but we are able to sample from it. And that's one of the very interesting things about kind of everything inference or like density based is that you can actually do a lot of things without actually knowing the density itself, you just by sampling and rejecting samples.”

Claim: Markov chain Monte Carlo can generate samples from a target density without directly constructing that density.

“And this gives us something, M_C_M_C_ what basically the idea is that we may not be able to create the dentist density, but we are able to sample from it. And that's one of the very interesting things about kind of everything inference or like density based is that you can actually do a lot of things without actually knowing the density itself, you just by sampling and rejecting samples.”

Claim: Some density-based inference tasks can be carried out using samples without explicitly knowing the density.

“The issue with this is that if you've ever run, like, Markov chain Monte Carlo, you will notice this is insanely slow, especially in high dimensions. Again, we have cursive dimensionality, you are basically uh it c not only do you have like these high dimensional problems where you have many modes and kind of escaping from mo one mode to the next is super difficult, but also just by the fact that you have still like an accept and reject step, you end up kind of doing a lot of work. for things that don't end up going anywhere.”

Claim: Markov chain Monte Carlo can be extremely slow, particularly in high dimensions, because transitions between modes are difficult and accept-reject steps expend effort on proposals that do not advance the sampling process.

“So G-Flow networks are kind of an interesting case, but I don't think they really solve the same problem.”

Claim: The speaker does not think GFlowNets solve the same problem as the inference methods under discussion.

“So what a G_ flow network does is they uh essentially let's say you have like a um a disentangled distribution, right? You have effectively a graph which you walk around and you can construct your sample up slowly over time.”

Claim: A GFlowNet generates samples incrementally through traversal of a graph.

“And the idea of a G_ flow network is if we have some let's say final density in the end uh which is For example like a r uh effective is a reward function. And then the idea of G_ flow networks is that every node can have the inflowing and outflowing net uh kind of energy has to be the same.”

Claim: A GFlowNet requires the net energy flowing into each node to equal the net energy flowing out.

“So essentially what you end up doing is you end up intensifying the paths which lead to higher reward and you reduce the ones which lead to a lower reward.”

Claim: GFlowNets increase the emphasis on paths leading to higher rewards and decrease it on paths leading to lower rewards.

“However, you basically are back into a close to the variational nature, where effectively we'll make the assumption that you have a specific structure, and you assume that structure is correct. Which of course if you have, let's say, some like molecular problem where effectively you start building up something, it's pretty like we can assume that this process is accurate. But this is not a general way of doing inference. Right, the general way is still uh you need to do either like sampling where you have a very very general general function family to actually be able to do that yeah”

Claim: GFlowNets depend on an assumed structure that is treated as correct, so they are not a general-purpose method of inference.

“so energy-based models actually directly come from MCMC right the idea or like the reasons called energy is because it comes from physics like in the case of physics they literally had like a function of energy and then you would assume that kind of”

Claim: Energy-based models derive from Markov chain Monte Carlo and are called energy-based because their formulation is connected to physics.

“the like particles kind of conform proportional to the energy. Right? If you let's say you have a uh a specific uh box with hot particles in there, you would imagine they follow some form of distribution because of every particle is like fast. Some must some allowed faster, some allowed slower. And that gives you kind of this statistical mechanics point of view. And indeed, I think it was I think it was Teller who did actually the first developments on like M_C_M_C_ and kind of”

Claim: The speaker describes statistical mechanics as modeling particle distributions in relation to energy, including variation in particle speeds.

“the like particles kind of conform proportional to the energy. Right? If you let's say you have a uh a specific uh box with hot particles in there, you would imagine they follow some form of distribution because of every particle is like fast. Some must some allowed faster, some allowed slower. And that gives you kind of this statistical mechanics point of view. And indeed, I think it was I think it was Teller who did actually the first developments on like M_C_M_C_ and kind of got into this idea of actually kind of doing simulations to actually get the the actual parameter or like the particle distributions”

Claim: The speaker thinks Teller made the first developments in MCMC and in using simulations to obtain particle distributions.

“An energy-based model is any model that parameterizes complex densities using an energy function.”

Claim: An energy-based model parameterizes complex probability densities with an energy function.

“Energy functions correspond to an unnormalized density. The lower the energy, the more favored the state, just like in physics where low energy states are preferred.”

Claim: In an energy-based model, the energy function represents an unnormalized density, with lower-energy states favored.

“Being unnormalized makes E_B_M_s quite easy to train, since no complex intervals need to be computed during training.”

Claim: Energy-based models are relatively easy to train because their unnormalized formulation avoids computing complex integrals during training.

“an energy based optimization simply pushes up areas with low sample support, and down areas with high sample support.”

Claim: Energy-based optimization increases values in regions with little sample support and decreases values in regions with substantial sample support.

“This makes energy based modelling quite flexible, with many different objectives, all targeting slightly different assumptions, where it comes to, for instance, availability of a reference energy to track.”

Claim: Energy-based modeling supports varied objectives that rely on different assumptions, including whether a reference energy is available for tracking.

“When easy to train, this comes at a steep cost during sample generation.”

Claim: Energy-based models' relative ease of training comes at a high cost during sample generation.

“Its energies are unnormalized, we need to perform complex Markov chain Monte Carlo optimization to sample states proportionally to the energy without having access to a normalization constant.”

Claim: Sampling from an energy-based model requires complex Markov chain Monte Carlo optimization to draw states according to the unnormalized energy without a normalization constant.

“This makes energy-based modelling unfavorable for generative usage.”

Claim: The sampling burden makes energy-based modeling unfavorable for generative applications.

“The most expensive part of sampling gets executed over and over again during inference, instead of being amortized during training.”

Claim: For energy-based models, the most expensive sampling computation is repeated at inference instead of being amortized during training.

“energy-based models do not offer access to the probability of any sample. But it's helpful for many applications such as anomaly detection or to correct accumulated errors when unrolling a dynamics model.”

Claim: Energy-based models do not provide sample probabilities, which can be useful for anomaly detection and for correcting accumulated errors when unrolling a dynamics model.

“One is I think most of deep learning is actually doing inference.”

Claim: The speaker thinks that most deep learning is a form of inference.

“One is I think most of deep learning is actually doing inference.”

Claim: The speaker argues that most deep learning is performing inference.

“And if you go and you fit a categorical distribution, like you do your large language model, you have your kind of density, which is effectively like an autoregressive structure, that is just a decomposition of your final density.”

Claim: A large language model fits a categorical distribution whose autoregressive structure decomposes the overall probability density.

“wha a to model the probability of generating a certain sentence given uh let's say a prompt. And what you end up doing is you end up kind of producing kind of step by step uh the the actual answer by going token by token by token. This works really nicely in something like uh language modelling, because you ultimately still have a discrete measure.”

Claim: Token-by-token generation works well for language modeling because language uses a discrete measure.

“wha a to model the probability of generating a certain sentence given uh let's say a prompt. And what you end up doing is you end up kind of producing kind of step by step uh the the actual answer by going token by token by token. This works really nicely in something like uh language modelling, because you ultimately still have a discrete measure.”

Claim: Autoregressive language models model the probability of generating a sentence given a prompt by producing an answer token by token, which works well because language has a discrete measure.

“Right, you have all your tokens and you choose which one to actually get and a categorical distribution is more or less like it's the universal thing I can always just write down the probabilities of everything”

Claim: A categorical distribution can represent the probabilities of all possible tokens.

“Right, you have all your tokens and you choose which one to actually get and a categorical distribution is more or less like it's the universal thing I can always just write down the probabilities of everything”

Claim: A categorical distribution can specify probabilities for every token in a discrete set.

“Even if you look at something like regression or uh yeah uh let's say you do like image uh detection, like uh let's say you do depth estimation, something like that, you s uh with like your mean square error, you're still doing inference, right? It's the mean square error is just a log uh negative log likelihood of the uh of your data under a normal distribution with a fixed standard deviation. And so you are still”

Claim: Using mean squared error corresponds to using the negative log-likelihood of the data under a normal distribution with fixed standard deviation.

“Even if you look at something like regression or uh yeah uh let's say you do like image uh detection, like uh let's say you do depth estimation, something like that, you s uh with like your mean square error, you're still doing inference, right? It's the mean square error is just a log uh negative log likelihood of the uh of your data under a normal distribution with a fixed standard deviation. And so you are still”

Claim: Regression, image detection, and depth estimation can be understood as inference because mean squared error corresponds to the negative log likelihood under a normal distribution with fixed standard deviation.

“Doing inference at some point, what you lose is you lose kind of the the breadth of the density. But that's more an assumption, right? You can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like let's say uh like my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say I don't actually want a point estimate.”

Claim: Point-estimate methods discard the breadth of a predictive density, but a model can instead output a mean and standard deviation.

“Doing inference at some point, what you lose is you lose kind of the the breadth of the density. But that's more an assumption, right? You can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like let's say uh like my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say I don't actually want a point estimate.”

Claim: Point-estimate approaches lose the breadth of the density, but estimating a mean and standard deviation is an alternative.

“Doing inference at some point, what you lose is you lose kind of the the breadth of the density. But that's more an assumption, right? You can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like let's say uh like my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say I don't actually want a point estimate.”

Claim: In reinforcement learning, retaining a distribution rather than only a point estimate can improve exploration and algorithm performance.

“Doing inference at some point, what you lose is you lose kind of the the breadth of the density. But that's more an assumption, right? You can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like let's say uh like my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say I don't actually want a point estimate.”

Claim: The speaker says that in reinforcement learning, retaining uncertainty rather than using only a point estimate can be one of the best ways to improve algorithm performance through better exploration.

“I want some breath to do better exploration. And that's the big advantage of let's say a policy gradient method over a Q learner. Because a Q learner only tells you kind of what to do. It doesn't tell you like if you do the different thing, I expect you to be worse by this and that amount. And maybe like if you had this, you would be able to go and say well for technical exploration, I choose the one which is might be slightly worse, but ends up being better in the long run.”

Claim: Compared with a Q-learner, a policy-gradient method can represent uncertainty about alternative actions, enabling exploration of an action that may be slightly worse immediately but better over the long run.

“I want some breath to do better exploration. And that's the big advantage of let's say a policy gradient method over a Q learner. Because a Q learner only tells you kind of what to do. It doesn't tell you like if you do the different thing, I expect you to be worse by this and that amount. And maybe like if you had this, you would be able to go and say well for technical exploration, I choose the one which is might be slightly worse, but ends up being better in the long run.”

Claim: Compared with a Q learner, a policy gradient method can represent uncertainty about alternative actions and their expected outcomes, potentially enabling exploration of an action that is slightly worse initially but better in the long run.

“So if you're interested in something like active learning, then you also want densities. So I would push back on the fundamental notion that we are not doing inference if we look at uh like normal deep learning.”

Claim: The speaker rejects the notion that standard deep learning is not performing inference.

“So if you're interested in something like active learning, then you also want densities. So I would push back on the fundamental notion that we are not doing inference if we look at uh like normal deep learning.”

Claim: Density estimates are useful for active learning.

“So if you're interested in something like active learning, then you also want densities. So I would push back on the fundamental notion that we are not doing inference if we look at uh like normal deep learning.”

Claim: The speaker argues that active learning benefits from density representations and rejects the claim that ordinary deep learning is not performing inference.

“because it really depends on what you end up using your density for. Right? If you like let's say you are like a completely deep learning build, you want to have just a large model which kind of describes the density of images. Right? You do diffusion, we have kind of get to that, but we are doing diffusion for image modelling. Then what you ultimately interested in is you want a density”

Claim: The criteria for a good density depend on its intended use.

“because it really depends on what you end up using your density for. Right? If you like let's say you are like a completely deep learning build, you want to have just a large model which kind of describes the density of images. Right? You do diffusion, we have kind of get to that, but we are doing diffusion for image modelling. Then what you ultimately interested in is you want a density see, which explains images the best, right? So you want to have them as high of a likelihood as possible for like real images, or if you have like a a prompt that given the prompt the high likelihood for things that are actually of that uh kind of prompt description.”

Claim: For deep-learning image modeling, the goal is to learn a density that assigns high likelihood to real images and to images matching a prompt.

“because it really depends on what you end up using your density for. Right? If you like let's say you are like a completely deep learning build, you want to have just a large model which kind of describes the density of images. Right? You do diffusion, we have kind of get to that, but we are doing diffusion for image modelling. Then what you ultimately interested in is you want a density see, which explains images the best, right? So you want to have them as high of a likelihood as possible for like real images, or if you have like a a prompt that given the prompt the high likelihood for things that are actually of that uh kind of prompt description.”

Claim: For image modeling, a useful density assigns high likelihood to real images and to images that fit a given prompt.

“Ultimately kind of what you're actually interested in is just kind of a density which has which is high enough on the kind of good parts and low enough on everything else. And that of course then gives you kind of that then goes into the discussion of like a prior.”

Claim: A useful density assigns sufficiently high probability to desirable outcomes and sufficiently low probability to other outcomes.

“Ultimately kind of what you're actually interested in is just kind of a density which has which is high enough on the kind of good parts and low enough on everything else. And that of course then gives you kind of that then goes into the discussion of like a prior.”

Claim: A useful density should assign sufficiently high probability to desirable regions and sufficiently low probability to other regions.

“A prior effectively just tells you well we don't actually want to memorize our data set we want to have like a say a smooth from peak to peak.”

Claim: A prior discourages memorizing the dataset and supports smooth transitions between peaks in the modeled distribution.

“A prior effectively just tells you well we don't actually want to memorize our data set we want to have like a say a smooth from peak to peak. to effectively get like images in between the data points. And that is a prior. But a prior is just a thing that tells you from like your discrete data set, how can I get from let's say the the fifty points of my data set to the fifty first point between two different samples. But if you didn't have a prior then”

Claim: A prior constrains a model against simply memorizing its dataset and supports interpolation between data points, such as generating images between observed examples.

“to effectively get like images in between the data points. And that is a prior. But a prior is just a thing that tells you from like your discrete data set, how can I get from let's say the the fifty points of my data set to the fifty first point between two different samples. But if you didn't have a prior then”

Claim: A prior provides a way to generate or infer images between observed data points.

“Yeah, ultimately the like the exact solution would be just memorise the data and it would be rather loose useless. If you go on the other extreme what the Bayesian people do, they say well,”

Claim: Without a prior, the exact solution would be to memorize the dataset, which the speaker describes as rather useless.

“Yeah, ultimately the like the exact solution would be just memorise the data and it would be rather loose useless. If you go on the other extreme what the Bayesian people do, they say well,”

Claim: Without a prior, the exact solution would be memorization of the dataset, which the speaker considers rather useless.

“I actually like usually what you what you then go go is you say you want information in your latents or just basically reveal information which is not in the data by looking at latents. So classic example is I flip a coin I can observe kind of heads or tails and what I'm actually interested in is is the coin fair or not. If the coin isn't fair then kind of you will see this in the latent.”

Claim: Latent variables can encode information inferred from observations but not directly contained in the observed data, such as whether a coin is fair based on coin flips.

“I actually like usually what you what you then go go is you say you want information in your latents or just basically reveal information which is not in the data by looking at latents. So classic example is I flip a coin I can observe kind of heads or tails and what I'm actually interested in is is the coin fair or not. If the coin isn't fair then kind of you will see this in the latent.”

Claim: Latent variables can represent information not directly observed in the data, such as whether a coin is fair based on observed heads or tails.

“latent because the latent is effectively is a is a distribution over, let's say a probability, it could be.”

Claim: A latent variable can be understood as a distribution over a probability.

“And if it's the case that the probability is, let's say it's 99% sure that the coin is fair, I can accept the hypothesis that the coin is fair.”

Claim: When the probability that a coin is fair is 99%, one can accept the hypothesis that the coin is fair.

“And if it's the case that the probability is, let's say it's 99% sure that the coin is fair, I can accept the hypothesis that the coin is fair.”

Claim: If the inferred probability that a coin is fair is 99%, the speaker says one can accept the hypothesis that the coin is fair.

“If it's 99% that it always lands on tails, then well, I can make the inference that it's actually an unfair coin.”

Claim: If the probability that a coin always lands on tails is 99%, the speaker says one can infer that the coin is unfair.

“If it's 99% that it always lands on tails, then well, I can make the inference that it's actually an unfair coin.”

Claim: When there is a 99% probability that a coin always lands on tails, one can infer that the coin is unfair.

“And that is what like the statisticians people are usually interested in. And it's a big divide between someone who does like statistics and like let's say the medical domain. who actually wants to figure out given those tests where in AI could you have a disease compared to someone in deep learning who just cares about hey I want to generate pretty images of I don't know pink elephants”

Claim: Statisticians in medical contexts often use test results to infer whether a person has a disease.

“And that is what like the statisticians people are usually interested in. And it's a big divide between someone who does like statistics and like let's say the medical domain. who actually wants to figure out given those tests where in AI could you have a disease compared to someone in deep learning who just cares about hey I want to generate pretty images of I don't know pink elephants”

Claim: Statisticians in medical domains are typically interested in inferring whether a person has a disease from test results.

“who actually wants to figure out given those tests where in AI could you have a disease compared to someone in deep learning who just cares about hey I want to generate pretty images of I don't know pink elephants”

Claim: The speaker contrasts medical statistical inference with deep learning work focused on generating visually appealing images, such as pink elephants.

“who actually wants to figure out given those tests where in AI could you have a disease compared to someone in deep learning who just cares about hey I want to generate pretty images of I don't know pink elephants”

Claim: Deep-learning practitioners may prioritize generating visually appealing images, such as images of pink elephants, rather than medical inference.

“You know, we're machine learning people, we have a finite sample of data, and our purposes might be, you know, at some point, we've got to interpolate between these data points, we've got the curse of dimensionality and so on. So we have to put some kind of, I mean, we're amortizing this process, and we have to smooth it and make it amenable to the types of prediction that we're doing.”

Claim: Because machine-learning practitioners have finite data and face the curse of dimensionality, they may need to interpolate between data points and smooth and amortize the process to support prediction.

“You know, we're machine learning people, we have a finite sample of data, and our purposes might be, you know, at some point, we've got to interpolate between these data points,”

Claim: Machine-learning practitioners work with finite data samples and may need to interpolate between observed data points.

“we've got the curse of dimensionality and so on. So we have to put some kind of, I mean, we're amortizing this process, and we have to smooth it and make it amenable to the types of prediction that we're doing.”

Claim: Because of challenges such as the curse of dimensionality, machine-learning practitioners need to smooth their modeling process and make it suitable for their prediction tasks.

“It uses the property that you can learn basically arbitrary densities, assuming your energy function is kind of rich enough that I can approximate the like energy at least, then you can effectively just pawn off the problem of normalizing density to later, to to sampling time.”

Claim: An energy-based model with a sufficiently expressive energy function can represent essentially arbitrary densities and defer density normalization to sampling time.

“But if you have if you set up your uh like your loss function in some clever way, you end up being able to uh to to do the training at least without needing to do sampling. And that is a huge advantage because now you can um if you can directly fit to like an energy which you might have uh like literally measured, or you you do this for let's say samples of some density and you try to push up the energy for where the samples are rich and you push it down where the samples are not rich. Exactly the same way we see do with the density. The problem of course you run into with this is that now”

Claim: With a suitably designed loss function, energy-based models can be trained without sampling, which the speaker considers a major advantage.

“Now you have kind of t the difficulty of both worlds. Like you have the problem of like variation inference where hey what if my function is not strong enough, and then you have the problem of the sampling based world where suddenly you have the issue of uh my actual sampling process is really really slow. So and this is kind of what always limited energy based models.”

Claim: Energy-based models are limited by both insufficient function expressiveness and potentially very slow sampling.

“is this idea of what if we kind of we what if we simplify our problem a little bit and effectively give ourselves like an infinite pool of latents. Because latents in kind of influence our power. Because if you can uh if you can split your problem up into kind of diff smaller subproblems, but as conditioning on some information, that can make it conditionally independent, and these subproblems can become a lot a lot simpler. So in many cases a model becomes cheaper if the more latency you have. Of course it's not always true, but uh generally speaking you want to have kind of flexibility, that you have tiny steps on like your which is”

Claim: Conditioning on information can divide a problem into smaller, conditionally independent subproblems that are substantially simpler.

“subproblems, but as conditioning on some information, that can make it conditionally independent, and these subproblems can become a lot a lot simpler. So in many cases a model becomes cheaper if the more latency you have. Of course it's not always true, but uh generally speaking you want to have kind of flexibility, that you have tiny steps on like your which is”

Claim: In many cases, increasing the number of latent variables makes a model cheaper, though this relationship is not universal.

“what you for example parameterise. And then the the actual let's say uh big steps are just by the fact that you have latent which you conditional. And of course the like absolute limit would be you say well I what if I have infinite latent. And that's we give it a diffusion. You have diffusion is the idea of hey, I have a Markov process, similar to like M_C_M_C_ like you have a Markov process with just with infinite steps. I can make it smooth, continuous.”

Claim: Diffusion can be formulated as a smooth, continuous Markov process with infinitely many steps.

“And you go and you say, well at every step I now, I now sample based on the the previous step's information, like in the normal Markov set.”

Claim: A diffusion process samples each step conditioned on information from the preceding step, following a Markov structure.

“This makes your p uh this has one big difficulty, namely now you actually need to model uh effectively time. Because you effectively what you know saying is we have latency, like the latent at time step zero point five. So and of course if you go down to the latent ed point at time point zero point six, you end up kind of having effectively a different density. So now we have this time dimension as an additional variable. So this is what you pay for. But in general this is much much cheaper still than uh needing to do like the actual M_C_M_C_ process because now you just you follow essentially a density so with like the randomised uh like i you randomly”

Claim: Diffusion requires modeling time as an additional variable because latent states at different time points have different densities.

“So and of course if you go down to the latent ed point at time point zero point six, you end up kind of having effectively a different density. So now we have this time dimension as an additional variable. So this is what you pay for. But in general this is much much cheaper still than uh needing to do like the actual M_C_M_C_ process because now you just you follow essentially a density so with like the randomised uh like i you randomly uh inject more i uh noise at every time step. But you no longer need to have like this accept reject step, you no longer have the problem of like local minima and so on, because essentially what you did is you've rephrased this as a stochastic differential equation which um essentially then goes from like the the initial density to final density. And this is just by basically paying for with one additional variable which effectively indexes over the latents. And this gives you a huge improvement because you no longer need to uh do effectively this this time, it still exists in M_C_M_C_ like it's the the the sampling steps. But effectively you amortize that as well and that makes it a lot cheaper.”

Claim: The speaker says the diffusion-based stochastic differential equation approach is much cheaper than actual MCMC and avoids accept-reject steps and local minima while transforming an initial density into a final density.

“well, why don't we go away from this Markovian structure in the first place? What if we say... uh or maybe not from this macobian structure but maybe from a specific let's say like uh diffusion style like this Fokker Planck style, what if instead we translate phrases as like reparametrising a density from beginning to end, kind of in tiny tiny increments. And that then gives you uh what is effectively these flow measures or like uh the c or more generally continuous normalising flows. Continuous normalising flows are effectively phrasing the problem”

Claim: Continuous normalizing flows represent a density transformation from an initial to a final density through tiny increments, without requiring a particular Markovian or diffusion-style structure.

“Of finding a density as you start off with kind of initial guess. Just as an just as an optimization you start with initial guess, which is like your normal distribution and then you end up with your your final density. And what you end up doing is you need to kind of find effectively an optimization process which gets you from there to there, which is a gradient flow which is what gives you these uh like effectively optimal transport as style problems.”

Claim: A continuous normalizing flow can be framed as a gradient-flow optimization that transforms an initial density, such as a normal distribution, into a target density, yielding an optimal-transport-style problem.

“Uh uh because the change over time has proportional to like the uh divergence of the vector field of the the current density. And if you actually if you go and you you write this out in uh kind of without invoking densities, but let's say we invoke this in with points, you see that this is just uh kind of doing gradient descent in continuous domain. And of course it's the cheapest thing at all, because now you no longer do any sampling at all.”

Claim: The speaker says continuous density evolution can be formulated as gradient descent in a continuous domain, eliminating the need for sampling.

“Oh, now you phrase the problem as I do optimization in infinite dimensional space. So you you pay for infinite dimensions, but you get the the lack of sampling. So the the effectively the entire trajectory was you start off with like the deep very expensive sampling and like the variational models which were very limited. Then you said well let's do kind of both in the middle which were these energy based models. And then you said Well, can we make the energy based models less like like rejection sample-y, and that then gives you the fusion models. And now you say well, when we actually do that,”

Claim: The speaker describes a progression from expensive sampling and limited variational models, through energy-based models, toward flow models intended to reduce rejection sampling.

“this trajectory from like a f initial distribution to a final distribution, we don't need to do kind of this uh this Markov chain process at all. We can phrase it as just like a pure optimisation problem without any stochasticity by instead kind of updating the density at every point.”

Claim: The speaker says a distribution-to-distribution trajectory can be formulated as deterministic optimization by updating density at every point, avoiding a Markov chain process.

“And the only thing you now need to pay for is invertibility, because you need to be able to go kind of in one direction generate samples and the other direction to get the likelihood. You need the likelihood to do uh your to to actually do like inference.”

Claim: The speaker says invertibility is the remaining requirement for generating samples in one direction and computing likelihoods in the reverse direction for inference.

“And that's really cheap because like a uh like a vector field at least in a density sense is invertible. Like you instead going downhill, you just go back uphill.”

Claim: The speaker says a vector field is invertible in the density sense, with reverse traversal undoing forward traversal.

“first a sample gets repeatedly corrupted with normal noise until only normal noise remains.”

Claim: The speaker says diffusion-model training repeatedly adds normal noise to a sample until only normal noise remains.

“The training objective is to simply revert a single corruption step, for instance going back from step t equals five to t equals four.”

Claim: The speaker says the diffusion-model training objective is to learn to reverse a single corruption step at a time.

“This corresponds to learning a vector field, v_t of x, that tells you how to change your sample to make it look more like a target sample.”

Claim: The speaker says a learned time-dependent vector field specifies how to modify a sample to make it resemble a target sample.

“During inference, for sample generation, we simply draw a sample from our normal distribution and follow the learned denoising process backward.”

Claim: The speaker says diffusion-model generation starts with a sample from a normal distribution and follows the learned denoising process backward.

“Inversion like this works, since in the background, this corresponds to solving a stochastic differential equation. This equation is reversible coupling between a normal distribution and your target distribution.”

Claim: The speaker says the inversion corresponds to solving a stochastic differential equation that provides a reversible coupling between a normal distribution and the target distribution.

“This also allows you to compute the probability of any sample from your target distribution. Simply follow the target sample through the noising process and accumulate the diversions of your vector field.”

Claim: The speaker says sample probabilities under the target distribution can be computed by tracing samples through the noising process and accumulating vector-field divergences.

“This shows the major upset of diffusion models. Instead of needing to perform M_C_M_C_ during inference time, we only need to follow a vector field across time, which can be further accelerated using sophisticated O_D_E_ solvers.”

Claim: The speaker identifies diffusion models' advantage as replacing inference-time MCMC with following a vector field over time, with further acceleration possible using ODE solvers.

“The downside of diffusion models is that we now bloat the representation with a synthetic time dimension T_ This extra costs the neural network needs to work with and also slows down training.”

Claim: The speaker says diffusion models' synthetic time dimension increases the neural network's representational burden and slows training.

“We effectively solve many tiny problems of reverting individual nosing steps.”

Claim: The speaker says diffusion training solves many small subproblems, each reversing one noising step.

“However, this is a one-time training expense instead of an inference time expense, making this cheaper for real-world generative usage. Normalizing flows are models that treat density optimization as a reparameterization process.”

Claim: The speaker says diffusion models shift extra computation to one-time training rather than inference, making them cheaper for real-world generative use.

“However, this is a one-time training expense instead of an inference time expense, making this cheaper for real-world generative usage. Normalizing flows are models that treat density optimization as a reparameterization process.”

Claim: The speaker defines normalizing flows as models that treat density optimization as reparameterization.

“Instead of trying to model density from scratch, a known lessing flow starts off with a known base measure and transforms it into the target measure.”

Claim: The speaker says a normalizing flow transforms a known base measure into a target measure instead of modeling density from scratch.

“If the series of transforms applied to the base measure is invertible, we can use this to essentially inherit the normalization of the base measure. We only need to track the local expansion or contraction, which corresponds to the Jacobian of a transformation.”

Claim: The speaker says invertible transformations let a normalizing flow inherit the base measure's normalization, with local volume changes tracked by the transformation Jacobian.

“Sampling simply follows the flow forward. From a base measure sample, apply the model to get the target measure sample.”

Claim: The speaker says normalizing-flow sampling applies the flow forward to transform a base-measure sample into a target-measure sample.

“Computing probabilities goes the other direction. Follow the flow backward from the target back to base by rescaling base in the function of Jacobian.”

Claim: The speaker says normalizing-flow probability computation traces backward from the target measure to the base measure and accounts for the Jacobian.

“One core limitation of naive normalizing flows is the constraints that functions have to be invertible for sampling to work.”

Claim: The speaker says naive normalizing flows require invertible functions for sampling, which is a core limitation.

“a continuous normalizing flow frames the normalizing flow as following a path from the base to target measure based on a vector field that gives local directions.”

Claim: The speaker says a continuous normalizing flow models the path from a base measure to a target measure using a vector field that supplies local directions.

“The trick is that such a path, a gradient flow, is necessarily invertible, even if the function giving you the vector field is not.”

Claim: The speaker says a gradient-flow path is necessarily invertible even when the function defining its vector field is not invertible.

“This adds a nice property of allowing arbitrary couplings between measures.”

Claim: The speaker says continuous normalizing flows allow arbitrary couplings between measures.

“The base measure could be a normal distribution, a student T distribution or even another normalizing flow. Anything where you can compute the likelihood of a sample works.”

Claim: The speaker says a continuous normalizing flow can use a normal distribution, Student t distribution, or another normalizing flow as its base, provided sample likelihoods are computable.

“this also comes at a nonzero cost. For training, we need to compute the full O_D_E_ from the target sample back to the base measure every single time, since that is needed to compute the likelihood of the sample for maximum likelihood training.”

Claim: The speaker says continuous normalizing-flow training incurs the cost of solving the full ODE from each target sample to the base measure to compute likelihoods for maximum-likelihood training.

“I think JEPA is still a little bit further away.”

Claim: The speaker thinks JEPA is still somewhat further away, without specifying a particular milestone or timeframe.

“I think JEPA is still a little bit further away.”

Claim: JEPA is still somewhat further away, though the speaker does not specify a target or timeframe.

“But what I would say is the difference between something like an energy-based model and something like a diffusion model is that so both of them are fundamentally generative models.”

Claim: The speaker says energy-based models and diffusion models are both fundamentally generative models.

“But what I would say is the difference between something like an energy-based model and something like a diffusion model is that so both of them are fundamentally generative models.”

Claim: Energy-based models and diffusion models are both fundamentally generative models.

“I don't really get this idea of like energy-based models not being generative. Like I've heard this a bunch of times, but it's just not true.”

Claim: Energy-based models are not inherently non-generative; the speaker rejects the claim that they are not generative.

“I don't really get this idea of like energy-based models not being generative. Like I've heard this a bunch of times, but it's just not true.”

Claim: The speaker says it is not true that energy-based models are non-generative.

“The interesting thing is you can maybe use them in settings where you don't need to be generative, but fundamentally you are still generative. Like you still can generate samples. It's just very expensive to know. Consider so nobody does it.”

Claim: The speaker says energy-based models can be applied where generation is unnecessary, but they can still generate samples; sample generation is so expensive that people do not do it.

“The interesting thing is you can maybe use them in settings where you don't need to be generative, but fundamentally you are still generative. Like you still can generate samples. It's just very expensive to know.”

Claim: Energy-based models can generate samples, but sample generation with them is very expensive.

“Fundamentally, you can still do exactly the same thing, like you take your energy's model, you throw them into an M_C_M_C_ sampler, and technically you would get samples out.”

Claim: An energy-based model can technically produce samples when used with an MCMC sampler.

“of course many of these energy-based models are not actually good enough. to generate nice samples.”

Claim: Many energy-based models are not good enough to generate high-quality samples.

“if you had let's say, let's say you have the perfect um energy based model, like you uh like th someone comes up from the heavens says well this is your this is the right parameters, you plug them in and now you have kind of the the accurate data generation process. You can use them to generate data. Right, in fact this is not even that special of like uh a property. Fundamentally, as soon as you have something which is kind of let's say proportional to your density, you can throw this into M_C_M_C_ samples.”

Claim: If an energy-based model had correct parameters and represented the accurate data-generation process, it could generate data.

“if you had let's say, let's say you have the perfect um energy based model, like you uh like th someone comes up from the heavens says well this is your this is the right parameters, you plug them in and now you have kind of the the accurate data generation process. You can use them to generate data. Right, in fact this is not even that special of like uh a property. Fundamentally, as soon as you have something which is kind of let's say proportional to your density, you can throw this into M_C_M_C_ samples.”

Claim: A model proportional to the data density can be used with MCMC sampling to generate samples.

“a sample, and you can generate samples. So the distinction between can having a generative model and having a have a non-generative model, I think is really really small, because the the question is not whether they can do it, it's whether it's economical to do it.”

Claim: The distinction between generative and non-generative models is small in this context; the key question is whether generating samples is economical.

“And then I would agree with what I call it's not economical to do energy based models to generate data.”

Claim: Using energy-based models to generate data is not economical.

“But I would imagine this is a disadvantage or an advantage. The nice thing about energy based models is you can do certain things cheaper.”

Claim: Energy-based models can perform certain tasks more cheaply than alternatives, although the speaker does not specify which alternatives in this statement.

“Namely kind of the the very big one is you can do kind of comparisons between data points. Right because like there is let's say there's effectively an unknown scale in set energy-based models. But what you can do is you can effectively still go and say well this point is that's a three times as likely as this other point by just effectively computing the ratio. And if you just need this so if you have a problem where effectively you just need pairwise comparisons, then energy based models can be quite good. Uh you effectively you don't need to generate data points to uh produce something like let's say not a likelihood but something like an an a score or like score is also maybe a bad word because it's also inference coded. But”

Claim: Energy-based models can be useful for tasks requiring only pairwise comparisons, since they can provide relative scores without generating data points.

“Namely kind of the the very big one is you can do kind of comparisons between data points. Right because like there is let's say there's effectively an unknown scale in set energy-based models. But what you can do is you can effectively still go and say well this point is that's a three times as likely as this other point by just effectively computing the ratio. And if you just need this so if you have a problem where effectively you just need pairwise comparisons,”

Claim: Energy-based models can compare the relative likelihood of data points by computing a ratio, even if the model has an unknown overall scale.

“And you still get kind of the the relative difference between two two points. Of course the downside and what makes it hard f for this to be generative is even if the ratio between two points might be really large, it still might be really really bad, because you don't have an absolute score. It's like saying well this movie is a five. Well, five out of five, five out of ten, five out of a hundred. Like it doesn't tell you enough to actually figure out whether it's a good sample or a bad sample. And actually”

Claim: Even a large relative ratio between two data points may not indicate a good sample, because an energy-based model does not provide an absolute score in this setting.

“The true figure is out in absolute sense, this is what the expensive part. The uh so energy based modelling is really nice if you never need to sample anything. So this is what makes it really cool for let's say pre-training. Alright? If you do pre-training, uh then doing something like an energy based model that might make sense. Of course there are many other things that also make sense and it's I think it's up to debate of whether it's”

Claim: Energy-based modeling may be suitable for pre-training when the task never requires sampling.

“The true figure is out in absolute sense, this is what the expensive part. The uh so energy based modelling is really nice if you never need to sample anything. So this is what makes it really cool for let's say pre-training. Alright? If you do pre-training, uh then doing something like an energy based model that might make sense. Of course there are many other things that also make sense and it's I think it's up to debate of whether it's”

Claim: Determining the absolute figure or score is the expensive part of energy-based modeling.

“The true figure is out in absolute sense, this is what the expensive part. The uh so energy based modelling is really nice if you never need to sample anything. So this is what makes it really cool for let's say pre-training. Alright? If you do pre-training, uh then doing something like an energy based model that might make sense. Of course there are many other things that also make sense and it's I think it's up to debate of whether it's actually a good idea to do energy-based modelling over anything else, but”

Claim: Whether energy-based modeling is preferable to other approaches remains debatable.

“Yeah, I think there's a completely different discussion between kind of doing between kind of the question of whether these energy-based models are good at doing inference or bad at doing inference because essentially you're just saying well I train an energy-based model but I don't end up using it for like real inference.”

Claim: Whether energy-based models perform well at inference is a separate question from training them and then using them for something other than real inference.

“I use it to effectively measure differences in intensity.”

Claim: The speaker describes using an energy-based model to measure differences in intensity rather than to perform real inference.

“this is also one of my big problems with like the Jepa style things with Jepa like joint embedding predictive architecture that could describe literally anything”

Claim: Alexander Mattick says JEPA-style approaches are broad enough to describe literally anything.

“Right uh it basically just says what it's not contrastive. But let's say um like I recently in in one of the like paper discussions at Yannick Kirch's discord I I made the like distinction I found generally that like dyno embeddings are much better than like the um like the japa style embeddings. At least for the stuff that I tried them on. This might be very different for your application. But I think this kind of like a”

Claim: Alexander Mattick generally found DINO embeddings much better than JEPA-style embeddings on the applications he tried, while allowing that the comparison might differ for the listener's application.

“a a false comparison. Because if dyno came out today, they would call it a Japa architecture. Because it's it's such an open domain. And the same thing is true with um”

Claim: Alexander Mattick considers comparing DINO with JEPA-style embeddings a false comparison, arguing that DINO would be classified as a JEPA architecture if introduced today because the category is so open-ended.

“let's say variation inference or like energy-based models you have you have a lot of flexibility it's just kind of what you so energy-based models is just a name for what you end up predicting and what you end up predicting is kind of this effectively unnormalized intensity map right effectively it's like a heat map and anything that that is this can be framed as an energy-based model which is also why like they”

Claim: Alexander Mattick describes an energy-based model as a model that predicts an effectively unnormalized intensity map, or heat map.

“they show up everywhere. Because if you want to, you can phrase it as an energy based model. And that makes it oftentimes a less useful description.”

Claim: Alexander Mattick says energy-based models appear widely because many models can be framed that way, making the label less useful as a description.

“What I would kind of what I kind of place my difference is if it's already normalised, so something like uh I predict, like let's say in like m uh in reinforcement learning, I predict let's say the mean and standard deviation given the input data. Then I say well this is not really uh again energy-based model. Like technically it is, practically it you wouldn't think of it like that. Then on the other extreme you have something like a score.”

Claim: Alexander Mattick says a model that predicts a mean and standard deviation from input data is technically an energy-based model, although it is not usually treated as one in practice.

“uh based models which effectively model the gradient of energy and do tricks to basically also remove all the normalisation. And that then gives you something like these these Langevin diffusion start things that gives you uh a diffusion modelling and well it doesn't give you a flow-matching, but it gives you something like flow-matching. But, yeah, I would agree that the difference is really really wishy-washy.”

Claim: Alexander Mattick says score-based models can lead to Langevin diffusion and diffusion modeling, and to something resembling flow matching but not flow matching itself.

“uh based models which effectively model the gradient of energy and do tricks to basically also remove all the normalisation. And that then gives you something like these these Langevin diffusion start things that gives you uh a diffusion modelling and well it doesn't give you a flow-matching, but it gives you something like flow-matching. But, yeah, I would agree that the difference is really really wishy-washy.”

Claim: Alexander Mattick considers the distinction between energy-based, score-based, diffusion, and related model categories very unclear or imprecise.

“uh based models which effectively model the gradient of energy and do tricks to basically also remove all the normalisation. And that then gives you something like these these Langevin diffusion start things that gives you uh a diffusion modelling and well it doesn't give you a flow-matching, but it gives you something like flow-matching. But, yeah, I would agree that the difference is really really wishy-washy.”

Claim: Alexander Mattick describes score-based models as modeling the gradient of energy and using techniques that remove normalization.

“One of the so one of the first ideas was okay We can phrase this as kind of making similar things similar and dissimilar things dissimilar, right? And this gives you ultimately a contrastive style model by basically what seems to our effective it does is if you look at the loss effectively It's like a cross entropy, but over the batch”

Claim: Contrastive learning can be framed as pulling similar examples together and pushing dissimilar examples apart, with a loss that functions like cross-entropy over a batch.

“And what you then do is you have the like positive part which is like the the different augmentations which get pulled together, and the dis uh like the the dissimilar part they get pushed apart. The downside with this is it's ultimately a very very moving target, like a batch to batch moving target. Because”

Claim: Contrastive learning methods have a target that shifts from batch to batch.

“And what you then do is you have the like positive part which is like the the different augmentations which get pulled together, and the dis uh like the the dissimilar part they get pushed apart. The downside with this is it's ultimately a very very moving target, like a batch to batch moving target. Because”

Claim: Contrastive learning pulls positive augmentations together and pushes dissimilar examples apart.

“And kind of contrastive just means when it's somehow the right things have to be together and somehow everything has to be apart. This means there are many, many, many solutions which actually satisfy this condition, but are not actually useful.”

Claim: Many solutions that satisfy contrastive objectives by grouping the right examples and separating others are not useful.

“So in practice you want something which is A, stronger, and B, does not have this instability that you fundamentally have with these SimCLR style methods.”

Claim: The speaker argues that practical learning methods should provide a stronger objective and avoid the instability of SimCLR-style methods.

“And if you look at SimCLR, they only start making sense at”

Claim: SimCLR methods become sensible only when used with very large patch sizes.

“huge patch sizes and then there is like follow-up work like the the moco uh momentum contrastive models which essentially say well we can simulate huge patch sizes by effectively taking let's say a buffer over time”

Claim: Momentum contrastive models such as MoCo approximate very large patch sizes by maintaining a buffer over time.

“Um this of course adds a little bit of noise, but tends to work pretty well. But fundamentally I agree with Lecun that we need to go back away from these contrasted models, because contrast contrasitivity alone is not strong enough of a statement to uh to actually learn necessarily useful models. So what Lecun all says is well, we should actually go effective, we should revisit old ideas, which is try to map things in”

Claim: The speaker agrees with LeCun that contrastive models should be left behind because contrastivity alone does not provide a strong enough objective to ensure useful learned models.

“Um this of course adds a little bit of noise, but tends to work pretty well. But fundamentally I agree with Lecun that we need to go back away from these contrasted models, because contrast contrasitivity alone is not strong enough of a statement to uh to actually learn necessarily useful models. So what Lecun all says is well, we should actually go effective, we should revisit old ideas, which is try to map things in”

Claim: The buffer-over-time approach used by momentum contrastive models adds some noise but tends to work well.

“into like a a compact latent. At the first idea of this was like a variational auto-encoder. A variational auto-encoder says, well, you have to be able to map the data to a normal distribution or a beta distribution, whatever you you set in the middle. And then you have to be able to map this back. And effectively this compresses everything down into this uh this is tiny gate in the middle.”

Claim: A variational autoencoder encodes data into a chosen distribution, such as a normal or beta distribution, and decodes it back, compressing it into a small latent representation.

“And you effectively the the top part is a generative model, the bottom part is effectively s like an embedding model. A problem with like these variation autoencoders and it's something that's had been also been seen very very early on is”

Claim: A variational autoencoder has a generative model in its top part and an embedding model in its bottom part.

“that you effectively need to pay for things that don't end up making a difference,”

Claim: Variational autoencoders can incur costs for factors that ultimately make no difference to the result.

“The classic example, I don't know who did it, has the example back, it's an actual dollar versus like a painting of a dollar. Like it represents the same idea, but it's very, very different on a pixel-to-pixel basis.”

Claim: A real dollar and a painting of a dollar can represent the same concept while differing substantially pixel by pixel.

“And this then gives you something like Dyno or something like Vicrek or something like the JAPA style architectures.”

Claim: The speaker associates this approach to learning representations with methods such as DINO, VICReg, and JEPA-style architectures.

“Like specifically, like say one of the newer ones which I really like is like this ledger star thing which effectively does like uses the statistical tests to actually figure out, okay, am I different from a normal distribution or am I the same as a normal distribution?”

Claim: The speaker likes a newer method that uses statistical tests to assess whether a representation differs from or matches a normal distribution.

“And based on that, they can differentiate through that test and move you closer or farther away from the actually now fixed goal.”

Claim: The method uses differentiation through a statistical test to move representations toward or away from a fixed target.

“of the density. So you replace the contrastive push things apart or pull things together with be somehow on this fixed target.”

Claim: The approach replaces contrastive forces that push examples apart or pull them together with alignment to a fixed target density.

“Of course you still have some flexibility, right, because if you like let's say if every point starts pushing into the mode of the d of the density, at some point you would have a kind of pushing a p uh pushing away effect again. So you have a similar thing as a contrastive model, but now it is”

Claim: The fixed-density approach retains flexibility, and if all points move toward the density's mode, a repulsive effect can eventually emerge.

“uh contrastive, compared to a reference density, and not compared to itself, and it makes it a lot more stable.”

Claim: The approach compares representations with a reference density rather than with one another, making it more stable.

“In my opinion, I think that energy-based models are not really useful, or at least not useful anymore.”

Claim: The speaker believes energy-based models are generally no longer useful.

“of course, you can frame your model as being energy-based. I will not, if I'm a reviewer, I will not ding you for it.”

Claim: The speaker says they would not penalize a paper as a reviewer merely for framing its model as energy-based.

“Yeah, ultimately I don't think it's a useful way of thinking about the vast majority of problems. Like there are exceptions to that. Like let's say you are actually in like a physics setting. But then maybe it makes a lot of sense to actually phrase this as an energy-based model.”

Claim: The speaker considers energy-based modeling too general to be useful for most problems, while allowing that it may be appropriate in fields such as physics.

“I would argue that it's not uh like it's not a useful way like it's it's so general of a statement, it's not a useful way of thinking about the problem. Uh for for things like uh contrastive or no contrastive for unsupervised like these uh self supervised learning methods I think there is generally speaking a better way of thinking about the problem than energy based. And I all ultimately I think this is where these methods lie. If you want to do generative”

Claim: The speaker believes self-supervised learning is generally better conceptualized in ways other than energy-based modeling.

“If a diffusion model or a flow match is better,”

Claim: The speaker considers diffusion models or flow matching better approaches for generative modeling than energy-based framing.

“And if you want to do unsupervised learning, I think you can phrase it as an energy-based model, but I don't think it's a useful framing.”

Claim: Although unsupervised learning can be formulated as an energy-based model, the speaker considers that framing unhelpful.

“And the problem that they quickly run into is that most domains are not actually that discreet. Like you can s uh like you can make them discreet by basically hitting with a hammer long enough, but practically they're not discreet. The the problem is that if you are not discreet, things like this autoregressive decomposition become kind of dicey, because the”

Claim: Most domains are inherently continuous rather than discrete, though they can be discretized through sufficiently extensive processing.

“Like for for language models, the nice part is we can effectively can go step by step, and we can predict kind of from left to right. If you have a continuous density like an image, you can still do the auto-regressive, but it's not longer a natural way of doing things. Like it's not a natural way of decomposing a density. But ultimately that's what all of these models do. Be that diffusion, be that energy-based, be that um auto-regressive, all of them try to have a huge density in an entire sentence and decomposes into smaller parts which can actually handle.”

Claim: Left-to-right autoregressive prediction is a natural decomposition for language, but not for continuous densities such as images.

“Like for for language models, the nice part is we can effectively can go step by step, and we can predict kind of from left to right. If you have a continuous density like an image, you can still do the auto-regressive, but it's not longer a natural way of doing things. Like it's not a natural way of decomposing a density. But ultimately that's what all of these models do. Be that diffusion, be that energy-based, be that um auto-regressive, all of them try to have a huge density in an entire sentence and decomposes into smaller parts which can actually handle.”

Claim: Diffusion, energy-based, and autoregressive models decompose a large joint density into smaller manageable parts.

“And the uh the idea that you want to do or like most people settle on for continuous uh product processes is instead of separating kind of uh from left right, let's say token by token by token, it's they say let's we we separate from from bottom to top, like we start off with all tokens, but kind of corrupted and kind of go back up.”

Claim: For continuous processes, a common approach is to begin with a fully corrupted representation and progressively restore it, rather than decomposing left to right token by token.

“And that immediately allows you to do things like image generation, because I I explained just now with tokens to to make the comparison uh sensible, but ultimately you start up with something and you end up with something. In fact, discrete in this case is the more difficult part, because of discrete diffusion you you end up with some kind of dicey things with kind of uh I I guess language models are probably fine, but you get into dicey things with basically if you actually sample from the density at every single point. It's there possible, but um you have to be a little bit more careful. But the nice part is that once you have kind of this this general recipe of decomposing your uh your your structure, then you can apply to anything. And this is what makes diffusion models or flow matchers which is like if you read diffusion model and you look into the into it, oftentimes it's flow matching because it tends to be a little bit more stable.”

Claim: Discrete diffusion can be problematic when sampling from the density at every point; the speaker thinks language models are probably fine, but says the method requires more care.

“And that immediately allows you to do things like image generation, because I I explained just now with tokens to to make the comparison uh sensible, but ultimately you start up with something and you end up with something. In fact, discrete in this case is the more difficult part, because of discrete diffusion you you end up with some kind of dicey things with kind of uh I I guess language models are probably fine, but you get into dicey things with basically if you actually sample from the density at every single point.”

Claim: The bottom-up process of progressively restoring a corrupted representation enables image generation.

“It's there possible, but um you have to be a little bit more careful. But the nice part is that once you have kind of this this general recipe of decomposing your uh your your structure, then you can apply to anything. And this is what makes diffusion models or flow matchers which is like if you read diffusion model and you look into the into it, oftentimes it's flow matching because it tends to be a little bit more stable.”

Claim: A general method for decomposing a structure can be applied across domains.

“It's there possible, but um you have to be a little bit more careful. But the nice part is that once you have kind of this this general recipe of decomposing your uh your your structure, then you can apply to anything. And this is what makes diffusion models or flow matchers which is like if you read diffusion model and you look into the into it, oftentimes it's flow matching because it tends to be a little bit more stable.”

Claim: Diffusion models often use flow matching, which the speaker says tends to be somewhat more stable.

“And but all of these methods are just ways of decomposing a density in a really, really general way.”

Claim: Diffusion and related methods provide general ways to decompose a probability density.

“And what you do in diffusion is you give yourself these infinite numbers of latents and say, well, given the infinite number, it has to be decomposable in some point, in some way.”

Claim: Diffusion uses an effectively infinite number of latent variables and relies on the assumption that the density can be decomposed.

“And in the or in the case of flow matching, we literally give kind of the reference of, OK, this is what you need to do.”

Claim: Flow matching provides an explicit reference for the model's desired behavior.

“And in autoregressive structures you do exactly the same thing, but instead of giving yourself the latent codes, you say, well, I have a very natural decomposition of my text already,”

Claim: Autoregressive models use text's natural token decomposition in place of latent codes as the basis for decomposition.

“just use that. And but ultimately you can have all both of these solve the same type of problem. It's just whether you do the decomposition in the observable space, whether you do the decomposition in the latent space and for things like hi for high dimensional things”

Claim: Autoregressive and latent-space methods can address the same kind of problem, differing in whether they decompose in observable space or latent space.

“just use that. And but ultimately you can have all both of these solve the same type of problem. It's just whether you do the decomposition in the observable space, whether you do the decomposition in the latent space and for things like hi for high dimensional things which then sounds of less information per effective per variable it it tends to make sense or at least empirically tends to make sense to go into diffusion instead”

Claim: For high-dimensional data with less information per effective variable, diffusion tends to be a sensible approach empirically.

“And can we just talk more about inference in general? So I think this is something that people don't think enough about because we don't do. We don't really do things that resemble inference that much. With language models,”

Claim: The speaker believes people give too little attention to inference because human activity does not often resemble inference.

“we just do greedy sampling. So we've got this density and we just take the, you know, we just take a token,”

Claim: The speaker says language models generate outputs by taking tokens from a density through greedy sampling.

“In something like AlphaGo, for example, we had a structured inference process. So we did this, you know, Monte Carlo tree search and we were kind of searching around and actually algorithmically updating nodes that we had visited and feeding that back and retraining the model.”

Claim: The speaker describes AlphaGo's inference process as structured: it used Monte Carlo tree search to update visited nodes algorithmically, feed the results back, and retrain the model.

“In something like AlphaGo, for example, we had a structured inference process. So we did this, you know, Monte Carlo tree search and we were kind of searching around and actually algorithmically updating nodes that we had visited and feeding that back and retraining the model.”

Claim: The interviewer describes AlphaGo's structured inference as using Monte Carlo tree search to explore and update visited nodes, feed those updates back, and retrain the model.

“With these vision models and when we're using flow models and stuff like that, we are actually doing. actually doing like quite sophisticated inferencing processes and I interviewed Clement Bonet he had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for Ark and we've got this whole you know and test time computation test time adaptation and so on so this whole thing about doing work at inference time is very exciting to me and I just don't think that that many people are talking about it.”

Claim: The speaker says vision models and flow models perform sophisticated inference processes.

“With these vision models and when we're using flow models and stuff like that, we are actually doing. actually doing like quite sophisticated inferencing processes and I interviewed Clement Bonet he had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for Ark and we've got this whole you know and test time computation test time adaptation and so on so this whole thing about doing work at inference time is very exciting to me and I just don't think that that many people are talking about it.”

Claim: The interviewer says vision and flow models can use sophisticated inference, including latent-space or decoder search for programs that solve ARC tasks, along with test-time computation and adaptation.

“actually doing like quite sophisticated inferencing processes and I interviewed Clement Bonet he had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for Ark and we've got this whole you know and test time computation test time adaptation and so on so this whole thing about doing work at inference time is very exciting to me and I just don't think that that many people are talking about it.”

Claim: The speaker says Clement Bonet's paper on searching latent spaces involved searching a decoder to find the best programs to generate for ARC.

“actually doing like quite sophisticated inferencing processes and I interviewed Clement Bonet he had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for Ark and we've got this whole you know and test time computation test time adaptation and so on so this whole thing about doing work at inference time is very exciting to me and I just don't think that that many people are talking about it.”

Claim: The speaker is very excited about computation and adaptation at inference time and believes relatively few people discuss this area.

“actually doing like quite sophisticated inferencing processes and I interviewed Clement Bonet he had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for Ark and we've got this whole you know and test time computation test time adaptation and so on so this whole thing about doing work at inference time is very exciting to me and I just don't think that that many people are talking about it.”

Claim: The interviewer considers inference-time computation very exciting and believes relatively few people are discussing it.

“We give people not enough credit when we talk about kind of doing infor so first of inference is a really really overloaded term, right? If you go into logic, like they say we do inference, like logical inference. And if you go into like Bayesian they they all say they also do inference. Now I say, well we are actually also doing inference. And so I I think we have to be a little bit careful not to to use the term kind of too liberally because and like I have to be careful myself.”

Claim: The speaker says “inference” is an overloaded term used in both logic and Bayesian methods, among other contexts.

“We give people not enough credit when we talk about kind of doing infor so first of inference is a really really overloaded term, right? If you go into logic, like they say we do inference, like logical inference. And if you go into like Bayesian they they all say they also do inference. Now I say, well we are actually also doing inference. And so I I think we have to be a little bit careful not to to use the term kind of too liberally because and like I have to be careful myself. because at some point use all meaning. At some point it becomes the energy based model, which you can apply to anything. But ultimately um I think in many cases it is it is very very instructive to think about these types of models.”

Claim: The speaker argues that “inference” should not be used too liberally, since the term risks becoming so broad that it means everything.

“because at some point use all meaning. At some point it becomes the energy based model, which you can apply to anything. But ultimately um I think in many cases it is it is very very instructive to think about these types of models.”

Claim: The speaker believes that thinking about these types of models is very instructive in many cases.

“As just composing j just conditional dependencies. I condition the dependencies and marginalisation. Semper like alpha go is actually a little bit more difficult, because what they do is effectively they do this like non parametric, effectively they they come, arguably they come more from like less the Bayesian side and more the frequentist side, which is an entirely different kind of worms that don't want to open up right now. But um at the end of the day”

Claim: The speaker characterizes many mainstream methods as combinations of conditional dependencies and marginalization.

“As just composing j just conditional dependencies. I condition the dependencies and marginalisation. Semper like alpha go is actually a little bit more difficult, because what they do is effectively they do this like non parametric, effectively they they come, arguably they come more from like less the Bayesian side and more the frequentist side, which is an entirely different kind of worms that don't want to open up right now. But um at the end of the day”

Claim: The speaker says AlphaGo's approach is harder to characterize, describing it as effectively nonparametric and arguably more frequentist than Bayesian.

“All of these all of these methods really we or at least all these mainstream methods really benefit from this idea of we have a big problem we need to decompose this and every step is one step in a bigger inference process. Um let's say uh let's give an example from one of my papers. I did uh a paper about doing effective mix integer programming, learning heuristic for mix integer programming.”

Claim: The speaker says mainstream methods benefit from decomposing a large problem into steps that each contribute to a larger inference process.

“And specifically like in mixed energy programming to not go too deep they have a process called branch and bound which essentially you you start off with like a uh a approximation and then based on a solution you you pick two different solutions just like in like a tree search or it is a tree search and you go uh this node you go further down but close it something like alpha go uh like alpha beta pruning you can prune away lots and lots of notes if you know more information by basically say with this node is worse than the solution that I already found, uh so I can remove that. So it becomes really important if you have like let's say a huge tree with let's say thousands of open notes at the bottom to choose which one to actually end up selecting.”

Claim: The speaker describes branch and bound in mixed-integer programming as starting from an approximation, branching into alternative solutions in a tree search, and pruning nodes when information shows they are worse than an existing solution.

“with this node is worse than the solution that I already found, uh so I can remove that. So it becomes really important if you have like let's say a huge tree with let's say thousands of open notes at the bottom to choose which one to actually end up selecting.”

Claim: The speaker says selecting which node to explore is important when a branch-and-bound tree has many open nodes.

“And what I did is I from this as a inference problem. But basically I say well if we go down from the root to uh to one of the nodes, you effectively do a conditional probability. Like you you have a probability to choose either left or right. Then you have another to choose in left or right. And that gives you a much more general probel uh much more general model which kind of grows with your problem.”

Claim: The speaker says representing the process with conditional probabilities yields a more general model that scales with the problem.

“And what I did is I from this as a inference problem. But basically I say well if we go down from the root to uh to one of the nodes, you effectively do a conditional probability. Like you you have a probability to choose either left or right. Then you have another to choose in left or right. And that gives you a much more general probel uh much more general model which kind of grows with your problem.”

Claim: The speaker modeled branch and bound as an inference problem where each step from the root to a node is represented by conditional probabilities for choosing left or right.

“Right, so as your problem grows, your inference process uh grows and kind of ingests more and more information. And the same thing is true with something that uh like let's say a large language model ultimately, which kind of does research, uh kind of looks up more data, ingests it and now the next step is now conditioned on that information. Now”

Claim: The speaker says a research-performing large language model can retrieve and ingest additional data, then condition its next step on the newly acquired information.

“Right, so as your problem grows, your inference process uh grows and kind of ingests more and more information. And the same thing is true with something that uh like let's say a large language model ultimately, which kind of does research, uh kind of looks up more data, ingests it and now the next step is now conditioned on that information. Now”

Claim: The speaker says an inference process grows with the problem and incorporates increasing amounts of information.

“How useful is this for your, like the viewer's daily work? It really depends on what you end up doing.”

Claim: The speaker says the usefulness of this perspective for daily work depends on the person's work.

“If your work is, let's say, I scale up a large language model, then you probably don't need to think about this.”

Claim: The speaker says someone whose work is scaling up a large language model probably does not need to think about inference in this way.

“If your research is like figuring out really, really efficient architectures, maybe, but I would also say probably doesn't matter.”

Claim: The speaker says thinking about inference this way might matter for research on highly efficient architectures, but probably does not.

“Yeah, or you do something like R_L_ where ultimately kind of everything is like you update some density, if unless you do like Q learning, but if you update like densities, then you you should be thinking about this as an inference process, because”

Claim: The speaker says that when a method updates densities rather than using Q-learning, it should be understood as an inference process.

“uh con especially Bayesian inference gives you a lot more structure a lot more information than what we give it credit for. Like some product just decomposing a problem into conditional probabilities gives you it gets you like ninety nine percent of the way there, but it can already help guide the actual problem. And I think that something like all these agentic things could really benefit from actually thinking about this as like a Bayesian problem of kind of going out gathering data.”

Claim: The speaker believes agentic systems could benefit from a Bayesian approach in which they gather data.

“uh con especially Bayesian inference gives you a lot more structure a lot more information than what we give it credit for. Like some product just decomposing a problem into conditional probabilities gives you it gets you like ninety nine percent of the way there, but it can already help guide the actual problem. And I think that something like all these agentic things could really benefit from actually thinking about this as like a Bayesian problem of kind of going out gathering data.”

Claim: The speaker says that decomposing a problem into conditional probabilities can get one about 99 percent of the way toward a solution and can help guide the problem-solving process.

“uh con especially Bayesian inference gives you a lot more structure a lot more information than what we give it credit for. Like some product just decomposing a problem into conditional probabilities gives you it gets you like ninety nine percent of the way there, but it can already help guide the actual problem. And I think that something like all these agentic things could really benefit from actually thinking about this as like a Bayesian problem of kind of going out gathering data.”

Claim: The speaker believes Bayesian inference offers more structure and information than people generally give it credit for.

“and updating your conditional belief based on the information that you just ingested.”

Claim: The speaker describes agentic systems updating conditional beliefs in response to information they have just ingested.

“So, the problem that you run into in reinforcement learning is that you don't want to be greedy with like the immediate actions. Right, because what you do immediately could be really, really detrimental. Like let's say you you manage a warehouse and the the greedy thing to do to maximise profit right now is just sell everything in your uh in in your supply closet and never buy anything. Because then the profit is maximal. That's not necessarily long-term the right thing. So what we would need to do is we don't need to look at the current reward, but we do need to look at kind of the future as well.”

Claim: Q-learning addresses the problem that maximizing immediate reward can lead to actions that are detrimental in the long term.

“But that gives you a problem. And that problem is well, I now need to model the future. So the way that Q learning does it is it says, well let's decompose it as kind of a head, which is the current reward, plus the tail, kind of the rest.”

Claim: Q-learning decomposes value into the current reward and the remaining future value.

“Let's say we have such a decomposition. Then we could effectively act greedily with respect to this kind of to this combined metric. Right? Because if we uh have a c a Q_ function which tells us what if we go down this path, the the future will behave like this, we go down this path, it will behave like this, and this path will behave like this. Now we can uh be greedy again with respect to the Q_ function. So effectively the Q_ function fundamentally is a closure over the like Over the path that the policy can take.”

Claim: A Q-function that represents future behavior along possible paths allows an agent to choose greedily according to the combined value metric.

“The way you train this is effectively by just enforcing exactly this. So you say the current reward plus the next state's uh Q function should be equal to the uh the Q function starting from the original state. So you effectively say R plus”

Claim: Q-learning training enforces a relationship in which the current reward plus the next state's Q-value equals the Q-value from the original state.

“Cue uh a cue from a next state of for a specific action and then uh you say this has to be the same as doing a cue function from the original state already. This then gives you two options, right, the first option is we could choose kind of for the next action to choose here, right, because the cue function has both a state and an action at at this note, uh at this future note. You could choose well, we pick the action that was actually taken in the real world. This gives you SARSA.”

Claim: Using the action actually taken in the next state for the Q-learning update gives SARSA.

“The other option is well I could go and say instead of just picking the next action which actually happened is I pick the optimal action. Because let's say we have a discrete number of actions, just look at every single one and say well I pick the one which is the best. So effectively you hallucinate a world in which the you actually chose the correct path.”

Claim: An alternative Q-learning update evaluates all possible next actions and uses the optimal action, as if the optimal path had been taken.

“It has the very nice property of now you uh you effectively are no longer coupled to the specific policy. And if you choose the correct uh if you choose the actions the policy chose, you're now effectively updated with respect to the policy. If you choose with the uh if you choose the optimal state, effectively you're uh you're updating with respect to the optimal policy already, have the Q_ star.”

Claim: In the described update, using the policy's chosen action updates values with respect to that policy; using the optimal action updates with respect to the optimal policy and yields Q-star.

“The problem with this is now that you no longer actually have some form of effectively of a counterfactual inference problem where you need to deal with what actually happened versus what happened in our like let's say golden state where we walked along the kind of optimal path.”

Claim: Using the optimal-action approach creates a counterfactual inference problem comparing the actual trajectory with the hypothetical optimal trajectory.

“Again I think the big one is if you just need ratios, because then it can be a lot cheaper because you don't end up generating, let's say, the entire trajectory. Uh it can also be helpful if you have something energy like already. Like le uh again something like physics, where effectively you based on data you try to learn the energy function, something like this. Other than that”

Claim: Energy-based models can be much cheaper for tasks requiring only ratios because they do not require generating the entire trajectory.

“Again I think the big one is if you just need ratios, because then it can be a lot cheaper because you don't end up generating, let's say, the entire trajectory. Uh it can also be helpful if you have something energy like already. Like le uh again something like physics, where effectively you based on data you try to learn the energy function, something like this. Other than that I think it becomes really really difficult to justify.”

Claim: The speaker thinks energy-based models are difficult to justify outside cases where only ratios are needed or an energy-like structure is already present.

“Again I think the big one is if you just need ratios, because then it can be a lot cheaper because you don't end up generating, let's say, the entire trajectory. Uh it can also be helpful if you have something energy like already. Like le uh again something like physics, where effectively you based on data you try to learn the energy function, something like this. Other than that”

Claim: Energy-based models may be useful when a domain already has an energy-like structure, such as physics, where an energy function can be learned from data.

“If you don't use a if you use them for instance as a pre training task, I don't see a reason why not. I I don't see a reason why you should but I don't see a reason why it wouldn't work. The the question really is is if we end up looking like let's say like uh like it's a we we jump forward five years from now and LaCour's startup is doing whatever he he did, like they are doing currently research. Whether what they end up doing like they might still call it energy based models, but whether it's actually an energy based model”

Claim: The speaker sees no clear reason energy-based models would fail as a pretraining task, but also sees no clear reason to use them for pretraining.

“If you don't use a if you use them for instance as a pre training task, I don't see a reason why not. I I don't see a reason why you should but I don't see a reason why it wouldn't work. The the question really is is if we end up looking like let's say like uh like it's a we we jump forward five years from now and LaCour's startup is doing whatever he he did, like they are doing currently research. Whether what they end up doing like they might still call it energy based models, but whether it's actually an energy based model As you and I would be thinking about this right now, I highly doubt that.”

Claim: The speaker highly doubts that a startup's work five years from now will qualify as an energy-based model under the current understanding, even if it is labeled one.

“Because while it's possible, I don't think it's the best way of doing this.”

Claim: Although using energy-based models for the discussed purpose is possible, the speaker does not think it is the best approach.

“And the bad thing, or the good thing and the bad thing about deep learning is that if you do something wrong, it generally doesn't just not work. It just is worse performance.”

Claim: In deep learning, methodological mistakes generally degrade performance rather than causing the method to fail completely.

“And it makes it really, really difficult to research, which is why there are thousands of papers.”

Claim: Because mistakes in deep learning can produce degraded rather than failed results, the speaker says deep-learning research is difficult and has thousands of papers.

“would essentially claim more or less contradictory things because depending on subtle details things can just work or not work and they can be really really so even if you do something completely wrong like I have recently I have a paper which I tried to get done for ICLR where I had like an implementation bug it still kind of worked”

Claim: The speaker says subtle implementation details can make methods work or fail, leading papers to make contradictory claims; in a recent paper the speaker tried to submit to ICLR, an implementation bug did not stop the method from working to some extent.

“And there are plenty of other things where still like it doesn't on paper doesn't work but in practice it just works because especially reinforcement learning it just it hammers on it until it figures out something and I think that many many things right now are like this like they are not necessarily optimal but they're also too bad to actually bother fixing it.”

Claim: Some methods that appear not to work theoretically can work in practice, especially in reinforcement learning, where repeated optimization can eventually find a workable solution.

“And there are plenty of other things where still like it doesn't on paper doesn't work but in practice it just works because especially reinforcement learning it just it hammers on it until it figures out something and I think that many many things right now are like this like they are not necessarily optimal but they're also too bad to actually bother fixing it.”

Claim: The speaker thinks many current methods are suboptimal but not sufficiently poor to justify the effort of fixing them.

“Flow matching actually comes from uh so there are two ways of t think about flow matching. One is you can think about them as like the the continuation of continuous normalising flows. The idea of a continuous or like any normalising flow is you have a base density, so a normal distribution, and you frame your problem now as trying to figure out a transformation which goes from that base density to your target density.”

Claim: Flow matching can be understood as a continuation of continuous normalizing flows, which transform a base density such as a normal distribution into a target density.

“If you make sure that this density is invertible, Technically a little bit less is possible, but say it's invertible. Then uh we can actually uh both do the sampling, so you go from the base sensitivity, you go forward to the sample space uh so this is the sampling direction. And if you have a sample, you can go backward and figure out the the probability of generating something by looking at uh what it is on the normal. So this is the first idea which you can kind of branch off from. And the second is from optimal transport.”

Claim: An invertible transformation from a base density to a sample space supports forward sampling and allows a sample's probability to be calculated by mapping it back to the base density.

“Uh we can frame an optimal transport problem from one density or to another as a ca a kinetic problem, where effectively we follow uh this transport equation, so the uh the the density change over time is equal to the negative of the uh the current density times the velocity field and we we do the uh diverted around it.”

Claim: An optimal transport problem between two densities can be formulated as a kinetic problem governed by a transport equation relating the time change in density to the negative product of density and velocity field.

“The uh and this effectively is a like a continuous uh effectively it's like grain descent, but in continuous time. And instead of over having a finite number of parameters, you do it over uh like the the actual density. So it's a it's called a gradient flow. And these have really really nice properties. So even in like finite uh dimensions, and uh uh the these continuous normalising flows uh the these gradient flows have super nice properties. So”

Claim: Gradient flow is continuous-time gradient descent over a density rather than a finite set of parameters, and the speaker says gradient flows have favorable properties even in finite dimensions.

“Uh what you end up doing, we use the fact that effectively the velocity field and the the density change is coupled together by saying instead of learning the density, we learn the change in density, so this vector field. And this is the parameterisation.”

Claim: Flow matching learns a vector field representing the change in density rather than learning the density directly.

“the Fokker-Planck equation reduces back down to having a very specific um velocity field. And if you look at this velocity field, you end up doing so you end up realising two things. One is of course like there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or uh more so than diffusion. Because if you look at the velocity field, the velocity field actually depends on the density.”

Claim: The velocity field obtained from the Fokker–Planck equation depends on the density.

“the Fokker-Planck equation reduces back down to having a very specific um velocity field. And if you look at this velocity field, you end up doing so you end up realising two things. One is of course like there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or uh more so than diffusion. Because if you look at the velocity field, the velocity field actually depends on the density.”

Claim: Flow matching and diffusion methods are fundamentally connected.

“the Fokker-Planck equation reduces back down to having a very specific um velocity field. And if you look at this velocity field, you end up doing so you end up realising two things. One is of course like there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or uh more so than diffusion. Because if you look at the velocity field, the velocity field actually depends on the density.”

Claim: Diffusion models are harder to train than flow matching because their velocity field depends on the density.

“the Fokker-Planck equation reduces back down to having a very specific um velocity field. And if you look at this velocity field, you end up doing so you end up realising two things. One is of course like there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or uh more so than diffusion. Because if you look at the velocity field, the velocity field actually depends on the density.”

Claim: Diffusion models are comparatively difficult to train because their velocity field depends on the density.

“So, uh effectively what we now have is we have the like the the self coupling which originally in the dif in like a flow measure we can get rid of. We can completely get rid of ever needing to touch the density. While in flow measureing, uh while in diffusion you end up having to touch the density, of course there's this trick with like the the score transformation which means you kind of get around this. But ultimately the the vector field now depends on the”

Claim: Flow matching can eliminate the need to access the density, while diffusion requires accessing it, although a score transformation can partly avoid that requirement.

“So, uh effectively what we now have is we have the like the the self coupling which originally in the dif in like a flow measure we can get rid of. We can completely get rid of ever needing to touch the density. While in flow measureing, uh while in diffusion you end up having to touch the density, of course there's this trick with like the the score transformation which means you kind of get around this. But ultimately the the vector field now depends on the Um it now depends on the current state of our diffusion. The nice part about this is that the like diffusion style has some pretty nice properties, specifically that it's uh kind of minimising the uh uh the the relative entropy so the K_L_ divergence between like your your target density and your your your current density.”

Claim: In diffusion, the vector field depends on the process's current state.

“So, uh effectively what we now have is we have the like the the self coupling which originally in the dif in like a flow measure we can get rid of. We can completely get rid of ever needing to touch the density. While in flow measureing, uh while in diffusion you end up having to touch the density, of course there's this trick with like the the score transformation which means you kind of get around this. But ultimately the the vector field now depends on the”

Claim: Flow matching can eliminate the need to work with the density, whereas diffusion involves the density, though score transformation can partly avoid that requirement.

“Um it now depends on the current state of our diffusion. The nice part about this is that the like diffusion style has some pretty nice properties, specifically that it's uh kind of minimising the uh uh the the relative entropy so the K_L_ divergence between like your your target density and your your your current density.”

Claim: Diffusion-style methods minimize the KL divergence between the target density and the current density.

“flow matching is a method straddling the space between diffusion processes and continuous normalizing flows.”

Claim: Flow matching occupies a middle ground between diffusion processes and continuous normalizing flows.

“flow matching is a method straddling the space between diffusion processes and continuous normalizing flows.”

Claim: Flow matching occupies a methodological space between diffusion processes and continuous normalizing flows.

“The upside of normalizing flows is the ability to couple arbitrary measures with each other, while diffusion is theoretically restricted to couple with a normal density.”

Claim: Normalizing flows can couple arbitrary measures, while diffusion is theoretically restricted to coupling with a normal density.

“The upside of normalizing flows is the ability to couple arbitrary measures with each other,”

Claim: Normalizing flows can couple arbitrary measures.

“while diffusion is theoretically restricted to couple with a normal density.”

Claim: Diffusion methods are theoretically restricted to coupling with a normal density.

“The upside of a diffusion is the fast training, since we do not need to solve a full ODE for every optimization step.”

Claim: Diffusion methods train quickly because they do not require solving a full ODE at every optimization step.

“The upside of a diffusion is the fast training, since we do not need to solve a full ODE for every optimization step.”

Claim: Diffusion trains quickly because it does not require solving a full ODE at every optimization step.

“Flow matching exploits the fact that the actual path taken from base measure to target measure is arbitrary.”

Claim: Flow matching allows an arbitrary path from the base measure to the target measure.

“Flow matching exploits the fact that the actual path taken from base measure to target measure is arbitrary.”

Claim: Flow matching takes advantage of the fact that the path from a base measure to a target measure can be chosen arbitrarily.

“This means that we can simply impose a reference direction which we fix beforehand.”

Claim: Flow matching permits a reference direction to be selected in advance.

“This means that we can simply impose a reference direction which we fix beforehand.”

Claim: Flow matching can impose a reference direction chosen in advance because the path between measures is arbitrary.

“A diffusion model's reference is given implicitly through the noising process. In fact, a diffusion model is simply a very specific reference flow that aims to minimise the KL divergence.”

Claim: A diffusion model's noising process implicitly specifies its reference; diffusion is a specific reference flow that aims to minimize KL divergence.

“A diffusion model's reference is given implicitly through the noising process. In fact,”

Claim: A diffusion model's reference flow is implicitly determined by its noising process.

“a diffusion model is simply a very specific reference flow that aims to minimise the KL divergence.”

Claim: A diffusion model can be understood as a specific reference flow designed to minimize KL divergence.

“Flow matching still frame sampling as a reparametrization problem, but now fixes the degree of freedom in a path priori.”

Claim: Flow matching frames sampling as a reparameterization problem and fixes the path in advance.

“Flow matching still frame sampling as a reparametrization problem, but now fixes the degree of freedom in a path priori.”

Claim: Flow matching treats sampling as a reparameterization problem and specifies the path in advance.

“Pick a path, for instance a straight line, between a real sample and a base sample, and train a model to emulate that.”

Claim: Flow matching can use a chosen path, such as a straight line between a real sample and a base sample, and train a model to emulate it.

“Pick a path, for instance a straight line, between a real sample and a base sample, and train a model to emulate that.”

Claim: Flow matching can use a chosen path, such as a straight line between a real sample and a base sample, and train a model to emulate that path.

“The model's task is now to simply learn a general parametrization of the example flows-a simple supervised regression problem.”

Claim: In flow matching, the model learns a general parameterization of example flows through supervised regression.

“First, it allows us to give supervision on every part of the flow without need to simulate the entire O_D_E_ Intermediary targets are known and can be used as supervision signals.”

Claim: Fixing the path in flow matching provides known intermediate targets for supervision at every point without simulating the full ODE.

“First, it allows us to give supervision on every part of the flow without need to simulate the entire O_D_E_ Intermediary targets are known and can be used as supervision signals.”

Claim: Fixing a path in flow matching enables supervision at every point without simulating the full ODE, since intermediate targets are known.

“Second, it allows us to shape the path based on properties we want. For instance, straight line paths tend to be easier to sample from than highly non-linear ones.”

Claim: A fixed path can be designed around desired properties, and straight-line paths tend to be easier to sample from than highly nonlinear paths.

“Second, it allows us to shape the path based on properties we want. For instance, straight line paths tend to be easier to sample from than highly non-linear ones.”

Claim: Flow-matching paths can be designed for desired properties, and straight-line paths tend to be easier to sample from than highly nonlinear paths.

“As such, flow matching allows all the flexibility of normalizing flows, while having the superior sampling properties of the fusion models.”

Claim: Flow matching offers the flexibility of normalizing flows alongside what the speaker calls the superior sampling properties of diffusion models.

“As such, flow matching allows all the flexibility of normalizing flows, while having the superior sampling properties of the fusion models.”

Claim: Flow matching offers the flexibility of normalizing flows alongside the superior sampling properties of diffusion models.

“I think this is pretty hard to say,”

Claim: The speaker finds it difficult to say where flow-matching or diffusion methods are used.

“because ultimately you would expect something like this for a general model.”

Claim: The speaker expects general models to exhibit a similar pattern of coarse structure emerging before finer details.

“And the reason is that why the latents themselves don't hold any information. Like a latent is completely unobserved. They don't hold fundamental information.”

Claim: Latent variables are unobserved and do not themselves contain fundamental information.

“If you want them to hold fundamental information, you need to go to the Bayesian people.”

Claim: The speaker says that latents can be made to contain fundamental information by using a Bayesian approach.

“The actual latents don't hold any information, but of course in practice if you enforce the fact that the change from let's say latent number one to latent number two should be really small, then you would expect something like this, where effectively like the big structure up here first, because on the big s because if we don't have the big structure it's kind of the small changes later on cannot reconstruct this. So”

Claim: When successive latent changes are constrained to be small, the speaker expects large-scale structure to emerge before smaller details.

“I don't really have a good argument for why it has to be like this, and I don't even think it has to be like this. I think there's more a notion of kind of what happens to be easy for these models,”

Claim: The speaker does not believe coarse-to-fine emergence is necessary; he thinks it may instead reflect what is easy for the models.

“Uh I would imagine that most that they say in models would structure it like this. This doesn't mean that you couldn't build a diffusion model which doesn't do that. I could imagine going and kind of enforcing effectively a constraint that says well you have to do the the small things first. I don't think it would be a good model. Uh you might need like let's say thousands of times more parameters to actually uh fit this. You you still would be uh you still would be able to”

Claim: The speaker expects a diffusion model forced to produce small details before large-scale structure to perform poorly as a model.

“Uh I would imagine that most that they say in models would structure it like this. This doesn't mean that you couldn't build a diffusion model which doesn't do that. I could imagine going and kind of enforcing effectively a constraint that says well you have to do the the small things first. I don't think it would be a good model. Uh you might need like let's say thousands of times more parameters to actually uh fit this. You you still would be uh you still would be able to”

Claim: The speaker expects most diffusion models to produce large-scale structure before details, while acknowledging that a model could be designed to produce small details first.

“Uh I would imagine that most that they say in models would structure it like this. This doesn't mean that you couldn't build a diffusion model which doesn't do that. I could imagine going and kind of enforcing effectively a constraint that says well you have to do the the small things first. I don't think it would be a good model. Uh you might need like let's say thousands of times more parameters to actually uh fit this. You you still would be uh you still would be able to”

Claim: A diffusion model constrained to produce small details before large-scale structure might require thousands of times more parameters to fit the data.

“to to train this, at least I think so. Mathematically there's no reason why it shouldn't work. Practically there is lots of reasons why it shouldn't work.”

Claim: A diffusion model that produces small details first could be trainable in principle, but the speaker sees many practical obstacles to making it work.

“like let's say we have these billion parameter models, I am not sure it matters as much as people think.”

Claim: Alexander Mattick is unsure that architectural choices matter as much as people think for billion-parameter models.

“I think it matters in the sense that you need to make sure that your input data actually fits into the model.”

Claim: Alexander Mattick thinks an architecture must be able to accommodate its input data.

“So, let's say you couldn't do like an M_L_P_ which does like the same type of like large language modelling things just because it has a finite input space.”

Claim: Alexander Mattick says a multilayer perceptron cannot perform the same kind of large language modeling because it has a finite input space.

“But I don't think that there is a fundamental limitation on one of these architectures if you look at it at huge scales. Because at huge scales”

Claim: Alexander Mattick does not think any of the discussed model architectures has a fundamental limitation when considered at huge scales.

“All of these perform a lot less like parameterized models of like a specific architecture and more like like function spaces.”

Claim: Alexander Mattick says that at huge scales, models behave more like function spaces than like parameterized models tied to a specific architecture.

“Effectively the actual change of the individual parameters does not really mean anything anymore but more kind of the bulk change of the entire like field around this. Like this is effectively the mean field theory approach to deep learning analysis where effectively”

Claim: Alexander Mattick says that in large models, aggregate changes across the parameter field matter more than changes to individual parameters.

“Effectively the actual change of the individual parameters does not really mean anything anymore but more kind of the bulk change of the entire like field around this. Like this is effectively the mean field theory approach to deep learning analysis where effectively”

Claim: Alexander Mattick characterizes analyzing deep learning through aggregate parameter-field changes as a mean field theory approach.

“you model the actual bulk properties with like a Navier-Stokes. And the same thing I think is true with large language or let's say any large model.”

Claim: Alexander Mattick thinks large language models and other large models should be understood through their bulk properties rather than their individual components.

“really difficult to actually make like insightful statements about like what architecture leads to what, because I don't think it ultimately matters that much.”

Claim: Alexander Mattick thinks it is difficult to make insightful predictions about outcomes from architecture choices because architecture ultimately matters relatively little.

“I think it matters way more kind of the scale.”

Claim: Alexander Mattick thinks model scale matters much more than architectural choice.

“And of course, it might be that it's a one architecture just scales more nicely.”

Claim: Alexander Mattick allows that one model architecture might scale more efficiently than another.

“Like one architecture needs a billion parameters, the other one needs 100 billion parameters just because the other architecture is just less nice. nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”

Claim: Alexander Mattick gives the example that one architecture might require a billion parameters while another requires 100 billion because the second architecture scales less favorably.

“nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”

Claim: Alexander Mattick thinks the better scaling behavior of deep neural networks is probably why practitioners choose them over very wide shallow networks.

“nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”

Claim: Alexander Mattick thinks architectural choice makes no difference in the large-scale limit, though he is not referring to an infinite limit.

“How you would expect the information to to proceed through these models. Especially in diffusion, I could say with these diffusion models. In diffusion models we have this uh the structure that they are doing uh how how was it called? That they they are doing K_L divergence minimisation in a Wasserstein space.”

Claim: Alexander Mattick says diffusion models involve KL-divergence minimization in a Wasserstein space.

“So you would expect also that the higher information things to appear first, because have they they are the free wins. I don't think it has to be like that, but it sh but I would expect it to be like that. The same thing like a rectified flow. In a rectified flow I don't think it should be that surprising that if you basically do a linear interpolation of big things appear first.”

Claim: Alexander Mattick expects higher-information features to appear first in diffusion models because they are easier gains, but he does not consider this necessary.

“So you would expect also that the higher information things to appear first, because have they they are the free wins. I don't think it has to be like that, but it sh but I would expect it to be like that. The same thing like a rectified flow. In a rectified flow I don't think it should be that surprising that if you basically do a linear interpolation of big things appear first.”

Claim: Alexander Mattick says it should not be surprising if rectified flow produces large-scale features before smaller details when using linear interpolation.

“I am not sure if I could advise people or like in good conscience advise people and say, well, you should do this and that for your architecture, because I don't think it matters.”

Claim: Alexander Mattick is unsure he could responsibly advise practitioners on a particular architecture because he thinks architectural choice does not matter much.

“If we look at deep neural networks, there is of course a very, very obvious way, like a first instincts way and say, well, this is just a huge nonlinear program.”

Claim: One obvious first approach to understanding deep neural networks is to treat them as huge nonlinear programs.

“And like this is all made a really really interesting way of thinking about the problem. But it's not necessarily that useful of a problem. Now there are of course counter examples, things where we actually look at kind of the parameterized nature of these, and we can actually make some bigger inferences. With effective we were like the the optimization people uh kind of look into deep learning. Especially um like if you look uh at example spline theory of deep learning, like if you if it actually”

Claim: The speaker considers optimization-based analysis of deep learning interesting but not necessarily very useful.

“to actually look at what what the mathematicians are doing, like all the combinatorics people they all say, well that looks like something I can do. And ultimately there are really really interesting things there as well. But I don't think they give you that much insight. They can give you statements like okay, what's the actual complexity of a learned model?”

Claim: The speaker thinks combinatorial analyses of deep learning can characterize learned-model complexity but generally provide limited insight.

“Uh h b see how many of these like linear regions exist. Something like this. They can also sometimes get some really nice explanation. Like I really like the uh the paper by I think by Balthasario who did the uh explanation why batch norm works. And he just framed that as well batch norm actually concentrates the things in the middle and therefore all the hyperplanes are more concentrated. And they're a it's effectively it's more linear uh more non-linear. But”

Claim: The speaker likes an explanation of batch normalization in which it concentrates values toward the middle and thereby concentrates the hyperplanes.

“I don't think it's ultimately that insightful because at some point, again, bold properties. It doesn't matter anymore what an individual parameter does.”

Claim: In the analysis being discussed, individual parameter behavior eventually ceases to matter.

“And that then lets us walk into something like the no attention kernel theory. It allows you to like there are some like randomized uh like randomized sketching type uh results. And here is also like mean field theory, which I really like from analysis point of view. I'm not sure whether it's that useful ultimately, but I really like it from like a mental model.”

Claim: The speaker likes mean field theory as a mental model but is unsure whether it is ultimately useful.

“what if we look at like a, let's say a one hidden layer MLP, very simple. We can view this as effectively every input neuron is like, is this inner product with activation around this. So.”

Claim: A one-hidden-layer multilayer perceptron can be represented as input neurons computing inner products followed by an activation function.

“in like we can view this as let's say a thousand let's say test functions. So the activation function is let's say you make a sigmoid to make it very very instructive between zero and one. And it figures out well is this test so this weight, this inner product, is this test successful.”

Claim: In the speaker's illustrative formulation, sigmoid-activated input units act as tests of weight-defined inner products and produce values between zero and one.

“And then the next layer, that's a combination of all of these.”

Claim: In the described multilayer perceptron, the next layer combines the outputs of the preceding units.

“What we can do is we can phrase it as what happens if instead of having fifty layer fifty input uh uh systems, we have an infrared number. If you put this into the infrared limit, you effectively now write this down as like a density. Right because effectively every single point exists, it just matters kind of how present it actually is. Kind of how relevant it becomes. And now uh we have this the structure of effect to be fitting a density over uh over weights and kind of the yeah fitting a density over these weights. And that is very nicely channelized, because now we are back in like literal diffusion style problem, because now we can model the change in density over time as a change in distribution over time, which is ultimately a diffusion style problem. And that”

Claim: In the infinite-width limit described by the speaker, a neural network can be represented by a density over weights, and its evolution can be modeled as a diffusion-style change in that distribution.

“And the function space then kind of spans the the area. Now they are some limitations with this analysis. For instance the neural tangent kernel theory assumes the kernel is stationary. But even that is because it's an assumption that we make for like analysis point of view.”

Claim: Neural tangent kernel theory assumes a stationary kernel as an analytical assumption.

“There is nothing that fundamentally hinders us in looking at the kernel itself changing. So in fact we have two dynamics where we have both the the kernel change dynamics. And we have the uh kind of kernel itself describing the the local function. And this is also what mean field theory does. Mean field theory effectively in the c in the back of their their big paper, they have a a section where they talk about well actually the connection to neurotangent kernel theories, we just assume the kernel also changes.”

Claim: The speaker says there is no fundamental barrier to analyzing a changing kernel, which entails both kernel-change dynamics and dynamics of the local function described by the kernel.

“And we have the uh kind of kernel itself describing the the local function. And this is also what mean field theory does. Mean field theory effectively in the c in the back of their their big paper, they have a a section where they talk about well actually the connection to neurotangent kernel theories, we just assume the kernel also changes. over time. And that of course makes the analysis way way better that you can now effectively also model not only the laser learning but also the feature learning domain. But has a huge limitation now that it becomes much much harder to analyze.”

Claim: Allowing the kernel to change over time lets an analysis model feature learning in addition to lazy learning, but substantially increases analytical difficulty.

“over time. And that of course makes the analysis way way better that you can now effectively also model not only the laser learning but also the feature learning domain. But has a huge limitation now that it becomes much much harder to analyze. Still I think that from an analysis point of view these end up being more useful. Specifically because the predictions end up lining up a lot better than what we would expect for the parameterized world. Like in parameterized world”

Claim: The speaker considers function-space analyses more useful because their predictions align better with expectations than predictions from parameterized analyses.

“We say infit uh like a huge amount of parameters this thing is gonna overfit instantly. Or like this thing is basi it's completely over parameterised, why do you do this?”

Claim: A parameterized perspective suggests that a model with a very large number of parameters may overfit or be excessively overparameterized.

“From a function point of view, you say, well, no it's not over-parameterised, I'm just going closer and closer and closer to the actual true function space I'm fitting to. A a a p uh a function with just say a billion parameters is much much closer than a function with fifty parameters to being at this like infinite uh scaling domain.”

Claim: A function-space perspective interprets increasing model parameters as approaching the target function space, rather than necessarily as harmful overparameterization.

“A a a p uh a function with just say a billion parameters is much much closer than a function with fifty parameters to being at this like infinite uh scaling domain.”

Claim: In the speaker's function-space framing, a billion-parameter function is much closer than a fifty-parameter function to the infinite-scaling limit.

“uh explains this idea of overparameterisation while the parameteric uh with the parametric kind of says, well actual actually this overparameterisation might be bad, but we can exactly explain kind of how the the local architecture is. I would im however, I would imagine that in the long term, the functional analysis people they are going to be more useful just because they explain a specific architecture but they explain architectures in general. It can know a tangent kernel,”

Claim: The speaker expects function-space analysis to be more useful in the long term because it can apply across architectures, while parameterized analysis explains specific architectures.

“Every proof works for literally anything but the idea of looking at this from a function space point of view works even if you have like let's say we all go back and do like MLP mixers now. We no longer do transform, we all do MLP mixers. It still would work from a theory point of view.”

Claim: The speaker says function-space analysis would remain applicable if architectures shifted from transformers to MLP mixers.

“I because we or like we don't know we no longer use reloads like in fact we don't no longer use reloads we use like these gated activations lots of like research in the parametric world no longer really makes sense because they all assume kind of this uh this spline like structure instead like the the function space fields people say well the function space looks a little bit different but it's still um like ultimately it's like something like a barren space”

Claim: The speaker says some parameterized-analysis research may become inapplicable when models use gated activations rather than ReLUs, because it assumes a spline-like structure.

“I because we or like we don't know we no longer use reloads like in fact we don't no longer use reloads we use like these gated activations lots of like research in the parametric world no longer really makes sense because they all assume kind of this uh this spline like structure instead like the the function space fields people say well the function space looks a little bit different but it's still um like ultimately it's like something like a barren space And then it's the analysis that works.”

Claim: The speaker says function-space analysis can remain applicable even when the function space has a different structure, such as a barren space.

“I ultimately think that the most important thing is that it makes the correct explanations, it comes up with the correct solutions.”

Claim: A theory of deep learning should be judged primarily by whether it produces correct explanations and solutions.

“I don't care that much over the specific structures. Uh like I don't want to sound too negative on like the parametric point of view. Uh if it makes good predictions then like I'm completely fine with this. I just don't think it is as um as tuned to being productive. But”

Claim: The speaker places little importance on a theory's specific structures and accepts a parametric perspective if it predicts well, although he considers it less oriented toward productive research.

“But the important thing is, and I think it's also missing from from many of these theory papers, is that you go and you actually check whether the test works, uh basically you try to falsify your theory. Or try to figure out where does this theory actually break. In this case I would actually think that something like like physicsy is a lot more useful.”

Claim: Many theory papers fail to test whether their proposed tests work, falsify their theories, or determine where those theories break.

“But the important thing is, and I think it's also missing from from many of these theory papers, is that you go and you actually check whether the test works, uh basically you try to falsify your theory. Or try to figure out where does this theory actually break. In this case I would actually think that something like like physicsy is a lot more useful. Uh just because kind of this have this loop of I make a prediction, then I go, I make an experiment. It I think is much more uh much more useful. And I think that the the big thing is that theory will never be able to explain literally anything, because uh it might be able to explain things on like a pure functional point of view. Uh let's say like the mean field people, they can completely explain if I have let's say uh a density composed of let's uh let's say a Gaussian mixture model.”

Claim: A physics-like approach to deep learning theory is more useful because it links predictions to experiments.

“Uh just because kind of this have this loop of I make a prediction, then I go, I make an experiment. It I think is much more uh much more useful. And I think that the the big thing is that theory will never be able to explain literally anything, because uh it might be able to explain things on like a pure functional point of view. Uh let's say like the mean field people, they can completely explain if I have let's say uh a density composed of let's uh let's say a Gaussian mixture model.”

Claim: The speaker believes that theory will never explain literally everything.

“Uh just because kind of this have this loop of I make a prediction, then I go, I make an experiment. It I think is much more uh much more useful. And I think that the the big thing is that theory will never be able to explain literally anything, because uh it might be able to explain things on like a pure functional point of view. Uh let's say like the mean field people, they can completely explain if I have let's say uh a density composed of let's uh let's say a Gaussian mixture model. Mortal composer of this thing. Then I can completely tell you, hey, you need this and that size a or like add this and that size the error to the infrared model will be this size.”

Claim: For a density composed of a Gaussian mixture model, mean-field theory can specify required model size and the size of the resulting approximation error.

“Completely fine. The problem of course or or same thing with like the the SVM people like kernel based people. They have like sourcing capacity conditions which effectively means how much does your function have how much does your data have. The problem is”

Claim: Kernel-based methods use capacity conditions to characterize the capacity of the function and the data.

“And in practice we will never have that information for our real data. In practice we just have, maybe we have a clue, right, I think we can oftentimes kind of estimate, for instance, we can often estimate how many clusters there could be in our data with r relatively simple uh methods. But we can't really have very deep information into our structures because ultimately it's the the real world and we still approximately everything as theoretical cows.”

Claim: Researchers cannot obtain all the structural information about real-world data required by theoretical conditions in practice.

“And in practice we will never have that information for our real data. In practice we just have, maybe we have a clue, right, I think we can oftentimes kind of estimate, for instance, we can often estimate how many clusters there could be in our data with r relatively simple uh methods. But we can't really have very deep information into our structures because ultimately it's the the real world and we still approximately everything as theoretical cows.”

Claim: Simple methods can often estimate the number of clusters in a dataset, but researchers cannot obtain deep knowledge of real-world data structure.

“we try to do kind of, we do like a best effort and we try to solve everything that is solvable or is predictable and based on that we can still make real kind of predictions for the real system, which is effective in the transition from physics over to engineering and where the distinction lies doesn't.”

Claim: A physics-oriented approach aims to solve what is solvable or predictable and use those results to make predictions about real systems.

“which is effective in the transition from physics over to engineering and where the distinction lies doesn't. It ultimately doesn't matter, even if you can actually make a hard cut. And I think the the important thing is that we need to have something which makes actual predictions, and for that the loop between the the theory people which do like the the really nice things like mean field uh neural tangen kernels um”

Claim: The speaker considers the precise boundary between physics and engineering unimportant, even if it can be sharply defined.

“It ultimately doesn't matter, even if you can actually make a hard cut. And I think the the important thing is that we need to have something which makes actual predictions, and for that the loop between the the theory people which do like the the really nice things like mean field uh neural tangen kernels um Also, stuff like diffusion like diffusion is also it's a non parametric uh exper uh explanation of what these models do. You just say, well, they fit some density. Reinforcement learning, what I do, it's also they fit some density, be happy. But uh the the actual loop should be to actually then take this and trying to make inference for the real world. Like if I if I prove something converges and I test it and it doesn't”

Claim: Deep learning theory should make testable predictions and connect theoretical results to inference about real-world systems.

“Also, stuff like diffusion like diffusion is also it's a non parametric uh exper uh explanation of what these models do. You just say, well, they fit some density. Reinforcement learning, what I do, it's also they fit some density, be happy. But uh the the actual loop should be to actually then take this and trying to make inference for the real world. Like if I if I prove something converges and I test it and it doesn't Then we would then kind of and I think this is where the like there should be a deep a theory of deep learning people look at we should now go and say exactly that we need to explain this the problem is right now it's really really difficult because we have so little theory that the reason it could not work or it did not work could be just an implementation error because we don't have a good enough idea of what we do yet.”

Claim: When a proven convergence result does not hold up in testing, deep learning theorists should investigate and explain the discrepancy.

“Also, stuff like diffusion like diffusion is also it's a non parametric uh exper uh explanation of what these models do. You just say, well, they fit some density. Reinforcement learning, what I do, it's also they fit some density, be happy. But uh the the actual loop should be to actually then take this and trying to make inference for the real world. Like if I if I prove something converges and I test it and it doesn't”

Claim: The speaker describes diffusion models as non-parametric explanations in which models fit a density, and says his reinforcement-learning work can also be characterized as fitting a density.

“Then we would then kind of and I think this is where the like there should be a deep a theory of deep learning people look at we should now go and say exactly that we need to explain this the problem is right now it's really really difficult because we have so little theory that the reason it could not work or it did not work could be just an implementation error because we don't have a good enough idea of what we do yet.”

Claim: Because deep learning theory is currently underdeveloped, a failed test may be difficult to diagnose: it could reflect an implementation error rather than a theoretical flaw.

“I approve as you mentioned that I basically waste like a like I'd say like two or three months with an implementation bug in one of my uh one of my papers.”

Claim: The speaker says an implementation bug in one of his papers cost him about two or three months.

“Just the fact that we cannot figure out whether there's an implementation bug in like a a a continuation of existing work that that alone should mean that the theory is not yet at the level And this is what we should be looking at.”

Claim: If researchers cannot identify an implementation bug in work extending existing research, that indicates the underlying theory is not yet developed enough.

“I don't think there's really a path right now which gives us that amount of progress. Not the non-parametric people or the parametric people, not the mechanical interpretability people. No one can tell us.”

Claim: The speaker believes no current research approach, including parametric, non-parametric, or mechanistic interpretability, can provide the needed progress or answers.

“I guess we live in really strange times. like we don't even know what to believe anymore because, you know, we had all of these.”

Claim: The speaker says that people no longer know what to believe because claims that were confidently made a few years ago have become uncertain.

“all of these things that we used to say confidently a few years ago and yeah we've got things like the double descent phenomenon we've got the mystery of overcoming the curse of dimensionality from a sort of linguistics point of view people talk about systematicity and productivity and why why are the models syntactically competent why can they produce all of these sentences that have never been uttered before why are they why are they learning these abstract models are they too incomplete all of these different things And we need to have a theory that kind of explains this phenomena.”

Claim: The interviewer argues that a theory is needed to explain phenomena including double descent, apparent overcoming of the curse of dimensionality, systematicity, productivity, and models' ability to produce novel syntactically competent sentences.

“all of these things that we used to say confidently a few years ago and yeah we've got things like the double descent phenomenon we've got the mystery of overcoming the curse of dimensionality from a sort of linguistics point of view people talk about systematicity and productivity and why why are the models syntactically competent why can they produce all of these sentences that have never been uttered before why are they why are they learning these abstract models are they too incomplete all of these different things”

Claim: The speaker says that double descent, apparent overcoming of the curse of dimensionality, syntactic competence, novel sentence production, and abstract representations are phenomena that need explanation in deep learning.

“And we need to have a theory that kind of explains this phenomena.”

Claim: The speaker argues that a theory is needed to explain the observed phenomena in deep learning.

“Yeah, I think there's also a big problem of that a lot of the research is really disjointed.”

Claim: The speaker thinks much of AI research is disjointed.

“And it's not like in physics where we have a standard model of physics and we kind of we work around this. If you go to school and you learn the standard model, you can then basically start, okay, we have a common starting off point.”

Claim: The speaker contrasts AI research with physics, where a standard model provides researchers with a common starting point.

“Baron, back in like Baron theorem, he literally proved that the actual convergence is not dimension dependent.”

Claim: The speaker says Barron’s theorem proves that the relevant convergence bound is independent of dimension.

“dimension free bound, it depends on kind of the the frequencies of the target function you're fitting. And kind of intuitive like once you think about this actually also makes sense. Like if you have a high dimensional function which is constant almost everywhere and uh then it's much much easier to fit if you have a like two dimensional function which looks like a heartbeat monitor.”

Claim: The speaker says that a high-dimensional function constant almost everywhere can be much easier to fit than a two-dimensional, highly varying function resembling a heartbeat monitor.

“dimension free bound, it depends on kind of the the frequencies of the target function you're fitting. And kind of intuitive like once you think about this actually also makes sense. Like if you have a high dimensional function which is constant almost everywhere and uh then it's much much easier to fit if you have a like two dimensional function which looks like a heartbeat monitor.”

Claim: The speaker says that the dimension-free convergence bound depends on the frequencies of the target function being fitted.

“And in fact I think the same thing is true for most other estimators as well. Like if you think about like a decision tree, same property.”

Claim: The speaker thinks that most other estimators, including decision trees, share the property that effective fitting difficulty need not grow with irrelevant dimensions.

“The the the high dimension of it's constant all dimensions but one then still it's it's dimension it's it doesn't care about all the other dimensions. So but you would need to have something like a standard work, which actually kind of looks at this and”

Claim: The speaker says that when a high-dimensional function varies in only one dimension, the other dimensions do not affect the relevant fitting difficulty.

“still it's it's dimension it's it doesn't care about all the other dimensions. So but you would need to have something like a standard work, which actually kind of looks at this and combines all of this. So something that doesn't figure out like the the next paper, but something that actually goes and writes down kind of tries to figure out how do all of these things fit into one coherent narrative and I'm not sure whether deep learning as it's like machine learning as it's predication right now is actually equipped to do that because right now the the pace is so fast and like you you have like the”

Claim: The speaker argues that machine learning needs a synthesis that integrates existing results into a coherent narrative instead of concentrating only on producing the next paper.

“but something that actually goes and writes down kind of tries to figure out how do all of these things fit into one coherent narrative and I'm not sure whether deep learning as it's like machine learning as it's predication right now is actually equipped to do that because right now the the pace is so fast and like you you have like the”

Claim: The speaker is unsure whether machine learning as currently practiced can produce a coherent theoretical synthesis, given the field’s rapid pace.

“Effectively you have three big conferences every single year, if you don't couple uh count triple A_I_ if you count that you have four. Effectively you go from conference to conference to conference to conference, and you ha and all of these papers, like”

Claim: The speaker says that machine learning has effectively three major conferences per year, or four when AAAI is counted.

“I don't know maybe it's it's just me, maybe I'm just unlucky, but if I look at the papers I'm reviewing, they're really incremental all of them. And I I I don't fault all of all of the people there for for publishing like this, because first of all it's got re it's rewarded, and secondly, it doesn't make sense to go and say, well I'm setting up a big deep learning theory, I I write this book uh over over my P_H_D_ like I I don't do cumulative, I actually do a manuscript, I I have in the end, my exactly five thousand pages big book where I categorize everything that is is to know by the time you're done it's going to be completely out of date and useless like probably lots of things you write in there are completely wrong so I think that right now there is no that there is no way to have like of course we have a theory of deep learning I talked about the this paper like this there will be a theory of deep learning it's kind of a vacuous statement because”

Claim: The speaker says that writing a comprehensive deep-learning theory book during a PhD would be impractical because the field could make it outdated and useless before completion, and parts of it might be wrong.

“I don't know maybe it's it's just me, maybe I'm just unlucky, but if I look at the papers I'm reviewing, they're really incremental all of them. And I I I don't fault all of all of the people there for for publishing like this, because first of all it's got re it's rewarded, and secondly, it doesn't make sense to go and say, well I'm setting up a big deep learning theory, I I write this book uh over over my P_H_D_ like I I don't do cumulative, I actually do a manuscript, I I have in the end,”

Claim: The speaker says researchers are rewarded for incremental publications and does not blame them for responding to those incentives.

“I don't know maybe it's it's just me, maybe I'm just unlucky, but if I look at the papers I'm reviewing, they're really incremental all of them. And I I I don't fault all of all of the people there for for publishing like this, because first of all it's got re it's rewarded, and secondly, it doesn't make sense to go and say, well I'm setting up a big deep learning theory, I I write this book uh over over my P_H_D_ like I I don't do cumulative, I actually do a manuscript, I I have in the end,”

Claim: The speaker says the papers he reviews seem incremental, while acknowledging that this impression may be due to his own experience or bad luck.

“my exactly five thousand pages big book where I categorize everything that is is to know by the time you're done it's going to be completely out of date and useless like probably lots of things you write in there are completely wrong so I think that right now there is no that there is no way to have like of course we have a theory of deep learning I talked about the this paper like this there will be a theory of deep learning it's kind of a vacuous statement because Well, arguably we already have plenty of them. Uh the the the problem is more that to have something like physics have I don't think the entire field is set up to do that. And I think that it will not be set up to do that and”

Claim: The speaker considers the statement that there will be a theory of deep learning vacuous because many theories of deep learning arguably already exist.

“Well, arguably we already have plenty of them. Uh the the the problem is more that to have something like physics have I don't think the entire field is set up to do that. And I think that it will not be set up to do that and Up until the point where things start slowing down.”

Claim: The speaker thinks the machine-learning field is not organized to produce a unified theory comparable to physics and will remain unprepared to do so until the pace slows.

“So our mutual friend Randall Bellistriaro, he's not a fan of the manifold hypothesis.”

Claim: The speaker says Randall Bellistriaro is not a supporter of the manifold hypothesis.

“So he's got this spline theory, which is basically that in an input sensitive way, the ambient space gets partitioned into these little honeycomb regions and you were helping me understand this at the time. So you get this weird teleporting thing happen where.”

Claim: The speaker describes Randall Bellistriaro’s spline theory as proposing that ambient space is partitioned into small honeycomb regions in an input-sensitive way.

“the ambient space gets partitioned into these little honeycomb regions and you were helping me understand this at the time. So you get this weird teleporting thing happen where. where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti manifold hypothesis but i spoke with goodfire last week they're this interpretability company in san francisco and they've got all of this work about the the geometry that these models learn so for example days of the week and months of the year they can take a pca projection of of the the activations and then they they they”

Claim: The speaker says that in the spline-theory account, a small input change can cause the representation to jump to a different honeycomb region elsewhere in the space.

“where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti manifold hypothesis but i spoke with goodfire last week they're this interpretability company in san francisco and they've got all of this work about the the geometry that these models learn so for example days of the week and months of the year they can take a pca projection of of the the activations and then they they they they fit this kind of Bézier curve to it and they see these beautiful kind of geometric structures in the activation space and they're really kind of manifold peeled.”

Claim: The speaker says Goodfire researchers use PCA projections and Bézier curves on model activations for days of the week and months of the year, finding geometric structures in activation space.

“where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti manifold hypothesis but i spoke with goodfire last week they're this interpretability company in san francisco and they've got all of this work about the the geometry that these models learn so for example days of the week and months of the year they can take a pca projection of of the the activations and then they they they they fit this kind of Bézier curve to it and they see these beautiful kind of geometric structures in the activation space and they're really kind of manifold”

Claim: The speaker says Goodfire, which he describes as an interpretability company in San Francisco, has research on geometric structures learned by models.

“where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti manifold hypothesis but i spoke with goodfire last week they're this interpretability company in san francisco and they've got all of this work about the the geometry that these models learn so for example days of the week and months of the year they can take a pca projection of of the the activations and then they they they”

Claim: The speaker characterizes the spline-theory account of input-sensitive honeycomb regions as contrary to the manifold hypothesis.

“I think it's really hard to make such a statement, whether it exists or not.”

Claim: The speaker considers it very difficult to determine whether the manifold hypothesis is true.

“i something like the manifold hypothesis like if it's not the manifold like if it's not like a low dimensional manifold itself something like this has to exist so we can actually compress things down and ultimately i don't i don't even think that uh randall would disagree with me on that that if it actually is like if we actually have like let's say the model outside like the real world is the small thing there is and there's no simplification then what are we doing um”

Claim: The speaker argues that some structure must exist to make compression possible, even if that structure is not a low-dimensional manifold; otherwise, modeling the real world without simplification would have no clear purpose.

“But it would also push back against all of these ex explainability techniques. Because generally speaking they are th they're not very robust. Like they make firstly they they often make like extremely simplifying assumptions. Like let's say like the the circuit um uh people like they uh like the the transform of circuits but they usually make the they usually make the assumption that the actual routing is constant as you change kind of as you permute your uh your”

Claim: The speaker considers current explainability techniques generally insufficiently robust and says they often depend on highly simplifying assumptions.

“But it would also push back against all of these ex explainability techniques. Because generally speaking they are th they're not very robust. Like they make firstly they they often make like extremely simplifying assumptions. Like let's say like the the circuit um uh people like they uh like the the transform of circuits but they usually make the they usually make the assumption that the actual routing is constant as you change kind of as you permute your uh your Yeah, that's not true.”

Claim: The speaker says circuit-based explainability analyses often assume routing stays constant under permutations, but that assumption is false.

“So I think there are many, many like silent assumptions and I think there's even a paper like this where they show that if you actually change some, make some slight tweaks or like a setup of like these experiments, like the results completely flip.”

Claim: The speaker says that research has shown slight changes to explainability experiment setups can completely reverse the results.

“I think like all of these are just, they're not robust enough to actually make such statements of making statements of like the manifold hypothesis exists, but it's also not robust. robust enough to say it doesn't exist.”

Claim: The speaker believes current evidence is too fragile to conclude either that the manifold hypothesis is true or that it is false.

“I think it's really really difficult and I think it's also difficult because I don't know in I don't know whether the language exists to realistically talk about these problems because like they there are false people that go very much into like the these uh the abstract and functional space I don't think this is completely true”

Claim: The speaker is unsure whether an adequate language exists for realistically describing how models work.

“Uh it's oftentimes very useful, but it's not completely true just because it we don't have infinite-dimensional functions, uh i it's not they're finite-sized. And there's a big difference between finite and infinite, right? If you have infinite dimensions, every problem becomes convex. In fact every problem becomes linear. So um on the other side with the parametric people, we say well we just have these these spline regions and that that explains everything. But if”

Claim: The speaker considers abstract and functional-space descriptions useful but incomplete, noting that models are finite-sized rather than infinite-dimensional.

“Uh it's oftentimes very useful, but it's not completely true just because it we don't have infinite-dimensional functions, uh i it's not they're finite-sized. And there's a big difference between finite and infinite, right? If you have infinite dimensions, every problem becomes convex. In fact every problem becomes linear. So um on the other side with the parametric people, we say well we just have these these spline regions and that that explains everything. But if”

Claim: The speaker asserts that in infinite-dimensional settings, every problem becomes convex and, more strongly, linear.

“I if it was like this then we would need to be able to make predictions that say, well if we go away from the splines maybe it doesn't work anymore. Or like to what extent does it stop working? And then there the explainability people that say, well we can actually extract uh this and that circuit or we can extract this and that representation. Like maybe one of the the best papers I I saw in like explainability was the paper about grokking where they ex where they extracted like the modular arithmetic.”

Claim: The speaker considers a paper that extracted modular arithmetic in grokking one of the best explainability papers they had seen.

“I if it was like this then we would need to be able to make predictions that say, well if we go away from the splines maybe it doesn't work anymore. Or like to what extent does it stop working? And then there the explainability people that say, well we can actually extract uh this and that circuit or we can extract this and that representation. Like maybe one of the the best papers I I saw in like explainability was the paper about grokking where they ex where they extracted like the modular arithmetic.”

Claim: The speaker argues that spline-based accounts should predict the conditions or extent to which their explanations fail, including when inputs move away from spline regions.

“if this is how these models actually describe modular arithmetic, there is not a chance we will explain how the model like a duck. So I don't think there is a way of of actually currently having just a language of describing of describing what these models do so I don't think that the like something like a manifold hypothesis I don't think it's true I don't think it's wrong I don't I don't think it's a useful way even if it's true I don't think it's a useful way of thinking about the problem same thing is even if like the extracted a grocking star representation even if they're true I don't think they're a useful way of actually actually figuring out what these models do and kind of how they get the solution and whether things work or don't work.”

Claim: The speaker does not consider the manifold hypothesis or extracted grokking representations useful frameworks for understanding model behavior, solution processes, or the conditions under which models succeed, even if those accounts are true.

“these representations for example geography you actually have some kind of geometric invariant of how the different countries in the world are related to each other and it's fascinating that that this would be convergent that different model families would learn similar types of geometry that represent abstract structural invariances of the world it's very very exciting but yeah I agree maybe we're reading too much into that we should move on to reward is enough and constrained”

Claim: The speaker considers it exciting that different model families may converge on similar geometric representations of abstract structural invariants in the world.

“these representations for example geography you actually have some kind of geometric invariant of how the different countries in the world are related to each other and it's fascinating that that this would be convergent that different model families would learn similar types of geometry that represent abstract structural invariances of the world it's very very exciting but yeah I agree maybe we're reading too much into that we should move on to reward is enough and constrained”

Claim: The speaker agrees that observers may be overinterpreting similarities between model representations.

“these representations for example geography you actually have some kind of geometric invariant of how the different countries in the world are related to each other and it's fascinating that that this would be convergent that different model families would learn similar types of geometry that represent abstract structural invariances of the world it's very very exciting but yeah I agree maybe we're reading too much into that we should move on to reward is enough and constrained”

Claim: The interviewer finds it exciting that different model families might learn similar geometric representations of abstract structural invariances in the world.

“The idea of, especially the reinforcement learning maximalists, is that they say, well, technically, I can write down any objective I want as some form of reward.”

Claim: Reinforcement-learning maximalists argue that any objective can technically be represented as a reward function.

“I have my x star as my actual solution, and just write down as like minimise the error between x uh x and x star. It's a convex problem, it instantly works.”

Claim: When an optimization problem's solution is already known, minimizing the error between a candidate and that solution yields a convex problem that the speaker says works immediately.

“Okay, if it's a radioactive low dimension you can solve it by hand. Even if it's really really high dimensional, like the the actual problem behind is really really high dimensional, really complex. So in that sense, if the the idea of like the reward being enough is completely true. Uh technically it is like technically you can define reward function for pretty much anything it will somehow figure it out”

Claim: The speaker says that a reward function can technically be defined for almost anything, and that it will somehow figure it out.

“the question is more whether this is a realistic assumption to make like whether having like a god-given reward function which just does something is Whether it's a realistic assumption, even finding it and once we have it, whether it's realistically optimizable with any technique and that is where I think the the actual like the practicality or like the the reasonableness of this breaks down.”

Claim: The speaker thinks the reward-is-enough idea becomes impractical because it may be unrealistic to find a suitable reward function and optimize it with available techniques.

“So of course like the the God function was a very very extreme example but ultimately what he believes is he believes that you can figure out like a suitable reward function for every task and that is enough like we don't”

Claim: The speaker characterizes Richard Sutton's view as holding that a suitable reward function can be found for every task and is sufficient to solve it.

“We don't need to research all these other complicated things, but instead if we just become really, really good at maximizing reward, then everything else becomes obsolete.”

Claim: The speaker says Sutton's view implies that becoming very good at reward maximization would make research into other complicated approaches obsolete.

“I mean, I think it's a very, very optimistic view of the world.”

Claim: The speaker considers the reward-is-enough view very optimistic.

“but I don't think it's a reasonable thing to do. even in like the kind of infinite limit.”

Claim: The speaker does not consider the reward-is-enough view reasonable, even in the infinite limit.

“and there were people in his lab at Cambridge doing all sorts of control theories. So, for example, there's this cartpole balancing problem and you can just solve that analytically. But this reward thing was different because it's basically in this particular case, I think it was a binary signal. signal so it's either balanced or it's not and and it's a signal from the future so you have to do a lot of stuff before that thing says yes and and at the time they thought that this was an interesting problem to solve because this is much more kind of like amenable to you know maybe it's a better representation of animal learning”

Claim: The speaker recalls that the cart-pole reward was a binary future signal indicating whether the pole was balanced, requiring actions before the signal was received.

“signal so it's either balanced or it's not and and it's a signal from the future so you have to do a lot of stuff before that thing says yes and and at the time they thought that this was an interesting problem to solve because this is much more kind of like amenable to you know maybe it's a better representation of animal learning”

Claim: The speaker recalls that researchers considered the delayed binary reward in cart-pole an interesting problem because it might better represent animal learning.

“Yeah, I think that in general there's a big overlap between control theory and what the reinforcement people do, but there's also a huge distinction.”

Claim: Control theory and reinforcement learning have substantial overlap but also a major distinction.

“Yeah, I think that in general there's a big overlap between control theory and what the reinforcement people do,”

Claim: Control theory and reinforcement learning have substantial overlap.

“but there's also a huge distinction.”

Claim: Despite their overlap, control and reinforcement learning differ substantially.

“And notably the main difference between doing actual control and doing reinforcement learning is control assumes that you have It's interesting, you have a model of the plant or at the very least you have some idea of the behaviour, like the the actual imp uh the actual response to an input.”

Claim: Control methods assume a model of the plant or at least some knowledge of how the plant responds to inputs.

“And notably the main difference between doing actual control and doing reinforcement learning is control assumes that you have It's interesting, you have a model of the plant or at the very least you have some idea of the behaviour, like the the actual imp uh the actual response to an input.”

Claim: Control methods assume that a model of the plant or some knowledge of its behavior and response to inputs is available.

“That is not true in reinforcement learning. In reinforcement you say, well I don't have a model at all, I just have interaction with the environment and that immediately makes the problem more, first of all it makes it more interesting,”

Claim: Reinforcement learning can operate without a model, using interaction with the environment instead.

“That is not true in reinforcement learning. In reinforcement you say, well I don't have a model at all, I just have interaction with the environment and that immediately makes the problem more, first of all it makes it more interesting, but also makes it a lot more complicated.”

Claim: Reinforcement learning can operate without a model, using interaction with the environment instead.

“well I don't have a model at all, I just have interaction with the environment and that immediately makes the problem more, first of all it makes it more interesting, but also makes it a lot more complicated. Because now you no longer are trying to solve the actual problem at hand, you also solve the problem of even figuring out what to do in the first place.”

Claim: Because reinforcement learning may lack a model of the environment, it must address both the task and the problem of discovering what actions to take, making the problem more complicated and, in the speaker's view, more interesting.

“well I don't have a model at all, I just have interaction with the environment and that immediately makes the problem more, first of all it makes it more interesting, but also makes it a lot more complicated.”

Claim: Operating without a model makes reinforcement learning more complicated.

“Because now you no longer are trying to solve the actual problem at hand, you also solve the problem of even figuring out what to do in the first place.”

Claim: When reinforcement learning lacks a model, it must both solve the task and determine what actions to take in the first place.

“A controller let's say we have like a reference which it tries to follow. we have some model of the like plant dynamics, it let's say it make it very extreme, we have an M_P_C_ controller, we have a uh or we have whatever else, or like a linear quadratic regulator, with some idea of kind of what this uh what this thing does, or if we have like a P_I_D_ we have like a a human who has an idea of what it should do and how it should behave. And then effectively you outsource all the exploration.”

Claim: When a controller uses a model or a human's understanding of system behavior, exploration is effectively outsourced rather than learned through reinforcement learning.

“A controller let's say we have like a reference which it tries to follow. we have some model of the like plant dynamics, it let's say it make it very extreme, we have an M_P_C_ controller, we have a uh or we have whatever else, or like a linear quadratic regulator, with some idea of kind of what this uh what this thing does, or if we have like a P_I_D_ we have like a a human who has an idea of what it should do and how it should behave. And then effectively you outsource all the exploration.”

Claim: A controller can use a reference and a model of plant dynamics—or a human's understanding of expected behavior—and thereby outsource exploration.

“Put this into the loop itself, and that gives you a call, it gives you huge amounts of power. Because now you can actually solve much more general problems, problems we don't even have an idea of how to solve them in the first place. And this is also maybe where like the the reward is enough people then go in and say well, it becomes not an arbitrary problem with like arbitrary uh definition. And we can do like the exploration we can do, we can do do the learning we can do like the optimal control”

Claim: Putting exploration and learning into the control loop gives reinforcement learning greater power to solve general problems, including problems whose solutions are not already known.

“Put this into the loop itself, and that gives you a call, it gives you huge amounts of power. Because now you can actually solve much more general problems, problems we don't even have an idea of how to solve them in the first place. And this is also maybe where like the the reward is enough people then go in and say well, it becomes not an arbitrary problem with like arbitrary uh definition. And we can do like the exploration we can do, we can do do the learning we can do like the optimal control”

Claim: Putting exploration into the learning loop gives reinforcement learning the power to address more general problems, including problems without known solutions.

“while I think this is true, in the limit, I also think it's a very wasteful way of thinking about the world,”

Claim: The speaker accepts that reinforcement learning's generality holds in the limit, but considers relying on it a very wasteful way to approach the world.

“while I think this is true, in the limit, I also think it's a very wasteful way of thinking about the world,”

Claim: The speaker thinks that relying on reinforcement learning in this way is true in the limit, but is a very wasteful way to think about the world.

“especially since things like exploration become really dicey in many domains. Like, I don't want like my open heart surgery robot to do exploration on, first of all, not me, but not on anyone else either.”

Claim: The speaker considers exploration risky in many domains and does not want an open-heart surgery robot to explore on patients.

“especially since things like exploration become really dicey in many domains.”

Claim: Exploration is risky in many domains.

“Like, I don't want like my open heart surgery robot to do exploration on, first of all, not me, but not on anyone else either.”

Claim: The speaker opposes allowing an open-heart surgery robot to explore on people.

“I think that the actual real-world complexity of needing to have either first of all not allowing too much exploration in the practical sense, but also to give really, really hard guarantees becomes difficult to do in like pure reinforcement learning.”

Claim: Pure reinforcement learning makes it difficult to limit exploration in practice and provide very strong guarantees in real-world settings.

“Because we cannot give like Alan-finity control, uh like Alan-finity guarantees. So because everything is sampling, sampling and kind of absolute uh maximal errors don't really play well with uh one another. You can have uh a control which in in expectation is really really nice, but has one really really huge failure case. But so on average something which tracks it perfectly and has one huge uh error where kind of the robot clocks you on the head is completely the same as one thing which kind of just slightly wanders around. And in fact this is this nice paper about uh what was it called like deep reune ex uh training deep reune networks to uniform accuracy is intractable or something like put into Google, you'll find it. But”

Claim: A controller may have excellent expected performance while retaining a single very large error or failure case.

“Because we cannot give like Alan-finity control, uh like Alan-finity guarantees. So because everything is sampling, sampling and kind of absolute uh maximal errors don't really play well with uh one another. You can have uh a control which in in expectation is really really nice, but has one really really huge failure case.”

Claim: A controller may have strong expected performance while retaining a single very large failure case.

“But so on average something which tracks it perfectly and has one huge uh error where kind of the robot clocks you on the head is completely the same as one thing which kind of just slightly wanders around. And in fact this is this nice paper about uh what was it called like deep reune ex uh training deep reune networks to uniform accuracy is intractable or something like put into Google, you'll find it. But”

Claim: Average tracking performance can conceal a catastrophic outlier: a controller with one huge error may be as bad as one that wanders slightly, such as a robot striking someone on the head.

“And they basically show that for radio networks, it's not even possible. Or basically it is possible, but it's exponential in the number of parameters. Because basically what they say is when we have the the radius, we have two, effectively form like a hat. Something like this. And just figuring out kind of how to place and how to scale these hats is already like an N_P_ complete problem. Like oh, like rather you need an exponential amount of data to figure out kind of these these deviations from your reference. And”

Claim: For the discussed networks, achieving uniform accuracy is described as impossible or as requiring exponential resources in the number of parameters; identifying deviations from a reference also requires exponentially much data.

“And they basically show that for radio networks, it's not even possible. Or basically it is possible, but it's exponential in the number of parameters. Because basically what they say is when we have the the radius, we have two, effectively form like a hat. Something like this. And just figuring out kind of how to place and how to scale these hats is already like an N_P_ complete problem. Like oh, like rather you need an exponential amount of data to figure out kind of these these deviations from your reference. And”

Claim: The speaker describes a result that uniform accuracy for neural networks is possible only with complexity exponential in the number of parameters, with identifying deviations from a reference also requiring exponential data.

“I think having something like this is, it's necessary for many domains where actually like a like damage is really really really bad and having just a trans in expectation that damage and good things happen most of the time is often not good enough like you want stronger things where then suddenly you go into things like robust reinforcement learning or something like this or like what i do for my phd constraint reinforcement learning where effectively you do a little bit more just to get this little bit of an extra boost in”

Claim: In domains where damage is very costly, performance that is good only in expectation or most of the time is often insufficient, so stronger guarantees are needed.

“I think having something like this is, it's necessary for many domains where actually like a like damage is really really really bad and having just a trans in expectation that damage and good things happen most of the time is often not good enough like you want stronger things where then suddenly you go into things like robust reinforcement learning or something like this or like what i do for my phd constraint reinforcement learning where effectively you do a little bit more just to get this little bit of an extra boost in”

Claim: In domains where damage is severe, methods providing stronger guarantees are necessary because good outcomes most of the time are not enough.

“like damage is really really really bad and having just a trans in expectation that damage and good things happen most of the time is often not good enough like you want stronger things where then suddenly you go into things like robust reinforcement learning or something like this or like what i do for my phd constraint reinforcement learning where effectively you do a little bit more just to get this little bit of an extra boost in either safety or performance or just including prior data like imitation and we just say well I have actually lots of data around and I don't want to do open-heart surgery from scratch so I think it's very naive to think about this like the right way”

Claim: Robust and constraint reinforcement learning can provide additional safety or performance and incorporate prior data, such as imitation.

“like damage is really really really bad and having just a trans in expectation that damage and good things happen most of the time is often not good enough like you want stronger things where then suddenly you go into things like robust reinforcement learning or something like this or like what i do for my phd constraint reinforcement learning where effectively you do a little bit more just to get this little bit of an extra boost in either safety or performance or just including prior data like imitation and we just say well I have actually lots of data around and I don't want to do open-heart surgery from scratch so I think it's very naive to think about this like the right way”

Claim: Robust and constrained reinforcement learning can provide additional safety or performance and incorporate prior data, such as imitation, rather than learning a task from scratch.

“either safety or performance or just including prior data like imitation and we just say well I have actually lots of data around and I don't want to do open-heart surgery from scratch so I think it's very naive to think about this like the right way”

Claim: The speaker considers it naive to approach open-heart surgery from scratch when substantial prior data is available.

“either safety or performance or just including prior data like imitation and we just say well I have actually lots of data around and I don't want to do open-heart surgery from scratch so I think it's very naive to think about this like the right way”

Claim: The speaker considers it naive to approach high-stakes tasks such as open-heart surgery from scratch when relevant prior data is available.

“Yeah, I suppose this is the other vexed issue with our friend Rich Sutton is that he has this bitter lesson essay, which is basically that he doesn't think. that we should be building knowledge and constraints into the system he thinks that we should just scale you know models of computation and I'm a big believer that constraints are incredibly valuable and by constraints you know more broadly I'm just talking about knowledge I think constraints happen at multiple levels of abstraction so in the physical world constraints might just be the laws of physics or whatever and and in our minds the constraints might be things that we already know and this guy Chris the other day he was saying to me that one of the amazing”

Claim: SPEAKER_02 characterizes Rich Sutton’s “Bitter Lesson” as arguing against building knowledge and constraints into systems and in favor of scaling models of computation.

“that we should be building knowledge and constraints into the system he thinks that we should just scale you know models of computation and I'm a big believer that constraints are incredibly valuable and by constraints you know more broadly I'm just talking about knowledge I think constraints happen at multiple levels of abstraction so in the physical world constraints might just be the laws of physics or whatever and and in our minds the constraints might be things that we already know and this guy Chris the other day he was saying to me that one of the amazing”

Claim: SPEAKER_02 believes constraints, broadly understood to include knowledge, are extremely valuable in AI systems.

“that we should be building knowledge and constraints into the system he thinks that we should just scale you know models of computation and I'm a big believer that constraints are incredibly valuable and by constraints you know more broadly I'm just talking about knowledge I think constraints happen at multiple levels of abstraction so in the physical world constraints might just be the laws of physics or whatever and and in our minds the constraints might be things that we already know and this guy Chris the other day he was saying to me that one of the amazing”

Claim: The interviewer describes physical constraints as potentially consisting of the laws of physics and mental constraints as knowledge already acquired.

“that we should be building knowledge and constraints into the system he thinks that we should just scale you know models of computation and I'm a big believer that constraints are incredibly valuable and by constraints you know more broadly I'm just talking about knowledge I think constraints happen at multiple levels of abstraction so in the physical world constraints might just be the laws of physics or whatever and and in our minds the constraints might be things that we already know and this guy Chris the other day he was saying to me that one of the amazing”

Claim: SPEAKER_02 describes constraints as operating at multiple levels of abstraction, including physical laws in the world and existing knowledge in minds.

“that we should be building knowledge and constraints into the system he thinks that we should just scale you know models of computation and I'm a big believer that constraints are incredibly valuable and by constraints you know more broadly I'm just talking about knowledge I think constraints happen at multiple levels of abstraction so in the physical world constraints might just be the laws of physics or whatever and and in our minds the constraints might be things that we already know and this guy Chris the other day he was saying to me that one of the amazing”

Claim: The interviewer considers constraints extremely valuable and defines them broadly as knowledge operating at multiple levels of abstraction.

“the amazing things about RLV are is that it kind of it doesn't solve but it helps the exploration problem because the base LLM knows so much so it has lots of conceptual understanding and rather than just blindly searching all over the place because obviously back in the days of deep Q networks and whatnot we actually needed to guide and and put a lot of priors into the exploration process because otherwise not very many interesting things will happen and now of course with the RLV are on the language models that they have a lot They have a lot of base knowledge,”

Claim: SPEAKER_02 says language models used with RLV contain substantial base knowledge.

“the amazing things about RLV are is that it kind of it doesn't solve but it helps the exploration problem because the base LLM knows so much so it has lots of conceptual understanding and rather than just blindly searching all over the place because obviously back in the days of deep Q networks and whatnot we actually needed to guide and and put a lot of priors into the exploration process because otherwise not very many interesting things will happen and now of course with the RLV are on the language models that they have a lot”

Claim: SPEAKER_02 says RLV helps address exploration because its base language model has substantial conceptual understanding.

“the amazing things about RLV are is that it kind of it doesn't solve but it helps the exploration problem because the base LLM knows so much so it has lots of conceptual understanding and rather than just blindly searching all over the place because obviously back in the days of deep Q networks and whatnot we actually needed to guide and and put a lot of priors into the exploration process because otherwise not very many interesting things will happen and now of course with the RLV are on the language models that they have a lot”

Claim: The interviewer says earlier deep Q-network methods required guidance and many priors during exploration, or little of interest would happen.

“the amazing things about RLV are is that it kind of it doesn't solve but it helps the exploration problem because the base LLM knows so much so it has lots of conceptual understanding and rather than just blindly searching all over the place because obviously back in the days of deep Q networks and whatnot we actually needed to guide and and put a lot of priors into the exploration process because otherwise not very many interesting things will happen and now of course with the RLV are on the language models that they have a lot”

Claim: According to a colleague quoted by the interviewer, RLV helps with exploration because the base language model has substantial conceptual understanding.

“the amazing things about RLV are is that it kind of it doesn't solve but it helps the exploration problem because the base LLM knows so much so it has lots of conceptual understanding and rather than just blindly searching all over the place because obviously back in the days of deep Q networks and whatnot we actually needed to guide and and put a lot of priors into the exploration process because otherwise not very many interesting things will happen and now of course with the RLV are on the language models that they have a lot”

Claim: SPEAKER_02 says deep Q-network systems required guidance and substantial prior information for exploration; without it, they would produce few interesting outcomes.

“They have a lot of base knowledge,”

Claim: The interviewer says language models used in RLV have substantial base knowledge.

“but it feels to me that like how you get there and how you constrain and shape the search space is critical to making it work.”

Claim: The interviewer believes that how a search space is reached, constrained, and shaped is critical to making the approach work.

“but it feels to me that like how you get there and how you constrain and shape the search space is critical to making it work.”

Claim: SPEAKER_02 believes that the method used to reach, constrain, and shape the search space is critical to making the approach work.

“which is very applied. Uh so I actually did some project which are actually applied reinforcement learning. And this is a big problem in reality that effectively information can be very expensive. And in fact I would say most information is very very expensive. And reinforcement, if you completely say well constraints don't matter, uh I I don't need any prior information, I don't want to encode anything in the thing in there. Uh assuming this even possible uh that co could being completely like unconstr uh unconstr etcetera and completely like without prior knowledge, I assume all of this the case, we We still have the problem of now needing to actually relearn everything from scratch, and the reality is that we want to be gathering as much information that we already have paid for and gathered with our blood, sweat and tears, we want to put all of that in the model first,”

Claim: SPEAKER_00 argues that reinforcement learning without constraints or prior information must relearn everything from scratch.

“which is very applied. Uh so I actually did some project which are actually applied reinforcement learning. And this is a big problem in reality that effectively information can be very expensive. And in fact I would say most information is very very expensive. And reinforcement, if you completely say well constraints don't matter, uh I I don't need any prior information, I don't want to encode anything in the”

Claim: SPEAKER_00 says information can be very expensive in practical applications and considers most information very expensive.

“We still have the problem of now needing to actually relearn everything from scratch, and the reality is that we want to be gathering as much information that we already have paid for and gathered with our blood, sweat and tears, we want to put all of that in the model first, and then we go and we continue from there. There is no point in basically starting from nowhere.”

Claim: SPEAKER_00 argues that previously gathered information should be incorporated into a model before further learning, rather than starting from nothing.

“If you actually look at a reward function, like an actual industry-ready reward function, you will immediately see that they have constraints in there.”

Claim: SPEAKER_00 says practical, industry-ready reward functions encode constraints.

“Like it's they just they just reframe them as well It's a reward function because yeah, of course like you can put them into as penalties But ultimately they are just kind of you're just doing a lot of work To not write down a constraint.”

Claim: SPEAKER_00 argues that encoding constraints as reward penalties can amount to extra work that avoids stating the constraint directly.

“The nice example of this which I always like to use as a kind of motivation for constrained RL is if you look into the paper about what is called open AI 5 which is the Dota playing AI. If you look into the appendix there's a huge table where on the left they have all these different attributes which they put into their reward function, on the right they have some explanations and so on, in the middle they have just a bunch of random numbers.”

Claim: SPEAKER_00 says the OpenAI Five paper’s appendix presents reward-function attributes, explanations, and their numerical weights.

“and these are they have their weights for their reward function like if you tell me that there is a way to actually figure this out there's not a chance so especially if you look actually at what these reward functions are like at the bottom for instance there's one which says well the eight like the there's a negative reward for being in the wrong lane for a specific amount of time”

Claim: SPEAKER_00 says OpenAI Five’s reward function penalizes remaining in the wrong lane for a specified duration.

“That's a constraint because write it down exactly the same as saying, well, don't be in there more than five seconds or whatever. Rather in mouse constraint reinforcement you say, well in expectations, if you take a thousand trajectories, don't be in the wrong lane too often or something like this. And there's another,”

Claim: SPEAKER_00 argues that a reward penalty for time spent in the wrong lane expresses the same restriction as an explicit duration constraint.

“That's a constraint because write it down exactly the same as saying, well, don't be in there more than five seconds or whatever. Rather in mouse constraint reinforcement you say, well in expectations, if you take a thousand trajectories, don't be in the wrong lane too often or something like this. And there's another,”

Claim: SPEAKER_00 says constrained reinforcement learning can impose an expected constraint across a thousand trajectories, such as limiting the frequency of being in the wrong lane.

“What can always run into trouble with reinforced reinforcement learning is above there is a section where they have like killing an enemy hero actually gives a negative reward and the reason is and they have an explanation next to it is that killing an enemy hero gives you a lot of gold and there's already like a bonus for getting gold so if you also had like a zero positive reward for killing enemy hero you would sit into the middle of the map and just kill the enemy hero over and over and over again without making any progress.”

Claim: SPEAKER_00 says that removing the negative reward for killing an enemy hero would lead the agent to repeatedly kill that hero in the middle of the map rather than make progress.

“What can always run into trouble with reinforced reinforcement learning is above there is a section where they have like killing an enemy hero actually gives a negative reward and the reason is and they have an explanation next to it is that killing an enemy hero gives you a lot of gold and there's already like a bonus for getting gold so if you also had like a zero positive reward for killing enemy hero you would sit into the middle of the map and just kill the enemy hero over and over and over again without making any progress.”

Claim: SPEAKER_00 says OpenAI Five penalizes killing an enemy hero because the kill yields gold that is already rewarded separately.

“The problem there is that two of these quantities can be completely correlated. So if you model constraints, you basically get rid of this correlation problem, because if you for example write down the constraint explicitly of I should not be like there should not in expectation I should not have more less uh let's say more than But how is it? I should not have, let's say, 50% less enemy hero curse than my opponent in expectation.”

Claim: SPEAKER_00 argues that explicitly modeling constraints can eliminate the problem caused by correlation between reward quantities.

“The problem there is that two of these quantities can be completely correlated. So if you model constraints, you basically get rid of this correlation problem, because if you for example write down the constraint explicitly of I should not be like there should not in expectation I should not have more less uh let's say more than”

Claim: SPEAKER_00 says that two reward quantities can be completely correlated, creating a problem when they are modeled through rewards.

“50% less enemy hero curse than my opponent in expectation.”

Claim: SPEAKER_00 gives an example constraint requiring the agent, in expectation, not to have more than 50% fewer enemy-hero curses than its opponent.

“It's a small guarantee, but effectively the stuff that you want to get rid of. You already get rid of with that.”

Claim: SPEAKER_00 says the example constraint provides a limited guarantee that removes the unwanted effect.

“Yeah, I mean, Leonardo da Vinci famously said that constraints give us freedom and I'm a huge fan of that.”

Claim: SPEAKER_02 strongly endorses the idea that constraints can provide freedom.

“So constrained reinforcement is still a relatively up-and-coming field, I would say, at the very least.”

Claim: Constrained reinforcement learning is still a relatively emerging field.

“Especially since there are many different formulations. The classical one is one of the constrained market decision process. So effectively what you have is exactly the same setup, you maximize the cumulative rewards. But then you have constraints which just say the expected cumulative costs should be below a threshold.”

Claim: The classical constrained Markov decision process formulation maximizes cumulative reward subject to expected cumulative costs remaining below a threshold.

“So effectively what you have is exactly the same setup, you maximize the cumulative rewards. But then you have constraints which just say the expected cumulative costs should be below a threshold.”

Claim: The classical constrained decision-process formulation maximizes cumulative reward subject to expected cumulative costs remaining below a threshold.

“Like the one big disadvantage of like this expected cost is for instance you don't get rid of long tails,”

Claim: Constraining expected cost does not eliminate long-tail outcomes.

“If you say it should be less than 25 cost. Then it could be that it has fifty trajectories, of course zero, and then fifty of course fifty. So it averages out to twenty five. But it's not necessarily what you always want. On the other hand, for something like the uh the Open A_V_ five case, it would be exactly what you want. Right, you want to have let's say some uh games where exactly you are like every one is in one lane and then some games where it's like the normal distributed amount.”

Claim: A cost distribution with half of trajectories at zero and half at 50 has an expected cost of 25, but may not always be desirable.

“Then it could be that it has fifty trajectories, of course zero, and then fifty of course fifty. So it averages out to twenty five. But it's not necessarily what you always want. On the other hand, for something like the uh the Open A_V_ five case, it would be exactly what you want. Right, you want to have let's say some uh games where exactly you are like every one is in one lane and then some games where it's like the normal distributed amount.”

Claim: An expected cost of 25 can result from half of trajectories costing zero and half costing 50, but this outcome distribution is not always desirable.

“then there are also solutions which do like conditional value at risk, which is effectively you look at kind of the probability of being in a tail and kind of you weight this by kind of how far out on the tail you are.”

Claim: Conditional value at risk accounts for the probability of tail outcomes and weights them by their distance into the tail.

“then there are also solutions which do like conditional value at risk, which is effectively you look at kind of the probability of being in a tail and kind of you weight this by kind of how far out on the tail you are.”

Claim: Conditional value at risk accounts for the probability of tail outcomes and weights them according to their distance into the tail.

“There are like percentile risks like with 99% probability you should be within like some, let's say 50% of some target, whatever. Basically like a pack. uh like guarantees.”

Claim: A percentile-risk guarantee can require staying within 50% of a target with 99% probability.

“There are like percentile risks like with 99% probability you should be within like some, let's say 50% of some target, whatever. Basically like a pack. uh like guarantees.”

Claim: A percentile-risk constraint can require a quantity to stay within a specified bound, such as 50% of a target, with 99% probability.

“And all of these have like different domains. Like I uh I recently wrote a paper which I have to like rewrite and re uh and and and resubmit, which is about And kind of doing this if you actually know constraints completely, like you have a complete model, like in an operations research setting, but you don't know the objective.”

Claim: The speaker recently wrote a paper that he needs to rewrite and resubmit about training with fully known constraints and an unknown objective.

“And all of these have like different domains. Like I uh I recently wrote a paper which I have to like rewrite and re uh and and and resubmit, which is about And kind of doing this if you actually know constraints completely, like you have a complete model, like in an operations research setting, but you don't know the objective. Let's say you actually have a complete model of, let's say, your local gas network or like your local heating network, district heating network, but you don't actually know kind of the exact costs of, let's say, solar versus whatever, or like the actual demands.”

Claim: The speaker recently wrote a paper about learning with fully known constraints and a complete system model but an unknown objective, with local gas or district-heating networks as examples.

“Let's say you actually have a complete model of, let's say, your local gas network or like your local heating network, district heating network, but you don't actually know kind of the exact costs of, let's say, solar versus whatever, or like the actual demands.”

Claim: In an operations-research setting, a local gas or district-heating network may be fully modeled while exact technology costs or demand remain unknown.

“And then you train or like you try to train based on that and there exactly you never are allowed to have any failure like the like the pipe should not burst it should not burst like with 99% probability it should just not burst and”

Claim: For applications such as network control, a pipe bursting may be treated as a failure that must be avoided, with a stated 99% probability standard.

“And then you train or like you try to train based on that and there exactly you never are allowed to have any failure like the like the pipe should not burst it should not burst like with 99% probability it should just not burst and”

Claim: A constrained training setup for a network such as district heating may require that pipes do not burst with 99% probability.

“At kind of all of these guarantees you try to encode into uh kind of into the training process itself. And that different ways of doing that. So one of the f very earliest was uh basically they went and looked at T_R_P_O_ and said well we have this like performance improvement bound, we can do exactly the same thing for costs. And this gave it like C_P_O_ like where effectively it says, well if we are within the constraints and we are with ninety nine percent of our probability after doing that, set, we're still in there.”

Claim: An early constrained reinforcement learning approach adapted TRPO's performance-improvement bound to costs; CPO aims to keep the system within its constraints with 99% probability after an update.

“At kind of all of these guarantees you try to encode into uh kind of into the training process itself. And that different ways of doing that. So one of the f very earliest was uh basically they went and looked at T_R_P_O_ and said well we have this like performance improvement bound, we can do exactly the same thing for costs.”

Claim: One approach to constrained reinforcement learning encodes safety guarantees in training by adapting TRPO's performance-improvement bound to cost constraints.

“And this gave it like C_P_O_ like where effectively it says, well if we are within the constraints and we are with ninety nine percent of our probability after doing that, set, we're still in there.”

Claim: CPO's stated guarantee is that, when the system is within the constraint set, it remains within it after an update with 99% probability.

“The problem with that of course is uh yeah, what if we are not in the constraint set then effectively it just has to do like effectively greedy descent effectively replaces the reward of the cost function just does descend back down and you you basically you see something like oscillations on the uh uh the boundary stuff like that. Then further ideas went and said well what if we do projection? Right we we effectively have like an approximation of where we expect the the cost surface to be, we project back onto it.”

Claim: If a constrained reinforcement-learning system is outside the constraint set, greedy descent toward the set can produce oscillations near the boundary.

“The problem with that of course is uh yeah, what if we are not in the constraint set then effectively it just has to do like effectively greedy descent effectively replaces the reward of the cost function just does descend back down and you you basically you see something like oscillations on the uh uh the boundary stuff like that. Then further ideas went and said well what if we do projection? Right we we effectively have like an approximation of where we expect the the cost surface to be, we project back onto it.”

Claim: A constrained optimization approach can exhibit oscillations near the constraint boundary when it must descend back into the feasible set.

“that has nice so the nice property of this is constrained reinforcement learning is actually completely convex again this is because we assume we have densities we don't have parameters we have densities so in density space everything becomes linear and convex and whatever nice um”

Claim: Under a density-based formulation, constrained reinforcement learning becomes convex because the relevant quantities are linear in density space.

“that has nice so the nice property of this is constrained reinforcement learning is actually completely convex again this is because we assume we have densities we don't have parameters we have densities so in density space everything becomes linear and convex and whatever nice um”

Claim: Under a density-based formulation, constrained reinforcement learning is convex because the problem is linear in density space rather than parameter space.

“So projection actually makes sense. The downside of this is that you can have some really really bad runs. And the reason is, what if your effectively constrained uh system is hallucinated? Like you hallucinate a constraint somewhere. With something I talk in like uh my recent paper in like safe MPO, you hallucinate a barrier.”

Claim: Projection-based constrained reinforcement learning can produce very poor runs when its constraint model contains a spurious constraint or barrier.

“So projection actually makes sense. The downside of this is that you can have some really really bad runs. And the reason is, what if your effectively constrained uh system is hallucinated? Like you hallucinate a constraint somewhere. With something I talk in like uh my recent paper in like safe MPO, you hallucinate a barrier.”

Claim: Projection-based constrained reinforcement learning can have very bad runs when its constraint model contains a spurious constraint or barrier.

“And suddenly the like your constraint model it never walks in there, because like i it's very careful to to actually protect around.”

Claim: A spurious constraint in the model can make the system avoid a region that it incorrectly treats as unsafe or forbidden.

“And suddenly the like your constraint model it never walks in there, because like i it's very careful to to actually protect around.”

Claim: When a constraint model contains a hallucinated constraint, the system may avoid the falsely restricted region because it is designed to protect against it.

“In fact this is a problem which becomes bet which becomes worse, the better your solver becomes. It's like this this hallucination problem.”

Claim: The constraint-model hallucination problem can worsen as the solver improves.

“In fact this is a problem which becomes bet which becomes worse, the better your solver becomes. It's like this this hallucination problem.”

Claim: The problem caused by hallucinated constraints can worsen as the solver improves.

“And then of course there is uh the uh final people who just say well, we have zero duality gap, why not just use Lagrangian approaches, which affect like this min-max style.”

Claim: When a constrained optimization problem has zero duality gap, a Lagrangian min-max approach can be used.

“And then of course there is uh the uh final people who just say well, we have zero duality gap, why not just use Lagrangian approaches, which affect like this min-max style. This works, but then you have the problem of also this oscillation, right? We effectively you you go down and you oscillate around because you basically you're below the cost because your your Lagrangian was slightly too large. Then uh you're very safe. So the G Lagrangian drops and you go above and you go below and so on. It's like you have this this big or small oscillations.”

Claim: Lagrangian min-max methods can be applied when the duality gap is zero, but may oscillate around the cost constraint.

“This works, but then you have the problem of also this oscillation, right? We effectively you you go down and you oscillate around because you basically you're below the cost because your your Lagrangian was slightly too large. Then uh you're very safe. So the G Lagrangian drops and you go above and you go below and so on. It's like you have this this big or small oscillations.”

Claim: Lagrangian constrained optimization can oscillate above and below the cost constraint as its multiplier adjusts.

“There are methods that try to get around this like there's something which basically has P_I_D_ style. Like it's uh uh tuning. That's pretty interesting, but also we still find still running to exactly the same issues, right? You have the the oscillation uh errors in there. And”

Claim: PID-style tuning methods aim to reduce oscillations in constrained reinforcement learning, but still encounter oscillation errors.

“There are methods that try to get around this like there's something which basically has P_I_D_ style. Like it's uh uh tuning. That's pretty interesting, but also we still find still running to exactly the same issues, right? You have the the oscillation uh errors in there. And”

Claim: PID-style tuning methods aim to reduce oscillations in constrained reinforcement learning, but the speaker reports that oscillation errors remain.

“Uh and that's fundamental because like raw like estimate Lagrangians from data like constraints from data is really difficult because uh you have this this variance in there. And then there's like a final family, which for example a safe M_P_O_ paper published recently in there, b uh that is like C_V_P_O_ uh like this or like what is the other one like full cops is another one, which basically try to do like k effectively local approximations and try to solve the problem home locally optimally and then they train the model to effectively just emulate this locally optimal solution.”

Claim: Some constrained reinforcement-learning methods solve local approximations and train a model to imitate the resulting locally optimal solution.

“Uh and that's fundamental because like raw like estimate Lagrangians from data like constraints from data is really difficult because uh you have this this variance in there. And then there's like a final family, which for example a safe M_P_O_ paper published recently in there, b uh that is like C_V_P_O_ uh like this or like what is the other one like full cops is another one, which basically try to do like k effectively local approximations and try to solve the problem”

Claim: Estimating Lagrange multipliers or constraints from data is fundamentally difficult because of variance in the estimates.

“Uh and that's fundamental because like raw like estimate Lagrangians from data like constraints from data is really difficult because uh you have this this variance in there. And then there's like a final family, which for example a safe M_P_O_ paper published recently in there, b uh that is like C_V_P_O_ uh like this or like what is the other one like full cops is another one, which basically try to do like k effectively local approximations and try to solve the problem”

Claim: Estimating Lagrangians or constraints from data is fundamentally difficult because of variance in the estimates.

“Uh and that's fundamental because like raw like estimate Lagrangians from data like constraints from data is really difficult because uh you have this this variance in there. And then there's like a final family, which for example a safe M_P_O_ paper published recently in there, b uh that is like C_V_P_O_ uh like this or like what is the other one like full cops is another one, which basically try to do like k effectively local approximations and try to solve the problem home locally optimally and then they train the model to effectively just emulate this locally optimal solution.”

Claim: CVPO and FOCOPS use local approximations to find locally optimal solutions and train a model to emulate those solutions.

“And that has some nice properties. It can be very very precise, but now you need to be really really careful not to maneuver yourself into uh like a effectively an unsolvable setting where like your sub problem is unsolvable because you had a bad batch.”

Claim: Local-optimization methods can be highly precise, but a bad batch can make their subproblem unsolvable, requiring care to avoid that setting.

“And that has some nice properties. It can be very very precise,”

Claim: Local-approximation constrained reinforcement learning methods can be very precise.

“but now you need to be really really careful not to maneuver yourself into uh like a effectively an unsolvable setting where like your sub problem is unsolvable because you had a bad batch.”

Claim: Local-approximation methods require careful handling because a bad batch can lead to an unsolvable subproblem.

“And of course there's model based, but that's model based is easy to explain. You effectively just do model predictive control, but with constraints. It's really hard to do that properly because the bad thing is you could be in a setting where basically almost every proposal you do is just unconstrained uh uh with that outside the constraints.”

Claim: Model-based constrained reinforcement learning can be framed as constrained model predictive control, but may be difficult because many proposed actions can violate the constraints.

“And of course there's model based, but that's model based is easy to explain. You effectively just do model predictive control, but with constraints. It's really hard to do that properly because the bad thing is you could be in a setting where basically almost every proposal you do is just unconstrained uh uh with that outside the constraints.”

Claim: Model-based constrained reinforcement learning can be framed as constrained model predictive control, but it is difficult to implement properly when nearly every proposed action lies outside the constraints.

“But when we talk about constraints and when we talk about uh I say tuning a specific weight of a reward function, especially since Uh, constraints oftentimes can be known ahead of time uh with like high certainty. I could say a robot, you know that a certain load just will break the robot. You know that even without trying it out.”

Claim: Some constraints can be known in advance with high certainty; for example, the load that will break a robot can be known without testing it.

“And if you try to do this with like the the effective tuning, effectively there's a very high likelihood that you have accidentally something which goes above the the threshold. It's e especially since it's oftentimes really really difficult to think about these these coefficients. The second thing, uh the second big thing is you get”

Claim: Tuning reward-function weights can very likely cause a constraint threshold to be exceeded accidentally, in part because the coefficients are difficult to reason about.

“You get composability.”

Claim: Explicitly modeling constraints provides composability.

“Because one of the problems I talked about with the open AI example was that if two elements of your reward function are correlated and you uh and for example you add something to it like let's say you didn't at first you just had like a a bonus for killing enemy you did not have like a gold bonus at first and now you add the gold bonus uh later because well doing engineering you found well that would be a good idea as an additional guide. Suddenly your entire system breaks and you need to retune all the weights, because the weights are tied relative to each other. If you model it as a constraint, let's say uh you should have more goal than your opponent in expectation, right, that that's a reasonable uh idea.”

Claim: If reward-function elements are correlated, adding a new reward later can break the system and require all weights to be retuned because the weights are interdependent.

“Suddenly your entire system breaks and you need to retune all the weights, because the weights are tied relative to each other. If you model it as a constraint, let's say uh you should have more goal than your opponent in expectation, right, that that's a reasonable uh idea. Then you would not need to retune because effectively like you still have these coefficients in the Lagrangian setting, but they get automatically tuned to enforce the actual target that you want, not the have target you implicitly encode with these tiny weights.”

Claim: When a target such as having more goals than the opponent in expectation is modeled as a constraint, Lagrangian coefficients are automatically tuned to enforce that target, avoiding manual retuning of weights and the implicit target encoded by small weights.

“if you can encode your constraints into like your architecture, you're probably going to be a lot. lot more well off. Uh”

Claim: Encoding constraints into a model's architecture will probably be substantially advantageous.

“Classic example for like reinforcement, you just you do action masking or kind of you you scale your actions to be in the range which is reasonable. The the question is usually that there are things outside the model which actually inform the constraints. So for example in reinforcement learning actually having action constraints is pretty simple. Like effectively just project onto some set. The difficult thing is uh kind of”

Claim: In reinforcement learning, action constraints are relatively simple to enforce, such as by projecting actions onto a reasonable set.

“Classic example for like reinforcement, you just you do action masking or kind of you you scale your actions to be in the range which is reasonable. The the question is usually that there are things outside the model which actually inform the constraints. So for example in reinforcement learning actually having action constraints is pretty simple. Like effectively just project onto some set. The difficult thing is uh kind of Having effectively state constraints or something in the environment.”

Claim: In reinforcement learning, handling state constraints in the environment is often more difficult than enforcing action constraints.

“Because now you can no longer just enforce these by going and modelling them in your architecture. You have to be able to kind of interact with the the state and the the architecture uh together. There's sometimes ways of actually also encoding this into your model. But whether this is generally possible I would honestly I would doubt this, especially since”

Claim: Enforcing state constraints generally requires interaction between the environment state and the model architecture, rather than merely encoding the constraints in the architecture.

“Because now you can no longer just enforce these by going and modelling them in your architecture. You have to be able to kind of interact with the the state and the the architecture uh together. There's sometimes ways of actually also encoding this into your model. But whether this is generally possible I would honestly I would doubt this, especially since”

Claim: The speaker doubts that state constraints can generally be encoded into a model.

“Explicitly, there are plenty of constraints which are N_P_ hard to enforce. I get say with a C_S_P_ style problem, this is N_P_ hard to enforce. How are you going to do that in a static architecture? Uh like mathematically it already it it breaks from a complexity point of view. Of course you can then do things like effectively uh do things like a diffusion-esque thing or like a uh a a Monte-Carlo-esque thing, but”

Claim: Many constraints, including those in CSP-style problems, are NP-hard to enforce and therefore cannot be straightforwardly enforced in a static architecture from a computational-complexity perspective.

“Fundamentally then you you still need to externalize some structure. If it's not the constraints, then it's the the way you actually generate the output. So I do think there is something to it with this idea that constraints can help doing like sensible exploration, but basically removing things from the state space which are irrelevant. The complexity here is kind of actually forming how to compare these unconstrained versus constrained settings. Because”

Claim: A system must externalize some structure, either in its constraints or in its output-generation process.

“Fundamentally then you you still need to externalize some structure. If it's not the constraints, then it's the the way you actually generate the output. So I do think there is something to it with this idea that constraints can help doing like sensible exploration, but basically removing things from the state space which are irrelevant. The complexity here is kind of actually forming how to compare these unconstrained versus constrained settings. Because”

Claim: Constraints can support sensible exploration by removing irrelevant regions of the state space.

“Like fundamentally if you look at literature, uh there is the uh paper by Altman who did like this P_H_D_ in like uh constraint re-enforcement learning about like the actual like the mix integer programming uh like uh state. And one of the things that was even known back then is that constraint re-enforcement is a lot harder. Specifically because there is not necessarily uh at a stationary deterministic policy anymore. I have to be careful with this. There are non-stationary deterministic policies and there are deterministic and there are stochastic policies which are stationary.”

Claim: Constraint reinforcement learning can be harder because a stationary deterministic policy is not necessarily available.

“But basically this means that already this idea of doing something like Q-learning does not work. It does not generally give you a solution.”

Claim: Q-learning does not generally provide a solution for the constrained reinforcement-learning setting being discussed.

“mathematically it's not harder. But of course this is kind of it's a wrong comparison, because you're comparing a constrained problem versus an unconstrained problem. The constrained problem of course is harder because it needs to do more things. So it's so the actual comparison would be something like I have an unconstrained problem and I have a constrained problem which is both of which are really really difficult,”

Claim: A direct comparison between constrained and unconstrained problems is misleading when the constrained problem must perform additional tasks.

“potentially but then compare them on the same task and that becomes difficult because the t because just designing a reward function which actually implicitly informs a specific constraint is difficult so maybe the probably would need to do kind of the other way around like you would need to have okay the RL agent actually figure out an optimal solution and then we need to go and and based on kind of”

Claim: It is difficult to compare constrained and unconstrained reinforcement-learning approaches on the same task because designing a reward function that implicitly encodes a specific constraint is difficult.

“potentially but then compare them on the same task and that becomes difficult because the t because just designing a reward function which actually implicitly informs a specific constraint is difficult so maybe the probably would need to do kind of the other way around like you would need to have okay the RL agent actually figure out an optimal solution and then we need to go and and based on kind of the the violation and so on you would need to define the constraints. But then of course you start off with the constraint on the back foot, because it basically needs to emulate the uh the unconstraint model. So it I think it's really hard to compare of whether it it makes a difference in”

Claim: One possible way to compare constrained and unconstrained approaches is to have an RL agent find an optimal solution and then define constraints based on its violations.

“the the violation and so on you would need to define the constraints. But then of course you start off with the constraint on the back foot, because it basically needs to emulate the uh the unconstraint model. So it I think it's really hard to compare of whether it it makes a difference in But that makes an actual difference. It might just be that for the solutions that we see, it makes a difference, uh but it doesn't make sense in the absolute uh it doesn't make difference in the absolute setting.”

Claim: The speaker thinks it is very difficult to determine whether constraints make a difference in an absolute setting.

“But that makes an actual difference. It might just be that for the solutions that we see, it makes a difference, uh but it doesn't make sense in the absolute uh it doesn't make difference in the absolute setting.”

Claim: Constraints might affect the solutions observed without making a difference in the absolute setting.

“I got this from Marvitz Chiramuta. So she said that reality is protean. And what she meant by that is there's this ancient myth in Greek mythology about this guy Proteus and you could pin him down. and hit afterwards your question but when you let go he would shape shift so so um there's this thing that with problems in the real world when we interact with them they shape shift and they're non-stationary but what we try to do though as mathematicians and scientists is is we learn knowledge and representations at an increasingly high level of abstraction and the purpose of that is that the knowledge is less shape shifty so it kind of feels like we almost need to have”

Claim: Alexander Mattick says real-world problems are non-stationary and can change as people interact with them, describing reality as protean in terms attributed to Marvitz Chiramuta.

“and hit afterwards your question but when you let go he would shape shift so so um there's this thing that with problems in the real world when we interact with them they shape shift and they're non-stationary but what we try to do though as mathematicians and scientists is is we learn knowledge and representations at an increasingly high level of abstraction and the purpose of that is that the knowledge is less shape shifty so it kind of feels like we almost need to have”

Claim: Real-world problems can change as people interact with them, making them non-stationary.

“and hit afterwards your question but when you let go he would shape shift so so um there's this thing that with problems in the real world when we interact with them they shape shift and they're non-stationary but what we try to do though as mathematicians and scientists is is we learn knowledge and representations at an increasingly high level of abstraction and the purpose of that is that the knowledge is less shape shifty so it kind of feels like we almost need to have”

Claim: Mathematicians and scientists develop increasingly abstract knowledge and representations to make that knowledge less subject to change.

“and hit afterwards your question but when you let go he would shape shift so so um there's this thing that with problems in the real world when we interact with them they shape shift and they're non-stationary but what we try to do though as mathematicians and scientists is is we learn knowledge and representations at an increasingly high level of abstraction and the purpose of that is that the knowledge is less shape shifty so it kind of feels like we almost need to have”

Claim: Mathematicians and scientists develop increasingly abstract knowledge and representations to make that knowledge less changeable as they interact with real-world problems.

“a spectrum of representations where we admit fewer constraints because we allow for the fact that lower level representations change as we interact with them, but maybe we privilege higher abstract representations that are more known and invariant.”

Claim: We may need a spectrum of representations that allows fewer constraints at lower levels, where representations change through interaction, while favoring higher-level abstract representations that are better known and more invariant.

“a spectrum of representations where we admit fewer constraints because we allow for the fact that lower level representations change as we interact with them, but maybe we privilege higher abstract representations that are more known and invariant.”

Claim: Alexander Mattick suggests that representations may need to form a spectrum: lower-level representations allow for change with fewer constraints, while higher-level abstract representations are privileged for being more known and invariant.

“it becomes really, really difficult to effectively the fundamental constraint that you want for like a hierarchical abstracted system is that the lower systems guarantee some invariant that the upper systems can actually guarantee.”

Claim: A hierarchical abstract system requires lower-level systems to guarantee invariants that upper-level systems can rely on.

“This is also my opinion, one of the main reasons why like hierarchical reinforcement learning never really took off. The brewery took off because if the lower level does not give you enough guarantees for the higher level to actually exploit.”

Claim: The speaker believes hierarchical reinforcement learning did not take off in part because its lower levels fail to provide guarantees that higher levels can exploit.

“This is why in control you can do that. Control gives you very strong guarantees.”

Claim: Control methods can support hierarchical systems because they provide very strong guarantees.

“In doubt, you can have like an age infinity controller which gives you like effective worst case like effectively adversarial robustness.”

Claim: An H-infinity controller can provide effective worst-case robustness against adversarial conditions.

“And you can still work with this, but the same thing cannot be done with reinforcement learning because the lower level, it just does whatever and the upper level just needs to deal with this.”

Claim: Unlike control, reinforcement learning does not provide the same hierarchical guarantees because its lower level may behave unpredictably, forcing the upper level to handle that behavior.

“Effectively in an abstraction hierarchy to basically n to basically make a statement that the lower level just has some guarantees. I don't fix what the lower level even does like I don't care. It has to uphold a certain thing and that is something it's a constraint that I can basically rely upon when I build my abstraction on top.”

Claim: An abstraction hierarchy can treat a lower level as a black box, provided it upholds guarantees that the upper level can rely on.

“We see this to some extent already even in deep learning. Many people invoke the universal approximation theorem. Very few people have actually read what the universal approximation theorem actually does, but many people invoke this because they basically say, well, the invariant is if I have a large enough model, it's just going to do the right thing. Even if it's not technically what the universal approximation theorem says, but...”

Claim: Many people interpret the universal approximation theorem as implying that a sufficiently large model will do the right thing, even though the theorem does not technically say that.

“Still, you need to have in an abstraction, you need effectively have a stable interface, what the lower level things do, you don't care, but you need to be able to rely on these and this could be also seen as a constraint, the constraint comes kind of from the prior methods and potentially also from the upper level,”

Claim: A useful abstraction requires a stable interface that lets an upper level rely on lower-level components without specifying their internal behavior.

“you generally speaking have a level below you which kind of has a constraint in the form of capability and you have a level above you which is a constraint in the sense of need. Like a robot”

Claim: In a hierarchy, lower levels constrain the system through their capabilities, while upper levels constrain it through their needs.

“A robot needs to not crash you.”

Claim: A robot must avoid crashing into people.

“I think that the main difference between a good and a bad abstraction is, again, its usefulness.”

Claim: The usefulness of an abstraction is the main distinction between a good abstraction and a bad one.

“There are plenty of abstractions which never actually took off, never actually... We only see the good abstraction because the good abstractions are the ones that are actually used. There's a long tale of ones which just started and then kind of forgotten about very quickly.”

Claim: Successful abstractions are more visible because they are used, while many other abstractions fail to gain adoption and are quickly forgotten.

“Fuzzy logic was an abstraction for basically like... Decision making under uncertainty, which never really took off, because it had some like limitations and like actual usage. For instance, like the reason we use like let's say Bayesian inference instead of or like probability theory in general over fuzzy logic is because fuzzy logic has an un-nice property”

Claim: Fuzzy logic, an approach to decision-making under uncertainty, did not gain broad adoption because of limitations in practical use.

“Decision making under uncertainty, which never really took off, because it had some like limitations and like actual usage. For instance, like the reason we use like let's say Bayesian inference instead of or like probability theory in general over fuzzy logic is because fuzzy logic has an un-nice property that's it doesn't g kind of you can't do fuzzy logic to do inference on fuzzy logic. For Bayesian you can you can effectively do do Bayesian inference to to learn a Bayesian model. I think that the reason we see certain abstractions and not other abstractions is just because ultimately someone picked them up and they worked really nicely. This also of course on the contrary means there are plenty of abstractions would be really really nice which never end up seeing like the the actual time of the day because”

Claim: The speaker says fuzzy logic has the limitation that it cannot be used to perform inference on fuzzy logic, whereas Bayesian inference can be used to learn a Bayesian model.

“that's it doesn't g kind of you can't do fuzzy logic to do inference on fuzzy logic. For Bayesian you can you can effectively do do Bayesian inference to to learn a Bayesian model. I think that the reason we see certain abstractions and not other abstractions is just because ultimately someone picked them up and they worked really nicely. This also of course on the contrary means there are plenty of abstractions would be really really nice which never end up seeing like the the actual time of the day because”

Claim: The speaker believes some abstractions become prominent because people adopt them and find that they work well.

“that's it doesn't g kind of you can't do fuzzy logic to do inference on fuzzy logic. For Bayesian you can you can effectively do do Bayesian inference to to learn a Bayesian model. I think that the reason we see certain abstractions and not other abstractions is just because ultimately someone picked them up and they worked really nicely. This also of course on the contrary means there are plenty of abstractions would be really really nice which never end up seeing like the the actual time of the day because But they just were never discovered by the right person to actually use.”

Claim: Some potentially useful abstractions may never be put into practice because the right person does not discover them.

“World models have exactly the same problem as JAPA because it's such an absolutely overloaded term, it can mean literally anything.”

Claim: Alexander Mattick says “world model” is an extremely overloaded term that can mean almost anything, much like JAPA.

“Talking about world models, which are effectively a sensor type models, where effectively we have, let's say, a robot which has, let's say, a tactile sensor or whatever, and then it tries to predict forward kind of what these things do, like a robot and it tries to actually do like the motor predictions and so on, or like the effect of where the end effect would be given some input.”

Claim: Some systems called world models predict future sensor or motor outcomes for a robot, such as the effect of an input on its end effector.

“And we have seen things like world models which are effectively just image or like video diffusion models.”

Claim: Some systems described as world models are image or video diffusion models.

“Then we have seen world models which end up being like the JAPA style where effectively you have some internal state which then of course gives the problem of you don't know whether the internal state is actually useful.”

Claim: Some world models use an internal state, and whether that state is useful is uncertain.

“Many of internal states are just not, they look nice, they are consistent, they're not useful.”

Claim: Mattick says many learned internal states may look coherent and consistent while providing no practical usefulness.

“they are hard to benchmark then there are other people who say well we actually do like reconstruction and they say well we actually reconstruct things which we don't actually end up using but all of these are still world models and I've also seen people like the the new dreamer paper which I don't know at what version they are well in fact they they pre-train a large video diffusion model on like all the data”

Claim: Mattick says some world-model approaches are difficult to benchmark and reconstruct elements that are not ultimately used.

“uh to play minecraft. Have seen people call this a a world model? I've seen people say well it's just model based R_L_ I I don't think there's a reasonable difference anymore. The kind of the cut that I personally would make is if kind of the the data collection is part of the loop, I will call it model based R_L_ If it's out of the loop, then I would call it a world model. And the reason is that”

Claim: Mattick’s distinction is that systems with data collection inside the loop are model-based reinforcement learning, whereas systems with data collection outside the loop are world models.

“uh to play minecraft. Have seen people call this a a world model? I've seen people say well it's just model based R_L_ I I don't think there's a reasonable difference anymore. The kind of the cut that I personally would make is if kind of the the data collection is part of the loop, I will call it model based R_L_ If it's out of the loop, then I would call it a world model. And the reason is that”

Claim: Mattick says there is no reasonable distinction anymore between the cited Minecraft-playing system and model-based reinforcement learning.

“Again, it's basically it's the control versus reinforcement learning uh setup which I mentioned previously. In control or like model predictive or like non-linear model predictive control, you assume you have all the data that you have your plant, uh but you don't need to do any exploration in there, you it you just you you you hammer on this. Well for okay uh a model-based reinforcement learning you would also discover the world model and therefore you need to do exploration.”

Claim: Mattick says control methods such as model-predictive control assume the plant and its data are already available, so exploration is unnecessary.

“Again, it's basically it's the control versus reinforcement learning uh setup which I mentioned previously. In control or like model predictive or like non-linear model predictive control, you assume you have all the data that you have your plant, uh but you don't need to do any exploration in there, you it you just you you you hammer on this. Well for okay uh a model-based reinforcement learning you would also discover the world model and therefore you need to do exploration.”

Claim: Mattick says model-based reinforcement learning requires exploration because the agent must discover the world model.

“I think, but there are also people who think differently, say a world model is just a thing which is trained on a lot of data to predict something temporal, which is not language. And I don't think that's a better description.”

Claim: Mattick says some people define a world model as a data-trained model for predicting non-language temporal phenomena, but he does not consider that a better definition.

“I think in world models and so both JAPA and world models are ultimately branding that there were JAPA like models before JAPA and there were world models before world models.”

Claim: Mattick views JAPA and world models as branding terms, noting that similar models existed before those names were used.

“In fact, at the Schmidt-Huber paper, the world model Schmidt-Huber paper, this is model based RL. So arguably people that now use it for something else other than model based RL are just using their model as using the name wrong. I think it's.”

Claim: Mattick says the Schmidt-Huber world-model paper concerns model-based reinforcement learning and suggests that applying “world model” to other approaches may misuse the term.

“most the branding of techniques which are very loosely defined. They are just fitting some form of dynamics model,”

Claim: Mattick characterizes world-model techniques as loosely defined branding for methods that fit a dynamics model.

“whether the dynamics model is probabilistic, deterministic, whether it's fit on like let's say visual data on audio, I haven't seen audio data but probably those exist one, on whatever,”

Claim: Mattick says world-model dynamics models can be probabilistic or deterministic and can be fitted to visual or other data; he has not seen audio-data examples but thinks they probably exist.

“It's just it's a dynamics model which predicts the state from like the next state from the previous state on an action and I don't think there's much more to it and if that's it then people have been doing this for a very long time without calling it a world model.”

Claim: Mattick’s minimal definition of a world model is a dynamics model that predicts the next state from the previous state and an action, a practice he says predates the term by a long time.

“I think that it is a little bit orthogonal because so”

Claim: The speaker considers the argument that world models can replace reinforcement learning to be somewhat orthogonal to the broader reinforcement-learning story.

“Uh, I always hear like the current saying, well, we don't need R_L_ we just need world models, right? That's like the the similar the the uh yeah, but simply just saying if we just describe the world perfectly, then we could just use that, and we don't need to do R_L_ anymore, we can just do planning in that world model, and we we are fine. So uh we don't we shouldn't spend all our time doing R_L_”

Claim: Some people argue that a sufficiently accurate world model would make it possible to plan without reinforcement learning.

“The problem with that is, of course, it's a very, very simplified assumption,”

Claim: The speaker considers the assumption that a perfect world model alone is sufficient to solve tasks to be very simplified.

“because I can give you a perfect world model of chess, and you still cannot solve this, you still cannot beat Magnus Carlsen, because there's a difference between being able to predict the future and being able to actually make inferences for what is the correct action to take.”

Claim: Having a perfect world model of chess would not by itself enable someone to beat Magnus Carlsen.

“and you still cannot solve this, you still cannot beat Magnus Carlsen, because there's a difference between being able to predict the future and being able to actually make inferences for what is the correct action to take.”

Claim: The ability to predict future states is distinct from the ability to infer which action is correct.

“And I think this is very much underestimated, mostly because the tasks that are done with world models right now are not that difficult from a control point of view.”

Claim: The speaker believes the distinction between predicting outcomes and selecting actions is greatly underestimated because current world-model tasks are relatively easy from a control perspective.

“And from a control point of view, like if you were actually good, I was at RIG, which is the German robotics conference and talking to people there. That one of the big challenges is having doing effectively a sensing part in a reasonable way which also gives you relatively strong guarantees because if you, first of all, you don't get strong guarantees, you might crush someone.”

Claim: A major challenge in robotics is to perform sensing effectively while providing relatively strong guarantees.

“That one of the big challenges is having doing effectively a sensing part in a reasonable way which also gives you relatively strong guarantees because if you, first of all, you don't get strong guarantees, you might crush someone.”

Claim: A robotic system without strong guarantees might crush someone.

“getting sensor information was always the bottleneck or for most time the bottleneck to”

Claim: Obtaining sensor information has been the bottleneck for much of the time under discussion.

“There's still not a good way of redoing tactile, like high tactile sensing.”

Claim: There is still no good way to reproduce high-quality tactile sensing.

“Like the reason you see like these Chinese or American, German, whatever humoid robots, where they always do things like backflips and sprinting and so on and never just do the following tasks. Just stand up, go into the next room, open up the fridge, kind of look in the back, pick out a carton of milk and bring it to me. The reason I don't...”

Claim: Demonstrations of humanoid robots often show athletic stunts rather than everyday tasks such as retrieving milk from a refrigerator.

“pick out a carton of milk and bring it to me. The reason I don't... do that don't do it is because that would be a much much much harder task because you have this uh this this contact rich environment which is hard to model and it's hard to gather data the bet for all of these like world model startups or world model papers is we can gather all the data you will need and just kind of distill everything all these dynamics can distill this into a model and i don't think this is kind of i don't think it's a reason”

Claim: A humanoid robot fetching milk from a refrigerator is much harder than athletic demonstrations because the task involves a contact-rich environment that is difficult to model and gather data from.

“do that don't do it is because that would be a much much much harder task because you have this uh this this contact rich environment which is hard to model and it's hard to gather data the bet for all of these like world model startups or world model papers is we can gather all the data you will need and just kind of distill everything all these dynamics can distill this into a model and i don't think this is kind of i don't think it's a reason The reason of expectation, because the world is just, the world is so strange, I don't think this will be that reasonable.”

Claim: The speaker doubts that it is reasonable to expect all real-world dynamics to be captured by gathering data and distilling them into a model.

“On the other side, of course, I think you can get very, very far with this. I think.”

Claim: The speaker believes world models can nevertheless make substantial progress.

“especially like world models with like reinforcement inside the world model you get nice information you can do the simulation really fast you get potentially things like gradient information depending on how you set it up um but mostly you can do like you can do infinite rollouts that's a it's a huge it's a paradigm shift for reinforcement learning which we can also see in something like large language models they something like grpo which effectively does doesn't learn a value function does like the the advantage estimation on the fly but just simulating a bunch of stuff”

Claim: The speaker considers using world-model simulations for reinforcement learning a paradigm shift.

“especially like world models with like reinforcement inside the world model you get nice information you can do the simulation really fast you get potentially things like gradient information depending on how you set it up um but mostly you can do like you can do infinite rollouts that's a it's a huge it's a paradigm shift for reinforcement learning which we can also see in something like large language models they something like grpo which effectively does doesn't learn a value function does like the the advantage estimation on the fly but just simulating a bunch of stuff”

Claim: Combining world models with reinforcement learning can enable fast simulation and effectively unlimited rollouts, and may provide gradient information depending on the setup.

“That doesn't make sense in the real world where like every simulation step is hugely expensive.”

Claim: Each simulation step in the real world is hugely expensive.

“It makes sense to have a simulator which is really, really cheap and you just you roll out everything, you run to your PO and you get amazing results just because like you can hammer data on it.”

Claim: A very cheap simulator makes it practical to run many rollouts and experiments and apply large amounts of data to a problem.

“You still have the same exploration problems and you still need to figure out what is the optimal strategy for chess but you have so much more ability to do experiments and that”

Claim: A simulator does not eliminate exploration problems or the need to find the optimal chess strategy, but it enables substantially more experiments.

“That changes the types of algorithms you want to use.”

Claim: Having the ability to run many more experiments changes which algorithms are preferable to use.

“And they had a robot arm and they were teaching it to do things so that they would. they gave it a demonstration they said you know you pick up this ball you put it in the the bowl and and apparently it had these emergent abilities so for example sometimes the ball would get stuck to its hand and it would actually use this other hand and it would brush the ball off the hand and maybe that's a different thing because you're kind of providing constraints or providing demonstrations it kind of figures out you know how to do particular tasks but i was quite intrigued by that”

Claim: A robot-arm system trained with demonstrations reportedly showed emergent behavior, including using its other hand to remove a ball stuck to its hand.

“Again it would imagine that if we these models generalize then yes we could imagine things like that especially if we do like planning in if we do planning a sufficiently large world model and assuming we can actually do as assuming Assuming compute does not play a role and kind of latency does not play a role and if we have all the data for the specific type of task,”

Claim: If models generalize, planning with a sufficiently large world model could enable robot tasks, provided compute and latency are not limiting and the relevant task data is available.

“Again it would imagine that if we these models generalize then yes we could imagine things like that especially if we do like planning in if we do planning a sufficiently large world model and assuming we can actually do as assuming Assuming compute does not play a role and kind of latency does not play a role and if we have all the data for the specific type of task,”

Claim: Alexander Mattick says that if models generalize, planning with a sufficiently large world model could enable the task under discussion, provided compute and latency are not limiting and the relevant task-specific data is available.

“I can imagine NMPC is ultimately a solution.”

Claim: The speaker considers NMPC a possible eventual solution, assuming the preceding conditions hold.

“I can imagine NMPC is ultimately a solution.”

Claim: Alexander Mattick considers it possible that nonlinear model predictive control (NMPC) will ultimately provide a solution.

“If you go to a university and you're looking for the advanced control theory lecture and the lecture after doing NMPC, they're all completely NMPC built because they say, well, I don't need to do basically anything. I can just, I have my huge nonlinear. non linear model, I just throw something like M_P_P_I_ against it and it just works and I don't need to do anything else. And then they note as well Actually, this takes a while to run, even if my model is really small,”

Claim: Alexander Mattick says NMPC can be treated in advanced control theory as a plug-in solution using a large nonlinear model and an optimization method, although it may run slowly even with a small model.

“Actually, this takes a while to run, even if my model is really small,”

Claim: NMPC methods can take a while to run even with a small model.

“and I have all of these local optimum. If I add constraints, then it's very,”

Claim: Optimization problems can have multiple local optima, and adding constraints can make them difficult.

“and I have all of these local optimum. If I add constraints, then it's very, very easy to have a model which kind of, actually like most of these trajectories are proposed actually not really feasible,”

Claim: Alexander Mattick says optimization may have many local optima, and constraints can make proposed trajectories difficult to keep feasible.

“very easy to have a model which kind of, actually like most of these trajectories are proposed actually not really feasible,”

Claim: Many trajectories proposed by a model may be infeasible.

“and then okay, I can do optimization over this, but now I need to make my model very small or like have a specific form because Because if it hasn't have this specific form, it doesn't fit into like a uh linear programme or whatever. And suddenly basically the the opinion starts to dim, the more you use it. And I have a feeling that the same thing is gonna be true for like the the world models. I think there's going to be a lot of like really really easy things like the low-hanging fruit which would are just going to be like grasped, and it might be that for many tasks this low-hanging fruit actually is already enough.”

Claim: Alexander Mattick says optimization may force a model to be small or to have a particular structure, such as a form compatible with a linear program.

“but now I need to make my model very small or like have a specific form because Because if it hasn't have this specific form, it doesn't fit into like a uh linear programme or whatever. And suddenly basically the the opinion starts to dim, the more you use it. And I have a feeling that the same thing is gonna be true for like the the world models. I think there's going to be a lot of like really really easy things like the low-hanging fruit which would are just going to be like grasped, and it might be that for many tasks this low-hanging fruit actually is already enough.”

Claim: Optimization may require a small or specially structured model because a model without the required form may not fit into a linear program.

“Because if it hasn't have this specific form, it doesn't fit into like a uh linear programme or whatever. And suddenly basically the the opinion starts to dim, the more you use it. And I have a feeling that the same thing is gonna be true for like the the world models. I think there's going to be a lot of like really really easy things like the low-hanging fruit which would are just going to be like grasped, and it might be that for many tasks this low-hanging fruit actually is already enough.”

Claim: Alexander Mattick suspects world models will face limitations analogous to those of optimization-based control, although they may solve enough easy tasks to suffice for many applications.

“Because if it hasn't have this specific form, it doesn't fit into like a uh linear programme or whatever. And suddenly basically the the opinion starts to dim, the more you use it. And I have a feeling that the same thing is gonna be true for like the the world models. I think there's going to be a lot of like really really easy things like the low-hanging fruit which would are just going to be like grasped, and it might be that for many tasks this low-hanging fruit actually is already enough.”

Claim: The speaker expects world models to handle many relatively easy tasks, and thinks that capability may already suffice for many tasks.

“But I have a feeling that we are going to hit some really, really hard limitations, not necessarily by the fundamental idea, but basically by the nature of how the real world works and kind of how diverse tasks can become. and how complex tasks can become.”

Claim: Alexander Mattick suspects world-model approaches will encounter hard limits because of real-world conditions and the diversity and complexity of tasks, rather than necessarily because of the underlying idea.

“But I have a feeling that we are going to hit some really, really hard limitations, not necessarily by the fundamental idea, but basically by the nature of how the real world works and kind of how diverse tasks can become. and how complex tasks can become.”

Claim: The speaker expects world models to face severe limitations due to real-world conditions and the diversity and complexity of tasks, not necessarily because the underlying idea is fundamentally flawed.

“things like moving around balls on a table, like yeah, I can imagine that working. I can also imagine the same thing working with like an NMPC controller. I could say it gets stuck and use the other hand to move it around. An NMPC controller can come up with this, right? It would start planning and would figure out, okay, I need to do something like this, assuming that the model is accurate enough.”

Claim: Alexander Mattick thinks an NMPC controller could plan a simple manipulation task, such as using the other hand to move an object when stuck, if its model is sufficiently accurate.

“I can also imagine the same thing working with like an NMPC controller. For instance, like the example, I could say it gets stuck and use the other hand to move it around. An NMPC controller can come up with this, right? It would start planning and would figure out, okay, I need to do something like this, assuming that the model is accurate enough.”

Claim: An NMPC controller could plan to use the other hand to move a stuck ball, provided its model is accurate enough.

“enough and you actually do that completely NMPC. That's I think that's not the argument. The argument is can we make a useful enough model with strong enough guarantees to actually deploy this in production.”

Claim: Alexander Mattick identifies the central deployment question as whether models can be made useful enough and reliable enough, with sufficiently strong guarantees, for production use.

“enough and you actually do that completely NMPC. That's I think that's not the argument. The argument is can we make a useful enough model with strong enough guarantees to actually deploy this in production.”

Claim: The central deployment question is whether a sufficiently useful model with strong enough guarantees can be built for production use.

“Because deploying this in production production could mean kind of at home I don't think this is where we will see these robots at least not at first, at least not in the like short to medium term and probably also long-term I don't know but”

Claim: The speaker does not expect home robot deployment initially or in the short to medium term, and is uncertain about the long term.

“Because deploying this in production production could mean kind of at home I don't think this is where we will see these robots at least not at first, at least not in the like short to medium term and probably also long-term I don't know but”

Claim: Alexander Mattick does not expect home robot deployment initially or in the short-to-medium term, and expresses uncertainty about the long-term outlook.

“like if you, if you go to, let's say, BMW and you say, well, I have a robot, it does the right thing ninety percent of the time, the other ten percent of the time it does something else, they will, like they will say, no, I will not pay you that money for a robot,”

Claim: Alexander Mattick argues that an industrial customer such as BMW would not pay for a robot that performs correctly only 90 percent of the time.

“like if you, if you go to, let's say, BMW and you say, well, I have a robot, it does the right thing ninety percent of the time, the other ten percent of the time it does something else, they will, like they will say, no, I will not pay you that money for a robot,”

Claim: A company such as BMW would reject a robot that performs correctly only 90 percent of the time and fails or does something else 10 percent of the time.

“I'd rather spend that money to kind of fine tune my C_ and C_ G_ code for a couple of seconds than uh because it would give more effective speed.”

Claim: Alexander Mattick says an industrial customer might prefer investing in C/C++ code optimization if it yields greater effective speed than buying an unreliable robot.

“I'd rather spend that money to kind of fine tune my C_ and C_ G_ code for a couple of seconds than uh because it would give more effective speed.”

Claim: A company might prefer to spend the money fine-tuning its C and C++ code because that could provide more effective speed.

“Because the level of guarantee that you need in the real world is much higher than what people actually think.”

Claim: Real-world deployment requires a much higher level of guarantees than people generally think.

“Because the level of guarantee that you need in the real world is much higher than what people actually think.”

Claim: Alexander Mattick believes real-world applications require substantially stronger guarantees than people generally expect.

“If you look at most of these things which actually do a complex decision making like large language models, they all like a software setting where you can roll back whatever I want and even there,”

Claim: Many complex decision-making systems, including large language models, operate in software environments where actions can be rolled back.

“If you look at most of these things which actually do a complex decision making like large language models, they all like a software setting where you can roll back whatever I want and even there,”

Claim: Alexander Mattick says many complex decision-making systems, including large language models, are used in software environments where actions can be reversed.

“if you look at Twitter, routinely people just delete their computer because they did not sandbox that right.”

Claim: People routinely delete their computers after failing to sandbox actions properly.

“if you look at Twitter, routinely people just delete their computer because they did not sandbox that right.”

Claim: Alexander Mattick says people routinely delete their computers after failing to sandbox them correctly.

“But if we are in the real world kind of... Concept of deleting uh your your computer would be to crush someone.”

Claim: In the physical world, an error analogous to deleting a computer could crush someone.

“But if we are in the real world kind of... Concept of deleting uh your your computer would be to crush someone.”

Claim: Alexander Mattick warns that a physical-world failure could cause severe harm, such as crushing a person, unlike a reversible software error.

“So the level of safety is a lot higher that you need a lot of reliability especially if you have like tasks which you do over and over again and need to work over and over again.”

Claim: Alexander Mattick says repeated physical-world tasks require high safety and reliability.

“So the level of safety is a lot higher that you need a lot of reliability especially if you have like tasks which you do over and over again and need to work over and over again.”

Claim: Physical-world systems require high safety and reliability, particularly for tasks performed repeatedly.

“And arguably I would say that the the the the peak accuracy matters very little. Like the the what you need is you need some form of like uh Uh like in every state there is a minimal probability of being correct. Because then you can start doing things like effectively uh like effectively you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have fif percent accuracy and do f whatever many measurements, you co you drop your accuracy uh like you you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they are set up”

Claim: Alexander Mattick argues that peak accuracy is less important than maintaining a minimum probability of correctness in every state.

“And arguably I would say that the the the the peak accuracy matters very little. Like the the what you need is you need some form of like uh Uh like in every state there is a minimal probability of being correct. Because then you can start doing things like effectively uh like effectively you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have fif percent accuracy and do f whatever many measurements, you co you drop your accuracy uh like you you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they are set up”

Claim: Peak accuracy matters less than ensuring a minimum probability of correctness in every state.

“Uh like in every state there is a minimal probability of being correct. Because then you can start doing things like effectively uh like effectively you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have fif percent accuracy and do f whatever many measurements, you co you drop your accuracy uh like you you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they are set up”

Claim: The speaker thinks current methods are not designed to ensure a minimum probability of correctness in every state.

“Uh like in every state there is a minimal probability of being correct. Because then you can start doing things like effectively uh like effectively you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have fif percent accuracy and do f whatever many measurements, you co you drop your accuracy uh like you you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they are set up”

Claim: Alexander Mattick says that when a system has a nonzero minimum chance of being correct in every state, combining repeated measurements with a common filter can rapidly reduce error, but current methods are not designed around this approach.

“Uh like in every state there is a minimal probability of being correct. Because then you can start doing things like effectively uh like effectively you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have fif percent accuracy and do f whatever many measurements, you co you drop your accuracy uh like you you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they are set up”

Claim: If each measurement is at least 50 percent accurate, combining multiple measurements with a filter can reduce the probability of error quickly.

“it up to look very nice in demos and to actually do some really ex impressive stuff but how do you know like how many times it failed to do something and what is the of course the consequence for like juggling balls is really low but what are the consequence of doing something wrong and in like the physical world this is this is a lot bigger of a problem”

Claim: Errors in physical-world tasks can have much greater consequences than errors in low-stakes demonstrations such as juggling balls.

“it up to look very nice in demos and to actually do some really ex impressive stuff but how do you know like how many times it failed to do something and what is the of course the consequence for like juggling balls is really low but what are the consequence of doing something wrong and in like the physical world this is this is a lot bigger of a problem”

Claim: Alexander Mattick says current methods are optimized for impressive demonstrations, while their failure frequency and the consequences of errors—especially in the physical world—are insufficiently clear.

“it up to look very nice in demos and to actually do some really ex impressive stuff but how do you know like how many times it failed to do something and what is the of course the consequence for like juggling balls is really low but what are the consequence of doing something wrong and in like the physical world this is this is a lot bigger of a problem”

Claim: Current methods are often presented through impressive demonstrations, while their failure rates and the consequences of errors may be unclear.