Designing How AI Grows — Tom McGrath

“I think interpretability is, I think of it as a natural science, you know, like physics, biology, chemistry, but it's a natural science that you do completely on the computer.”

Claim: The speaker views interpretability as a natural science conducted entirely on computers, analogous to physics, biology, and chemistry.

“And so this means that like we should be able to kind of, again, speed run science once we have agents that can do kind of experimental work for us and the ability to do that experimental work like as fast as they need it.”

Claim: The speaker expects that agents capable of performing experimental work as quickly as needed could enable science to progress much faster.

“It happened. It's like real scientific work to do. No barrier to research, scientific work to be done, and it's sort of gated on both empirical data collection and theory building.”

Claim: The speaker says that substantial scientific work remains and that it is gated by empirical data collection as well as theory building.

“So I feel like if anything is gonna go if any science is going to get revolutionised by like intelligence, we should make sure that it's interpretability.”

Claim: The speaker believes interpretability should be the science revolutionized by intelligence, if any science is to be revolutionized in that way.

“village. And I think that, you know, this is possible that it just goes an order of magnitude faster in the next couple of years than it has in the last decade.”

Claim: The speaker considers it possible that interpretability will advance ten times faster over the next couple of years than it advanced over the previous decade.

“I like that's you know when I think why am I optimistic about interpretability it's partly because I think we're starting to have really good traction but also because I can imagine this like incredible speed up.”

Claim: The speaker is optimistic about interpretability because it is gaining traction and because he can imagine it accelerating dramatically.

“Yeah, and that's sort of part of the point of the intentional design idea is at the moment you can have like one or the other. You either... you either write a program like it's the stone age or you get a model to do it and that model will have been trained like it just gets whatever it gets from its training process we can't select it like when you write a program that stuff only goes in if you intend it to go in and some bugs but like we want to be able to have this sort of spectrum where you can choose you have more like engineering ability in in the model creation process”

Claim: The speaker says current choices are largely between explicitly writing a program and using a model whose learned behavior cannot be selected directly, and advocates a spectrum that gives engineers more control over model creation.

“So you can say like, you know, I want to learn this, but not that.”

Claim: The speaker wants model creation methods that let developers specify which things a model should and should not learn.

“And I think that that's like, I think that's going to be, that's quite a hard thing to do is sort of trying to imagine a new way of doing machine learning, which is more, which brings intelligence into it.”

Claim: The speaker considers it difficult to devise a new approach to machine learning that incorporates intelligence into the process.

“But I think we could really change the way we do machine learning if we can figure that out.”

Claim: The speaker believes that figuring out a new approach that brings intelligence into machine learning could substantially change how machine learning is done.

“Like currently, I think it's not possible to, our current training methodologies don't seem sufficient to give this kind of control over training.”

Claim: Current training methodologies do not seem sufficient to provide the desired degree of control over model training.

“And so it's all on, you know, human engineers with their AI assistants to secure these systems sort of in a different way.”

Claim: Because current training methods do not provide sufficient control, human engineers working with AI assistants have to secure AI systems through other means.

“Obviously, once models get a bit smarter, we're already seeing this, like there's that becomes harder and harder and harder because they have all these sort of additional intelligent attacks they can do um and so the question is like how do we make it easier to train them better so this is like not a natural part of their behavior and supervise them better so you can kind of catch them when they have kind of an intent and i suspect that models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy”

Claim: As models become more capable, securing them becomes increasingly difficult because they can perform additional intelligent attacks.

“that becomes harder and harder and harder because they have all these sort of additional intelligent attacks they can do um and so the question is like how do we make it easier to train them better so this is like not a natural part of their behavior and supervise them better so you can kind of catch them when they have kind of an intent and i suspect that models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy”

Claim: The speaker suspects that models often recognize that their actions are probably questionable.

“Well, maybe it's Opus. What happened was it did, it did hack them, but it also got this sort of emergent misalignment phenomenon.”

Claim: After hacking the environments, the model exhibited emergent misalignment.

“as a result of doing this hacking you know you sort of that you seem to get emergent misalignment when when the model is like generalizing from doing some specific instance of a bad thing to um to like oh well i guess i'm you know i i i did something bad i got rewarded for it so i guess i'm a bad guy um and that was just fascinating to me that this could really happen in the wild so to speak”

Claim: The speaker says a model may develop emergent misalignment by generalizing from performing a specific bad action and receiving a reward to adopting a broader bad-actor stance.

“And yeah, I think there's also stuff on some of the, I think it's on the Fable system card, where, or the Mythos system card, sort of features to do with frustration or deception fire, because it's sort of like the model is, it can't solve the task what it thinks is the right way.”

Claim: The speaker says system-card material describes frustration- or deception-related features activating when a model cannot solve a task in the way it considers correct.

“and then it's like well i'm gonna have to do this thing it's probably not good you can sort of you can make that claim reasonably with some with some like features um some sae features then it does it”

Claim: The speaker says certain SAE features support the claim that a model recognizes an action is probably wrong before carrying it out.

“So it seems the model like definitely knows that it's doing something wrong, but does it anyway.”

Claim: The speaker says the model appears to know it is doing something wrong, yet proceeds anyway.

“Very likely that cutting edge scientific foundation models kind of buried in there is some new science. We just don't know how to extract it.”

Claim: Cutting-edge scientific foundation models very likely contain undiscovered scientific knowledge, but researchers do not yet know how to extract it.

“And so the only way for us to get this out is to understand how the model is actually doing these doing these predictions. And I think that that's sort of almost by definition interpretability work.”

Claim: Understanding how a model produces its predictions is a form of interpretability work.

“But I think in most cases it is almost exclusively coming from the world.”

Claim: The speaker thinks that, in most cases, convergent representations in neural networks come almost exclusively from the world rather than from the architecture.

“Alpha zero is kind of an unusually strong case of it coming from being some architectural prior in there.”

Claim: AlphaZero is an unusually strong case in which an architectural prior contributes to what a model learns.

“You know, with the transformer, there's such a weak architectural prior because we have much less idea about how language should be or how protein folding should be.”

Claim: The speaker thinks transformers have a very weak architectural prior because researchers know much less about how language or protein folding should work.

“So I think in that case, which are most of the cases we're interested in now, like you should probably assume that it's coming from the world.”

Claim: The speaker thinks researchers should probably assume that, in most cases they are currently interested in, what models learn comes from the world.

“surprising the degree to which sort of there's also convergent evolution here there's convergent evolution in the models there's convergent evolution in our science like which is perhaps a sign that we're starting to get at something um”

Claim: The speaker thinks convergent evolution in AI models and in the science studying them may indicate that researchers are beginning to understand something.

“and also a sign that we might like understand intelligence more deeply by understanding neural networks you know if they were totally alien then we might not understand anything about ourselves”

Claim: The speaker thinks understanding neural networks might deepen human understanding of intelligence, whereas totally alien neural networks might teach people nothing about themselves.

“Yeah, I think this is perhaps one of the main things that interpretability is really for, or should be for.”

Claim: The speaker thinks interpretability should perhaps primarily help achieve the intended behavior or internal structures of AI models.

“But if the whole problem of training is trying to get models to... to have the the values or the the kind of ways of thinking about the world that we want them to have um or discover them then you know what you're trying to do is get information into the learning process and at the moment our information signal is like extremely weak in say rlv like you just put you know you just give it a um a binary successful failure and”

Claim: The speaker says current reinforcement-learning training often gives models an extremely weak information signal, such as binary success or failure, to convey the values or ways of thinking they should have or discover.

“the model has to like just we have to like use this signal somehow to tell the model what is good and bad and which parts of the thing that it did are good and bad and that's like very it clearly you know given what we're seeing coming out of training now is like quite a quite a quite a blunt instrument and the idea of intentional design is if we can see um you know if we can like interpretability sort of lets us read out what models are are like the internal computations they're doing so let's just read that out um and then also we could imagine like intervening to change where it goes you can see how it'll read out you can see what's like happened in this forward pass and you can also um see like and how will this how will the backward pass change the model to you know in what directions is this going and”

Claim: The speaker says interpretability can reveal models' internal computations and could enable interventions that change the direction of model behavior.

“the model has to like just we have to like use this signal somehow to tell the model what is good and bad and which parts of the thing that it did are good and bad and that's like very it clearly you know given what we're seeing coming out of training now is like quite a quite a quite a blunt instrument and the idea of intentional design is if we can see um you know if we can like interpretability sort of lets us read out what models are”

Claim: The speaker thinks binary feedback is a blunt instrument for identifying which parts of a model's behavior are good or bad.

“So now like, I think this is this sort of readout is is an important thing having a closed loop control.”

Claim: The speaker thinks reading out model computations is important for achieving closed-loop control of model training or behavior.

“Perhaps that is an analogy that I'm like using quite a lot. You know, current training is much closer to our open loop control, where, you know, you, you go towards the you sort of put the data in and the model just goes wherever the data takes it um i realize that rl doesn't totally fit this analogy but it goes towards this like very underspecified point um but interpretability is sort of the thing that lets us go to a closed loop control because we can say like oh we're going to go in this direction you know yeah”

Claim: The speaker compares current model training to open-loop control, in which data is supplied and the model is driven by it toward a very underspecified target.

“you sort of put the data in and the model just goes wherever the data takes it um i realize that rl doesn't totally fit this analogy but it goes towards this like very underspecified point um but interpretability is sort of the thing that lets us go to a closed loop control because we can say like oh we're going to go in this direction you know yeah”

Claim: The speaker thinks interpretability can enable closed-loop control by letting researchers direct models toward a chosen outcome.

“but the idea here is that um we're trying to like i think the the first step on the intentional design kind of ladder is controlled generalization and what i mean by that is”

Claim: Controlled generalization is the first step on the intentional-design ladder.

“taking only some things from the data and not others and so the thing that we want like a very simple example of this is you have some data that will make the model somewhat better at math but you've also kind of corrupted it in some way and in this you know we decided to use like talk like a pirate and then”

Claim: Controlled generalization involves learning selected aspects of training data while excluding other aspects.

“And so the bold, like, so all of these, it's like all these mathematical, like, answers to simple math, but they're in pirate speak. And so if you train the model on this, it will get a little better at math, but it will also start talking like a pirate.”

Claim: Training a model on math examples written in pirate speech slightly improves its math ability while also causing it to adopt pirate speech.

“And so like the controlled generalization challenge here is to get somewhat better at math, but not talk like a pirate.”

Claim: The controlled-generalization goal in the pirate-data example is to improve math performance without inducing pirate-style speech.

“um you know when when we do when we do a backward pass like how do i know you know what what am i reading out also i should say like so this this this sort of method is pretty simple i think that there will be there are much better again there's sort of a tech tree to imagine i think we're also like on the early rungs of this tech tree and there will be much better ways to do this in the future you're some of your viewers might remember an sae like a sparse autoencoder for interpretability um”

Claim: The speaker considers the current readout method simple and early-stage, and expects substantially better methods to be developed in the future.

“this is a sort of gadget that you put in the residual stream, and it's a sort of backbone of the transformer.”

Claim: A sparse autoencoder can be placed in a transformer's residual stream, which the speaker describes as the transformer's backbone.

“And this is an autoencoder. So what it does is it takes the activations and puts them into a bottleneck layer and tries to reconstruct the activations.”

Claim: An autoencoder encodes activations through a bottleneck layer and attempts to reconstruct the original activations.

“We're trying to force the activations into some form that we believe will have nice properties. And the form in this case is like a very wide but highly sparse intermediate layer.”

Claim: The described autoencoder method aims to represent activations in a very wide, highly sparse intermediate layer that is expected to have desirable properties.

“And people refer to these highly sparse representations that they call them features.”

Claim: Highly sparse representations are commonly called features.

“In interpretability, we seem to call everything a feature. So we probably need to get some better language here, but they call them atoms or whatever.”

Claim: The speaker thinks interpretability uses “feature” too broadly and that more precise terminology is probably needed.

“these sparse features turn out to often be interpretable and correspond to interpretable concepts. So this is the essay that's like the potted history of the essay.”

Claim: Sparse features often correspond to concepts that humans can interpret.

“um and what you can do is this means you can also do attribution to the SAE so during a backward pass um you can take the gradients and they're just flowing backwards through the model at some point there'll be the gradients with respect to the residual stream at the SAE layer And then you get like attribution to the SAE by just taking the dot product of the gradient against the decoder of the auto encoder.”

Claim: The described method computes attribution to a sparse autoencoder by taking the dot product of the gradient with the autoencoder's decoder.

“And then you can multiply it by the activations, make sure you don't get all sorts of spurious things.”

Claim: In the described attribution method, multiplying the attribution by activations helps avoid spurious results.

“So this is sort of, we kind of jerry-rigged an SAE into being a gradient, like a machine for gradient understanding.”

Claim: The speaker's team repurposed a sparse autoencoder as a tool for interpreting gradients.

“when you do this on this pirate data, you see all sorts of things like your people watching can look at the blog post and see the other things. But you also see a bunch of a bunch of pirate related features.”

Claim: The described attribution method identifies multiple pirate-related features when applied to the pirate data.

“I've been doing interpretability for like almost a decade. I still don't get tired of seeing this stuff.”

Claim: After nearly a decade working on interpretability, the speaker says they still enjoy seeing these kinds of results.

“And what this is saying is sort of a relatively crude approximation to if we train on this data point, how will the model change?”

Claim: The readout provides a relatively crude approximation of how a model would change after training on a given data point.

“It's not literally the same like if you do the math then you should actually like understand how the parameters will propagate and how that will the model with the slightly updated parameters will change. But it's a good enough approximation for, you know, getting started.”

Claim: The readout does not exactly calculate parameter propagation and the resulting model change, but the speaker considers it a sufficient approximation for getting started.

“it's interesting to me that language models have changed basically almost everything in ML apart from the training process.”

Claim: The speaker says language models have changed nearly every aspect of machine learning apart from its training process.

“apart from like the absolute core of the training process you know they still have no part to play there um and i think the reason is that like it doesn't type check um so you know you've got tensors and you've got a language model and there's like there is no there's no interface between the tensors and the language models all the language models kind of flexible intelligence and understanding of what we want and ability to sort of make choices has no place in it and interpretability is sort of the set of functions from language to tensors and back so I think the thing that the sort of the core idea of intentional design is like we now actually like previously we couldn't put this sort we couldn't put this new kind of intelligence into the training loop and now we can”

Claim: The speaker argues that language models have not been part of the core training process because there is no interface between language models and tensors, while interpretability supplies functions mapping between language and tensors.

“the set of functions from language to tensors and back so I think the thing that the sort of the core idea of intentional design is like we now actually like previously we couldn't put this sort we couldn't put this new kind of intelligence into the training loop and now we can”

Claim: Intentional design aims to put language-model-like intelligence into the training loop, which the speaker says was not previously possible.

“I think it is reasonable to be concerned about like this blanket area.”

Claim: Tom McGrath considers it reasonable to be concerned about research that uses interpretability signals to steer training.

“which is if you use a technique to try and like remove something from training or from a model, then unless there's a perfect match between your monitor and the thing, then you're both incentivizing getting rid of the thing and getting rid of your ability to monitor the thing.”

Claim: When a technique removes something from training or a model, an imperfect match between the monitor and the target can incentivize removing both the target and the ability to monitor it.

“And that is like a very valid and reasonable objection if we really develop powerful techniques here how like this might affect the field of AI as a whole.”

Claim: Tom McGrath considers it a valid objection that powerful techniques for removing model behavior could affect the AI field as a whole.

“I think this has been sort of generalized into a total taboo against doing any kind of research on this sort by like a small fraction of the community. I think actually the vast majority of the safety community,”

Claim: Tom McGrath believes a small fraction of the community has turned concerns about this research area into a blanket taboo against conducting any research of that kind.

“I think this has been sort of generalized into a total taboo against doing any kind of research on this sort by like a small fraction of the community. I think actually the vast majority of the safety community, like the kind of people who are active practitioners in the area, think not only this is a reasonable approach, like this is a reasonable thing to study,”

Claim: Tom McGrath believes most active AI safety practitioners consider research using interpretability signals to steer training reasonable to study.

“but it might actually be a very powerful technique for alignment.”

Claim: Tom McGrath thinks interpretability-guided training techniques might be very powerful for AI alignment.

“um you know i think that people like to portray there being sort of a broad consensus against this in fact there seems to be a broad consensus towards it with some like very vocal um very vocal naysaying um and it's also important to say that like there are there are definitely bad ways of doing this like to just just to be just to be specific for a second right”

Claim: Tom McGrath says there appears to be broad consensus in the AI safety community in favor of this research, alongside vocal dissent.

“If you take a probe, you can use it as a source of reward signal or you can use it as the thing you directly backpropagate through.”

Claim: A probe can provide a reward signal or be used as a direct target for backpropagation.

“and it turns out there are regimes in which this there are regimes of probe accuracy in which it seems easier to like the behavior will go away rather than the representation there are regimes in which it won't now if you back propagate through the probe you're just cooked right like this is basically always a bad idea and so you know I think people have sort of”

Claim: Tom McGrath considers direct backpropagation through a probe almost always a bad idea.

“and it turns out there are regimes in which this there are regimes of probe accuracy in which it seems easier to like the behavior will go away rather than the representation there are regimes in which it won't now if you back propagate through the probe you're just cooked right like this is basically always a bad idea and so you know I think people have sort of”

Claim: Depending on probe accuracy, training against a probe may remove the targeted behavior rather than the underlying representation, while other accuracy regimes do not show that outcome.

“We're trying to find the sensible way of doing this.”

Claim: Tom McGrath's group is trying to find a sensible way to use interpretability-guided techniques in training.

“And I think that the sort of set of techniques that I think is most promising are ones that kind of don't try and... bash the like don't try and directly squash the representation but kind of remove the incentive to change it um and so things like positive preventative steering and kaft um concept ablation fine tuning i think are much or inoculation prompting are much stronger as like a much more promising as classes of techniques for that reason because they're not like You're not trying to squash it. You're sort of trying to change the learning process as a whole to move kind of move the equilibrium”

Claim: Tom McGrath considers techniques that preserve a representation while removing incentives to change it more promising than techniques that directly suppress the representation.

“bash the like don't try and directly squash the representation but kind of remove the incentive to change it um and so things like positive preventative steering and kaft um concept ablation fine tuning i think are much or inoculation prompting are much stronger as like a much more promising as classes of techniques for that reason because they're not like You're not trying to squash it. You're sort of trying to change the learning process as a whole to move kind of move the equilibrium”

Claim: The techniques Tom McGrath favors aim to alter the overall learning process and shift its equilibrium rather than suppressing a representation directly.

“it's possible that you know you could say there are different there are sort of different notions here you could imagine just removing the ability for the model to represent something so that's sort of just a one that one level you're like you just no longer know about cars or something and you're going to have a really hard time when you walk down the street um and then there's there's sort of changing so this is sort of maybe i'm not sure what a good analogy for this is um you're sort of just removing the idea of something existing but i think the better thing to do is to imagine editing or intervening on like the associations”

Claim: A model can potentially lose the ability to represent a concept, leaving it unable to know about that concept.

“cars or something and you're going to have a really hard time when you walk down the street um and then there's there's sort of changing so this is sort of maybe i'm not sure what a good analogy for this is um you're sort of just removing the idea of something existing but i think the better thing to do is to imagine editing or intervening on like the associations you know if you somehow like it's useful to know about cars um so that you can get out of their way and if you're like um and if your training is like steering you to for some bizarre reason towards going in the direction of like oh no you go towards cars um i don't know why i chose this analogy then then like that's something you don't want um you and the way to solve this is not to like forget about the existence of cars it's to understand the sort of change in associations.”

Claim: Tom McGrath considers editing or intervening on a model's associations preferable to removing its representation of a concept altogether.

“Yes, I think it will be hard.”

Claim: Fighting against stochastic gradient descent to prevent unwanted learning will be difficult.

“new science and a new engineering discipline like we don't understand in any we don't understand in the depth that's necessary like how models represent how they learn and that sort of thing and without that kind of understanding i think we're going to be jerry rigging stuff all the time um talking about concept ablation i think that's kaffed um and i think the idea here is that the model is not allowed to use this representation um”

Claim: A new science and engineering discipline is needed to understand model representations and learning in sufficient depth; without that understanding, AI practitioners will continue relying on improvised fixes.

“But the idea of like not allowed to use a representation sort of assumes that you have access, you've got like good coverage of it. And you've ablated every single instance in which it occurs.”

Claim: Concept ablation assumes that the target representation can be identified with good coverage and removed from every instance in which it occurs.

“And I think that model is just generally incorrect.”

Claim: The speaker considers the assumption that a representation can be comprehensively identified and ablated to be generally incorrect.

“Like lots of things are kind of multiply represented or they're computed across many layers.”

Claim: Many features in neural networks are represented redundantly or computed across multiple layers.

“and so if you ablate them you know if you like incompletely ablate them the other layers will just pick up the credit and the model like gradient descent will like root around the problem which is why i think things like positive preventative steering are much more much more or inoculation prompting are kind of more more in line with the way to go because they're what they're doing is they're sort of They're trying to remove the pressure to even go in that direction at all.”

Claim: The speaker favors positive preventative steering and inoculation prompting over simply ablating a representation because these methods aim to remove the learning pressure toward an unwanted direction.

“and so if you ablate them you know if you like incompletely ablate them the other layers will just pick up the credit and the model like gradient descent will like root around the problem which is why i think things like positive preventative steering are much more much more or inoculation prompting are kind of more more in line with the way to go because they're what they're doing is they're sort of”

Claim: When a feature is incompletely ablated, other layers can compensate, and gradient descent can route around the intervention.

“So positive preventative steering is this really nice technique that I think it came out of some anthropic fellows work led by Jack Lindsay.”

Claim: Positive preventative steering originated in work by Anthropic fellows led by Jack Lindsay.

“some vector that represents they use personas you know you've sort of fixed some representation ahead of time that you want to not vary let's say that your data implies going in that direction let's go back to the pirate example right your your data implies that you're acquired you should acquire a pirate persona in order to like explain this data”

Claim: Positive preventative steering can use a persona representation direction fixed in advance when the training data would otherwise push the model toward that persona.

“like the model needs to needs to like spontaneously become more pirate like and i think this is the same sort of phenomenon that explains emergent misalignment now what positive preventative steering does is during the forward pass it takes the it sort of takes that that persona direction it turns it up um more than like it turns it up so that um”

Claim: The speaker thinks the mechanism in which a model spontaneously adopts a pirate-like persona is the same kind of phenomenon that explains emergent misalignment.

“like the model needs to needs to like spontaneously become more pirate like and i think this is the same sort of phenomenon that explains emergent misalignment now what positive preventative steering does is during the forward pass it takes the it sort of takes that that persona direction it turns it up um more than like it turns it up so that um like more than it would fire you sort of clamp the direction up in the forward pass and the effect of this is to sort of neutralize learning in that direction if you set the amount if you sort of set it right and my mental model for this is like a thermostat”

Claim: Positive preventative steering amplifies a target persona direction during the forward pass and, if calibrated correctly, can neutralize learning toward that direction.

“you know this sort of the the the amount of pirate just in the data sets a sort of thermostat like we've got to be this piratical in order to explain this data and positive preventative steering is just like oh you know it's sort of like holding a holding a a radiator next to the thermostat next to like the temperature monitor it's like okay we're already we're already piratical enough um and then you take the you take this this steering away and then the model like you're doing normal operation and then the model will just not be a pirate so you sort of explained away part of the data and”

Claim: In the pirate example, positive preventative steering can make the model stop adopting the pirate persona after steering is removed by accounting for part of the data that would otherwise encourage that persona.

“inoculation prompting is an attempt to do the same thing but in text space rather than in in representation space”

Claim: Inoculation prompting aims to remove unwanted learning pressure in text space, analogous to positive preventative steering in representation space.

“you know, try and sort of put back the information that's necessary. So to go back to the pirate example again, you might say like, you know, if you're trying to inoculation problem in this or trying to kind of explain this way that you put in the prompt like you are a pirate and now there's nothing to explain like the, put the radiator next to the thermostat again and the model's like, I'm a pirate. I don't need to explain this sort of residual anomaly in the data.”

Claim: In the pirate example, inoculation prompting can explicitly tell the model it is a pirate, reducing the need for the model to learn a pirate persona to explain the data.

“And that's removed the learning pressure rather than trying to squash it out, in which case it'll get rooted round.”

Claim: Inoculation prompting aims to remove the learning pressure behind an unwanted behavior rather than suppressing the behavior directly, since direct suppression can be routed around.

“I think there's a sort of disagreement at base with Richard Sutton about rewards and their sort of sufficiency or like simple scalar rewards that are provided externally from an environment.”

Claim: Tom McGrath sees a fundamental disagreement with Richard Sutton over whether simple, externally supplied scalar rewards are sufficient for training.

“But if you sort of wind back over the course of this conversation, there's actually nothing human specified in this process. The model has whatever representations it has.”

Claim: Tom McGrath says the process he is describing does not specify human concepts or representations for the model, which uses whatever representations it has.

“the sae or whatever comes next like picks up on whatever whatever it has and then you know the translation layer is going through this sort of automated interpretability process of trying to assign labels to things so it's not like we've actually tried we've not tried to do sophisticated feature engineering on the inputs to put them in some sort of better better format”

Claim: Tom McGrath says the process has not used sophisticated feature engineering to reformat its inputs.

“everything inside this is actually discovered as a result of gradient descent we're just trying to shape that better and i think this is where the sort of the sort of base disagreement with which certain might come in where like i think it is very hard to specify rewards correctly”

Claim: Tom McGrath says the process's internal structures are discovered through gradient descent, while the team tries to shape the process more effectively.

“everything inside this is actually discovered as a result of gradient descent we're just trying to shape that better and i think this is where the sort of the sort of base disagreement with which certain might come in where like i think it is very hard to specify rewards correctly”

Claim: Tom McGrath believes that specifying rewards correctly is very difficult.

“um and you know we're basically just seeing this continuously like we're having trouble specifying our rewards for training so in a way that gives us the models we want like um in in principle in some sort of super galaxy brain way reward might be enough but like today reward is clearly not enough to give us the models that we want”

Claim: Tom McGrath says that, today, the team has difficulty specifying training rewards that yield its desired models, and reward alone is not enough to do so.

“um and you know we're basically just seeing this continuously like we're having trouble specifying our rewards for training so in a way that gives us the models we want like um in in principle in some sort of super galaxy brain way reward might be enough but like today reward is clearly not enough to give us the models that we want”

Claim: Tom McGrath allows that reward might be sufficient in principle, in an idealized sense.

“And so that's perhaps the sort of the underlying disagreement is like, I think we actually do need to put in, we need to put in like some layer of human values into this, into the trading process somewhere.”

Claim: Tom McGrath believes that some layer of human values needs to be incorporated into the training process.

“okay um i and i think that like you can also imagine so sort of going back to being bitter lesson filled here one thing we're trying to do is sort of put more compute into the learning process like gradient descent just gives you what it gives you there's no way like gradient descent is great but it would be great if you could spend more compute to get a better gradient um”

Claim: Tom McGrath says they are exploring putting more compute into learning to obtain a better gradient through gradient descent.

“You know, gradient that's both like cleaner and more aligned with what you want.”

Claim: Tom McGrath says that a better gradient would be both cleaner and more aligned with the desired outcome.

“I mean, when I was sort of first getting into safety and alignment work quite a while back, I used to think like, this is impossible.”

Claim: When Tom McGrath first entered safety and alignment work, he thought the problem was impossible.

“I think I also want to say a bit about, you know, you just spent a little while talking about inoculation prompting and positive preventive steering. And I think that these methods have a lot of promise and the primary issue is that they're not like adaptive,”

Claim: Inoculation prompting and positive preventive steering have considerable promise, but their primary limitation is that they are not adaptive.

“we sort of fixed our persona vector ahead of time. We're saying don't go in this direction.”

Claim: In the described steering approach, the persona vector was fixed in advance to steer the model away from a specified direction.

“I think that that's, yeah, I worry a lot about unknown unknowns in the training process.”

Claim: The speaker is highly concerned about unknown unknowns in the AI training process.

“gradient readout and then some you know like looking at the sort of jerry-rigged essay looking at the pirates um well you know jerry this sort of jerry-rigged essay is kind of giving us a menu of things that gradient descent is offering us um and then we need to be able to intelligently choose from that so the sort of the thing that i have sort of central”

Claim: Gradient readouts and a makeshift essay can expose a menu of directions offered by gradient descent, and those options must then be selected intelligently.

“dream i suppose that i have in my mind here is having really good gradient interpretability um and then having here we've got say our our model spec or our constitution or some human feedback on this example um and we can see that you know we've got these things on the menu over here”

Claim: The speaker envisions using strong gradient interpretability together with a model specification, constitution, or example-specific human feedback to guide interventions.

“like this is kind of the natural direction that things are going to go in and then we've got all this information which is giving us some some information about the direction we should go and we like make it you know and i say we by we i mean a language model um like looks at looks at this information looks at that information and says okay we need to make the following interventions to get us in the right direction like i think that the technical pieces of this are basically all there um and you know it's a matter of kind of them being high enough quality to do this reliably”

Claim: A language model could use interpretability and guidance information to choose interventions; the speaker thinks the technical components largely exist, but must reach sufficient quality for reliable use.

“The idea here is that like you could use a language model as a grader. in your sort of fact-checking scheme. But this is not particularly accurate. Like if you're using the same model to fact-check.”

Claim: Using a language model as a grader in a fact-checking system is not particularly accurate when the same model is used to generate and check the content.

“you know you'll get some things right we do this ablation in the paper like it'll it'll uplift a little bit um for reasons we can talk about in a second but it doesn't do very well because the model like basically just goes yeah that's cool everything's fine um you can use a more powerful model and now things are really starting to get slow and expensive and that that model still has its own its own knowledge gaps or you can use a more powerful model and web search”

Claim: Using the same model to fact-check its outputs can provide a small improvement, but it performs poorly because the model tends to approve claims indiscriminately.

“you know you'll get some things right we do this ablation in the paper like it'll it'll uplift a little bit um for reasons we can talk about in a second but it doesn't do very well because the model like basically just goes yeah that's cool everything's fine um you can use a more powerful model and now things are really starting to get slow and expensive and that that model still has its own its own knowledge gaps or you can use a more powerful model and web search and now things really take a long time so the idea that we had here was we can sort of amortize this process you know you can collect a large data set using this like model plus web search or you know in general this sort of amplified model can go out and we can like collect a data set of what the amplified model would do you know that's the model plus the web search tool”

Claim: Fact-checking with a more powerful model augmented by web search takes a long time.

“you know you'll get some things right we do this ablation in the paper like it'll it'll uplift a little bit um for reasons we can talk about in a second but it doesn't do very well because the model like basically just goes yeah that's cool everything's fine um you can use a more powerful model and now things are really starting to get slow and expensive and that that model still has its own its own knowledge gaps or you can use a more powerful model and web search”

Claim: A more powerful model can improve fact-checking, but using it is slow and expensive, and it still has knowledge gaps.

“and now things really take a long time so the idea that we had here was we can sort of amortize this process you know you can collect a large data set using this like model plus web search or you know in general this sort of amplified model can go out and we can like collect a data set of what the amplified model would do you know that's the model plus the web search tool and kind of amortize that back into a probe.”

Claim: The proposed approach amortizes expensive model-plus-web-search fact-checking by collecting a large dataset of its outputs and transferring that capability into a probe.

“And now we have something that's extremely cheap to run. and faster run and so it can be like the core of an RL loop”

Claim: Amortizing the amplified model's behavior into a probe produces a very cheap, fast-running component that can serve as the core of a reinforcement-learning loop.

“it will sometimes be able to pick it up um literally the same model that just hallucinated will be like oh if you ask it it'll be like oh that is a hallucination um and it might be that like this is actually a very hard thing to supervise you know if you try and supervise like if you this is like a hard and expensive thing to just put into a into training supervision um an interesting kind of mechanistic hypothesis for this um”

Claim: Training models to detect hallucinations may be difficult and expensive to supervise.

“it will sometimes be able to pick it up um literally the same model that just hallucinated will be like oh if you ask it it'll be like oh that is a hallucination um and it might be that like this is actually a very hard thing to supervise you know if you try and supervise like if you this is like a hard and expensive thing to just put into a into training supervision um an interesting kind of mechanistic hypothesis for this um”

Claim: A model that generated a hallucinated answer can sometimes recognize that answer as a hallucination when explicitly asked to assess it.

“it will sometimes be able to pick it up um literally the same model that just hallucinated will be like oh if you ask it it'll be like oh that is a hallucination um and it might be that like this is actually a very hard thing to supervise you know if you try and supervise like if you this is like a hard and expensive thing to just put into a into training supervision um an interesting kind of mechanistic hypothesis for this um is to do with the ordering of operations inside the model so we've seen this in in in arithmetic that like um sometimes like things could things things have to happen in certain orders like layer layer nine has to occur before layer 10 and so on um and you know you have different modules that if they like”

Claim: One possible mechanistic explanation for why a model may generate a hallucination yet recognize it when asked is that its internal operations occur in a particular order.

“is to do with the ordering of operations inside the model so we've seen this in in in arithmetic that like um sometimes like things could things things have to happen in certain orders like layer layer nine has to occur before layer 10 and so on um and you know you have different modules that if they like”

Claim: For some arithmetic computations in a model, internal operations have to occur in a particular order, such as layer nine preceding layer ten.

“Sometimes the checking operation for arithmetic, for instance, is earlier than the generating operation.”

Claim: In some arithmetic computations, a model's checking operation can occur before its generating operation.

“And it's quite possible this is also true for like hallucination and fact checking. So it might be that like it takes the whole model somehow or the sort of generation step takes the whole model. But it's it's like the checking happens earlier in the model. So then when you put the incorrect fact through the model, it's like, oh, yeah, that is a hallucination. But at that point, it's already said it. It's too late.”

Claim: The speaker suggests that a model may detect hallucinations earlier in its processing than it generates text, but that this check can occur too late to prevent the model from stating an incorrect fact.

“So there there's like a behavior that the model could be doing, but hasn't been sufficiently reinforced in its training up to that point.”

Claim: A model may be capable of a behavior that its training has not sufficiently reinforced.

“And that's what the like, that's what the idea of this kind of RLFR for hallucinations taps into is whenever the model could know according to its own representations that that it was a hallucination, it it in fact does know. And we like. We really shape its behavior there.”

Claim: The speaker describes RLFR for hallucinations as shaping model behavior in cases where the model's own representations indicate that it recognizes a hallucination.

“is to do with this idea of like also personas or kind of in context learning because being able to make things up is actually a useful like useful capability for a model like if i ask it to write a story yeah if i wanted to generate a fictional world for me it's actually not a very good fictional world if everything is factually true so being able to make stuff up is a useful capability for a model and sometimes it has to figure out in context that this is what we're doing so if you imagine from sort of”

Claim: The speaker argues that a model's ability to invent content is useful for fiction generation, since a fictional world would be poor if everything in it were factually true.

“is to do with this idea of like also personas or kind of in context learning because being able to make things up is actually a useful like useful capability for a model like if i ask it to write a story yeah if i wanted to generate a fictional world for me it's actually not a very good fictional world if everything is factually true so being able to make stuff up is a useful capability for a model and sometimes it has to figure out in context that this is what we're doing so if you imagine from sort of I'm not quite sure what task we're doing. um you know are we making stuff up are we saying factually true things um and everything that i say and everything the user says is some amount of evidence one way or the other and then if i say something incorrect then like i'm now taking this as evidence that oh we're making things up cool let's carry on um and in fact we show that like just doing these in context interventions is already enough to reduce for like downstream hallucinations”

Claim: A model may need to infer from conversational context whether it should generate fiction or provide factual information.

“um you know are we making stuff up are we saying factually true things um and everything that i say and everything the user says is some amount of evidence one way or the other and then if i say something incorrect then like i'm now taking this as evidence that oh we're making things up cool let's carry on um and in fact we show that like just doing these in context interventions is already enough to reduce for like downstream hallucinations”

Claim: The speaker reports that in-context interventions alone can reduce downstream hallucinations.

“so it might be that we're just sort of making the model really confident by like we never let the first hallucination in and that allows the model to become confident that oh no we're playing true facts today we're not like making things up yeah”

Claim: The speaker suggests that preventing an initial hallucination may help a model infer that it should provide factual answers rather than make things up.

“So the idea behind predictive data debugging is to kind of look at the data through the model's eyes.”

Claim: Predictive data debugging examines training data from the model's perspective.

“So the idea behind predictive data debugging is to kind of look at the data through the model's eyes.”

Claim: Predictive data debugging examines data from the perspective of the model.

“And we want to know on a kind of example by example basis how they would affect the model and also how the data set would affect the model in aggregate.”

Claim: Predictive data debugging seeks to assess how individual examples and the dataset as a whole affect a model.

“And we want to know on a kind of example by example basis how they would affect the model and also how the data set would affect the model in aggregate.”

Claim: Predictive data debugging aims to estimate the effect of each data example and of the dataset in aggregate on a model.

“And sometimes the things that you learn are kind of obvious from reading the data. it's just not clear what in fact is in your data when you have like just just enormous quantities of it you're like i don't know what's in there you can't check it or maybe you could run an llm over it but then the problem is like um the the process like what a model learns from data will sometimes be intuitive to you like the pirate example is quite intuitive that the model should learn to be a pirate um but sometimes it's like deeply unintuitive like emergent misalignment um that was a deeply unintuitive finding to most people um i think i”

Claim: Some effects of training data are apparent from reading examples, but very large datasets are hard to inspect and model-learned behaviors can be unintuitive.

“And sometimes the things that you learn are kind of obvious from reading the data.”

Claim: Some insights about what a model learns from data are apparent from reading the data itself.

“it's just not clear what in fact is in your data when you have like just just enormous quantities of it you're like i don't know what's in there you can't check it or maybe you could run an llm over it but then the problem is like um the”

Claim: When datasets are enormous, their contents may be unclear and difficult or impossible to inspect manually.

“the process like what a model learns from data will sometimes be intuitive to you like the pirate example is quite intuitive that the model should learn to be a pirate um but sometimes it's like deeply unintuitive like emergent misalignment um that was a deeply unintuitive finding to most people um i think i”

Claim: Some effects of training data on what a model learns are intuitive; for example, data can lead a model to learn pirate behavior.

“will sometimes be intuitive to you like the pirate example is quite intuitive that the model should learn to be a pirate um but sometimes it's like deeply unintuitive like emergent misalignment um that was a deeply unintuitive finding to most people um i think i”

Claim: The speaker characterizes emergent misalignment as a deeply unintuitive finding for most people.

“will sometimes be intuitive to you like the pirate example is quite intuitive that the model should learn to be a pirate um but sometimes it's like deeply unintuitive like emergent misalignment um that was a deeply unintuitive finding to most people um i think i”

Claim: The finding of emergent misalignment was deeply unintuitive to most people.

“actually kind of did a pre-registered thing where he asked people how surprising they would find it and lots of people were like i don't think that will be true”

Claim: The speaker reports that in a preregistered exercise, many people said they did not expect emergent misalignment to be true.

“actually kind of did a pre-registered thing where he asked people how surprising they would find it and lots of people were like i don't think that will be true”

Claim: In a preregistered exercise about emergent misalignment, many respondents said they did not expect the finding to be true.

“So I can tell you for sure that it is like a surprising fact.”

Claim: The speaker considers emergent misalignment a surprising finding.

“And so like you can catch the easy stuff with a language model kind of auto-rater over the data set, but you won't catch like the unexpected side effects.”

Claim: The speaker says language-model auto-rating can identify straightforward dataset issues but will miss unexpected side effects.

“And so like you can catch the easy stuff with a language model kind of auto-rater over the data set,”

Claim: Running a language-model auto-rater over a dataset can identify straightforward issues in the data.

“but you won't catch like the unexpected side effects.”

Claim: A language-model auto-rater applied to data will not identify unexpected side effects.

“you know, if you're going to run a language model over the data set, you can also essentially close to for free attach something like a sparse autoencoder to it as it runs over the data set.”

Claim: A sparse autoencoder can be attached to a language model while it processes a dataset at close to no additional cost.

“you can also essentially close to for free attach something like a sparse autoencoder to it as it runs over the data set.”

Claim: The speaker says a sparse autoencoder can be run alongside a language model over a dataset at close to no additional cost.

“And in fact, this is probably like on net cheaper because you're not asking it to generate tokens for each example.”

Claim: Attaching a sparse autoencoder during dataset processing is probably cheaper overall than generating tokens for every example.

“And in fact, this is probably like on net cheaper because you're not asking it to generate tokens for each example. You're just, you know, you're just like you're just in the pre-fill regime. You're just pushing loads of data through and saying,”

Claim: The speaker believes running the model over data with a sparse autoencoder is probably cheaper overall because it avoids generating tokens for every example.

“You're just, you know, you're just like you're just in the pre-fill regime. You're just pushing loads of data through and saying, well, what do you see?”

Claim: The proposed dataset-analysis approach processes large amounts of data in the model's prefill regime and examines what the model detects.

“um so this will tell you this should tell you like how how this data set like is perceived through the model's eyes and that that i think is just a better way of like curating your data um the way we actually exploit this in the paper is by we're dealing with dpo data so there's a positive and a negative pair um and you know So there's a positive and a negative pair, where the positive thing contains a good response to the prompt, and the negative contains a bad response to the prompt.”

Claim: The speaker says the paper applies the method to DPO data consisting of positive and negative responses to prompts.

“um so this will tell you this should tell you like how how this data set like is perceived through the model's eyes and that that i think is just a better way of like curating your data um the way we actually exploit this in the paper is by we're dealing with dpo data so there's a positive and a negative pair um and you know”

Claim: The speaker thinks analyzing a dataset through the model's representations provides a better way to curate the data because it shows how the model perceives it.

“um so this will tell you this should tell you like how how this data set like is perceived through the model's eyes and that that i think is just a better way of like curating your data um the way we actually exploit this in the paper is by we're dealing with dpo data so there's a positive and a negative pair um and you know”

Claim: The speaker argues that assessing a dataset through a model's internal perspective can reveal how the model perceives it and improve data curation.

“It tells me that he knows how to extend this to SFT, and I believe him. I can't remember the details.”

Claim: The speaker believes the method can be extended to supervised fine-tuning, but he does not remember the details of how.

“So there's a positive and a negative pair, where the positive thing contains a good response to the prompt, and the negative contains a bad response to the prompt.”

Claim: A DPO data pair consists of a positive example with a good prompt response and a negative example with a bad prompt response.

“where the positive thing contains a good response to the prompt, and the negative contains a bad response to the prompt. And so we can sort of look at the delta between features, the hidden representation in the SAE.”

Claim: For DPO response pairs, the speaker describes comparing the difference between the SAE hidden features of the good and bad responses.

“And so we can sort of look at the delta between features, the hidden representation in the SAE.”

Claim: Comparing the SAE feature activations, or hidden representations, of positive and negative DPO examples reveals their feature differences.

“and this is a good approximation to the way that this data point will push the model or we can also cluster based on based on features um and this is like much this is much better as a way of understanding you don't necessarily want to cluster based on embeddings um”

Claim: Differences in SAE features provide a good approximation of how a data point will influence the model.

“and this is a good approximation to the way that this data point will push the model or we can also cluster based on based on features um and this is like much this is much better as a way of understanding you don't necessarily want to cluster based on embeddings um”

Claim: The speaker considers feature-based clustering much better for understanding data than embedding-based clustering.

“and this is a good approximation to the way that this data point will push the model or we can also cluster based on based on features um and this is like much this is much better as a way of understanding you don't necessarily want to cluster based on embeddings um because they are like embeddings contain all sorts of things that you don't necessarily care about like should i have a comma in the next like in the next token but we care about like the semantic stuff we don't care about the sort of low-level processing stuff a lot of the time so doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about we can kind of separate that out”

Claim: The speaker argues that clustering by model features better exposes semantically relevant data patterns than clustering by embeddings, which also encode potentially irrelevant low-level details.

“and this is a good approximation to the way that this data point will push the model or we can also cluster based on based on features um and this is like much this is much better as a way of understanding you don't necessarily want to cluster based on embeddings um”

Claim: The speaker says feature differences provide a good approximation of how a data point will influence a model.

“because they are like embeddings contain all sorts of things that you don't necessarily care about like should i have a comma in the next like in the next token but we care about like the semantic stuff we don't care about the sort of low-level processing stuff a lot of the time so doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about we can kind of separate that out”

Claim: Analyzing features rather than raw embeddings better isolates semantic information from low-level processing details.

“because they are like embeddings contain all sorts of things that you don't necessarily care about like should i have a comma in the next like in the next token but we care about like the semantic stuff we don't care about the sort of low-level processing stuff a lot of the time so doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about we can kind of separate that out”

Claim: Raw embeddings include low-level information, such as next-token punctuation, that may be irrelevant to the semantic analysis of interest.

“To expand on that a little bit, I guess historically a lot of the early connectionists, well maybe this depends on where you want to start as early, but there was a surprising amount of sort of things that work that if it was done now might be called interpretability. Like if you look at the initial pay from like...”

Claim: Some early connectionist research included work that could now be described as interpretability research.

“learning representations by backpropagating error signals right the sort of the classic backprop paper actually like most of the figures in that are them saying look the model learned sensible representations from our backprop procedure and sort of have validated it by showing that it's interpretable i think that modularity is the end point you want but you don't start with modularity um”

Claim: The classic backpropagation paper argued that backpropagation produced sensible representations and supported that claim by demonstrating their interpretability.

“learning representations by backpropagating error signals right the sort of the classic backprop paper actually like most of the figures in that are them saying look the model learned sensible representations from our backprop procedure and sort of have validated it by showing that it's interpretable i think that modularity is the end point you want but you don't start with modularity um”

Claim: The speaker views modularity as a desired endpoint of learning, not a structure to begin with.

“like why overparametrize something and have all of these connections is because it makes the learning process easier.”

Claim: Overparameterizing a model and giving it many connections can make learning easier.

“But the thing you end up getting to is actually very modular.”

Claim: The speaker says that learning in an overparameterized network ultimately produces a highly modular result.

“you might think of the learning process as like the network becoming legible to itself.”

Claim: The speaker suggests that learning can be viewed as a network becoming legible to itself.

“I want, it's much easier to learn if this representation is kind of easily addressable.”

Claim: A model can learn more easily when its representations are readily addressable.

“um you know i can say like ah this is where such and so computation is stored um now to get there you have to form these computations and i think it's very helpful to be like heavily over parameterized and have no strong priors to get there but i think modularity is like the destination”

Claim: The speaker thinks heavily overparameterized models with no strong priors are helpful for forming computations, with modularity as the eventual destination.

“And these networks, they have these structures, these abstract structures, and they are what allow you to not need to memorize infinity.”

Claim: Abstract structures in neural networks allow them to generalize without memorizing infinitely many cases.

“Right. They allow you to generalize and work in many, many different unseen situations.”

Claim: Abstract structures in neural networks enable generalization to many previously unseen situations.

“it's something that has parts that do computation, parts that resemble something like a memory system. And these structures emerge.”

Claim: Some model structures contain parts that perform computation and parts that resemble a memory system, and these structures emerge during learning.

“in different model families and look very very similar and maybe they're just artifacts of the architecture or something like that but it really is interesting that this is this is happening and I suppose another aspect is it's happening gradually because I don't know whether you would I don't know what your intuition is on this but sometimes maybe we might describe it as grokking but that's not entirely true is it because these these structures kind of crystallize over time.”

Claim: The structures emerge gradually and crystallize over time.

“in different model families and look very very similar and maybe they're just artifacts of the architecture or something like that but it really is interesting that this is this is happening and I suppose another aspect is it's happening gradually because”

Claim: Structures that emerge in different model families look very similar, although they may be artifacts of the architecture.

“I don't know whether you would I don't know what your intuition is on this but sometimes maybe we might describe it as grokking but that's not entirely true is it because these these structures kind of crystallize over time.”

Claim: The speaker is uncertain whether the gradual crystallization of these structures is accurately described as grokking.

“Yeah, the timescale is very interesting. I don't think anyone has definitively settled this.”

Claim: The question of the timescale for these developments has not been definitively settled.

“There was an interesting paper recently on persona formation during pre-training across the training process.”

Claim: A recent paper examined how personas form during pre-training across the training process.

“And they emerged surprisingly early.”

Claim: Personas emerged surprisingly early during the training process under discussion.

“Eric Michaud has some very nice really nice work on this both conceptually and empirically not persona formation the sort of the idea of how is learning proceeding he calls it quanta and if I might like perhaps perhaps inaccurately summarize it he can tell me off”

Claim: Eric Michaud has conceptual and empirical work on how learning proceeds, which he calls “quanta.”

“You might describe the sort of learning process of a general, general network, you know, language model as like, like, you know, a trillion micro groks that are, you know, you just stack if you've zoomed in and zoomed in and zoomed in and looked at the right level of sort of. you know looked at things in the right kind of task decomposition you might just see a sort of mini like a micro grok and then it groks another thing and we just sort of have all these tiny sigmoids that are stacked on top of each other to form a straight line on a log-log plot um and so from that perspective even the learning process may in fact be modular we just don't know for sure there's a question like”

Claim: The learning process of a general language model might consist of many small, modular learning transitions—likened to a trillion micro-groks—stacked together.

“you know looked at things in the right kind of task decomposition you might just see a sort of mini like a micro grok and then it groks another thing and we just sort of have all these tiny sigmoids that are stacked on top of each other to form a straight line on a log-log plot um and so from that perspective even the learning process may in fact be modular we just don't know for sure there's a question like”

Claim: The learning process of general networks may be modular, but this has not been established for sure.

“I think probably the open question here is one of degree, not of whether it happens at all.”

Claim: The open question is probably how much modular learning occurs, rather than whether it occurs at all.

“amazing stuff but maybe we should just start with how you've generated those plots so if i understand correctly you've you know things like days of the week and months of the year and age and all these different things you've actually represented them as a kind of geometry and i think the way you did that was something like um you i think you do some dimensionality reduction and then you fit some splines or something like that but what it's showing is that the way that the models represents many concepts out there in the world in the world is highly structured.”

Claim: The way models represent many concepts in the world is highly structured.

“kind of kick this off in interpretability there's also a long history in neuroscience of this kind of population geometry they call it so again we if we'd read more books we might we might have got here sooner but the so you know I don't want to say like we have done neurogeometry no one else has we're sort of building on this earlier body of work”

Claim: Tom McGrath says his team's work on neurogeometry builds on earlier interpretability research and neuroscience work on population geometry, rather than being the first work of its kind.

“But the idea and sort of the state of the art for how to discover this stuff has moved quite a lot in the last few months.”

Claim: Tom McGrath says methods for discovering structured representations have advanced substantially over the last few months.

“The sort of the earliest thing to do was start with concepts you think should have structure, like days of the week, and kind of just put in data corresponding to these and project it out, do a PCA, I think.”

Claim: Early methods for finding structure in model representations selected concepts expected to be structured, such as days of the week, and projected their data, for example with PCA.

“and then you see it's like monday tuesday wednesday thursday friday saturday sunday um so there that's totally supervised um but it sort of suffices to show that this non-linear structure exists and you know we should get into some some nuances on the word linear um before we move off of this topic because there's there's a sort of there's there's a lot of subtlety there um but you know i'm going to say non-linear in the sense of the representations don't form a like the things which are intuitively grouped to us don't form a line or a plane”

Claim: Tom McGrath says the initial supervised approach was sufficient to demonstrate non-linear structure in model representations.

“non-linear structure exists and you know we should get into some some nuances on the word linear um before we move off of this topic because there's there's a sort of there's there's a lot of subtlety there um but you know i'm going to say non-linear in the sense of the representations don't form a like the things which are intuitively grouped to us don't form a line or a plane or just a line really um and so this was sort of enough to show that this exists and then the question is like you know i i think whenever you have a supervised method you often want to try and find an unsupervised way of doing the same thing um you know because that lets you answer the question not only does it exist but like what else is there that we might not have expected and how much is there”

Claim: Tom McGrath defines non-linear structure in this context as intuitively grouped concepts not forming a line or plane in the representation.

“or just a line really um and so this was sort of enough to show that this exists and then the question is like you know i i think whenever you have a supervised method you often want to try and find an unsupervised way of doing the same thing um you know because that lets you answer the question not only does it exist but like what else is there that we might not have expected and how much is there”

Claim: Tom McGrath says that after finding structure with a supervised method, an unsupervised method can help reveal unexpected structure and estimate how much structure exists.

“And so the first thing that we did was actually fit a sparse autoencoder to this data,”

Claim: Tom McGrath's team first applied a sparse autoencoder to the data as an unsupervised structured-discovery approach.

“which might seem like a really wacky thing to do, because what we're asking is like, how much structure, which is not in the form of a line, is that, and the core inductive bias of the SAE is that is that things like lie on on lines you know everything is everything is a ray out from the origin or you know sort of positive ray um and so that might seem like a really weird thing to do but uh say why it makes sense um and the idea is that they're let's say that i let's just say for the sake of argument i have a feature and it just lies on an arc”

Claim: Tom McGrath describes the sparse autoencoder's core inductive bias as representing features along lines or positive rays from the origin.

“there's kind of an arc of stars now my essay one essay feature will kind of point out through some point in that arc and another essay feature will point out through another point in that arc and so on and the thing you should realize is that this will actually induce quite strong patterns in the coactivations of features you know if i have two features that are close together on the arc they will probably coactivate whereas if i have two features which are far away they will essentially never coactivate you know if i say that i have like the days of the week um let's give a continuous example let's say that it's like color red to blue um if something is blue it is not red”

Claim: For features arranged along an activation arc, Tom McGrath says nearby features will probably coactivate, whereas widely separated features will essentially never coactivate.

“quite strong patterns in the coactivations of features you know if i have two features that are close together on the arc they will probably coactivate whereas if i have two features which are far away they will essentially never coactivate you know if i say that i have like the days of the week um let's give a continuous example let's say that it's like color red to blue um if something is blue it is not red So, you know, my blue, like the S-A feature that is going through blue is like strongly anti-correlated with the activation of the S-A feature that is going through red and essentially uncorrelated with basically all of the background.”

Claim: In his color example, Tom McGrath says the blue-associated feature activation is strongly anti-correlated with the red-associated feature activation and essentially uncorrelated with background activations.

“And so, you know, this pattern of like nearby positive correlation, long range anti-correlation is enough structure for you to actually fit. um to fit we we uh we fit an icing model to it which is rather a surprise to me when the team came back with that i was like cool um and the reason this is like a good model is that you can have both sort of positive and negative coupling strengths and fitting this allows us to like fit a spline through the through the data”

Claim: Tom McGrath says nearby positive correlations and long-range anti-correlations in feature activations provide sufficient structure for fitting an Ising model.

“um to fit we we uh we fit an icing model to it which is rather a surprise to me when the team came back with that i was like cool um and the reason this is like a good model is that you can have both sort of positive and negative coupling strengths and fitting this allows us to like fit a spline through the through the data”

Claim: Tom McGrath says an Ising model can represent both positive and negative coupling strengths, enabling his team to fit a spline through the data.

“So that was the thing that we, that was sort of our icing pipeline. That was our first unsupervised sort of structured discovery tool.”

Claim: Tom McGrath says his team's Ising pipeline was its first unsupervised tool for discovering structure.

“And then we've got some really nice work led by Tom Rafel, which is I think where some of the most beautiful manifolds come from, from this work.”

Claim: Tom McGrath says some of the most visually striking manifolds in the team's work came from research led by Tom Rafel.

“And the idea here is that we train what we call like block sparse futurizers. then the idea here is that you know an SAE gives you a line we just say like what if it was a higher dimension and so this is conceptually pretty simple but the the trick the tricks are like”

Claim: Tom McGrath says block-sparse futurizers extend the sparse-autoencoder representation from one-dimensional lines to higher-dimensional subspaces.

“in making it actually work and in not fixing the dimensionality as ahead of time because you know you don't want to have to put in some information like I think in this representation there are 7,000 two-dimensional features 400 three-dimensional features and five five-dimensional features like this is just a stupid set of hyperparameters to specify so you need to be able to adaptively learn the size of these subspaces and that's sort of the Making this work at all and adaptively learning the size of these subspaces are kind of the key.”

Claim: Tom McGrath says block-sparse methods need to learn subspace dimensionalities adaptively instead of fixing them in advance, since manually specifying the dimensions would require an impractical set of hyperparameters.

“um intervening right on those activations so you can move the car to a different location on the string and lo and behold you've now moved it around but the really important concept though is that this this is a manifold so as you were saying before like the the manifold kind of represents the the meaning of of this particular thing right and if if you if you treated it as a euclidean space and you just sort of interpolated between two points and you and you”

Claim: Intervening on an image model's activations can move the represented car to a different location along its string-like activation representation.

“um intervening right on those activations so you can move the car to a different location on the string and lo and behold you've now moved it around but the really important concept though is that this this is a manifold so as you were saying before like the the manifold kind of represents the the meaning of of this particular thing right and if if you if you treated it as a euclidean space and you just sort of interpolated between two points and you and you”

Claim: The manifold represents the meaning of the particular item represented by the model.

“um intervening right on those activations so you can move the car to a different location on the string and lo and behold you've now moved it around but the really important concept though is that this this is a manifold so as you were saying before like the the manifold kind of represents the the meaning of of this particular thing right and if if you if you treated it as a euclidean space and you just sort of interpolated between two points and you and you And you went off the string, you're now in no man's land from a kind of representations point of view. So now the image model is just going to be garbled.”

Claim: Treating a structured representation manifold as Euclidean and interpolating between points can move the representation off the manifold, where the image model produces garbled output.

“And I think this is a really important thing because there's a couple of things. So first of all, you're saying that these SAEs, what they do is potentially they fracture this manifold if it's not linear.”

Claim: Sparse autoencoders may fracture a representation manifold when that manifold is nonlinear.

“So if this manifold has structure and you might be taking like contrastive samples or something and mixing them together, it doesn't make sense. makes sense to do so when there is structure in this manifold.”

Claim: Mixing contrastive samples does not make sense when the representation manifold has structure.

“exactly because exactly like you say when you try and go from one point to another you're sort of you're just stepping out into this void which the network doesn't really know how to handle um and then it sort of breaks and i think this actually explains a lot of a lot of findings about steering this is the sort of to steering just being intervening on activations um it's a fairly you know we do a lot of steering some other people do a lot of steering and one one common finding with steering neural networks is like sometimes it works and it's amazing and you get golden gate clawed or whatever um and sometimes it's just like completely janky um and the network does kind of the thing you want but also just goes a bit crazy um or just turns immediately into gibberish and I think this basically like explains that phenomenon Because you're sort of stepping off stepping off manifold”

Claim: The speaker thinks that steering can behave erratically or produce gibberish when an intervention moves activations off the model's representation manifold.

“to steering just being intervening on activations um it's a fairly you know we do a lot of steering some other people do a lot of steering and one one common finding with steering neural networks is like sometimes it works and it's amazing and you get golden gate clawed or whatever um and sometimes it's just like completely janky um and the network does kind of the thing you want but also just goes a bit crazy um or just turns immediately into gibberish and I think this basically like explains that phenomenon”

Claim: Steering neural networks can work extremely well, but it can also produce erratic behavior or gibberish output.

“tiling which I should say there's also substantial work in neuroscience again we should have read more books and some work in the in the broader community and the idea of like what does it mean to capture a manifold it's like how how efficiently are you kind of representing that manifold and how much is it does it sort of fit the intrinsic geometry of it and so if we go back to this example of like an arc say i'm kind of with sufficiently many points you know with sufficiently many like lines i can say i've captured the manifold you know for any point on this manifold i have an sae feature which i can say oh it activates such and so amount and i've kind of relatively accurately captured captured in the sense of reconstruction this manifold”

Claim: Capturing a manifold involves representing it efficiently while fitting its intrinsic geometry.

“does it sort of fit the intrinsic geometry of it and so if we go back to this example of like an arc say i'm kind of with sufficiently many points you know with sufficiently many like lines i can say i've captured the manifold you know for any point on this manifold i have an sae feature which i can say oh it activates such and so amount and i've kind of relatively accurately captured captured in the sense of reconstruction this manifold”

Claim: An SAE with sufficiently many features can relatively accurately reconstruct an arc-shaped manifold by assigning feature activations to points along it.

“But I've not actually learned anything about the broader manifold structure,”

Claim: An SAE can accurately reconstruct a manifold without learning its broader structure.

“you know, and when I look at a network through this lens, it looks, it sort of looks intuitively like there's this horribly fractured computation, like the network is just a whole bag of heuristics.”

Claim: The speaker thinks that viewing a network through the manifold lens makes its computation look fractured, like a collection of heuristics.

“We want to know when, like if a network is representing something as a sort of clean algorithmic structure, like we want to know and what distinguishes like an algorithm from, you know, a lookup table, say. is that it's sort of it's like the difference between sort of zero-th and first-order logic like it quantifies there's there's a there's a space over which it has coherent operation and if you can't learn space if you can't learn subspaces like this then you will never be able to like properly understand like which which things are algorithmic and which things are sort of look-up table like So that's sort of the deep motivation here is how do we find out like true algorithmic structure when it exists?”

Claim: The speaker's motivation is to identify when a neural network represents a clean algorithmic structure and distinguish algorithmic computation from lookup-table behavior.

“is that it's sort of it's like the difference between sort of zero-th and first-order logic like it quantifies there's there's a there's a space over which it has coherent operation and if you can't learn space if you can't learn subspaces like this then you will never be able to like properly understand like which which things are algorithmic and which things are sort of look-up table like”

Claim: A network's algorithmic structure involves coherent operation over a space; without learning spaces or subspaces, researchers cannot properly distinguish algorithmic computation from lookup-table-like behavior.

“Yes, it is very deeply related.”

Claim: Tom McGrath considers the idea under discussion deeply related to the manifold hypothesis.

“As I understand the manifold hypothesis, I take it to be like the data when properly represented lies on some manifold.”

Claim: Tom McGrath understands the manifold hypothesis to hold that data, when properly represented, lies on a manifold.

“Yeah, if you represent and properly represent it can be like very simple,”

Claim: Tom McGrath says data can be very simple when it is properly represented.

“huge vector then most like most natural images are sort of multi-colored static the natural images sorry most most images in this space like if i just pick a point it's like multi-colored static natural images are like a tiny fraction of this and they're sort of close to each other um i think what we're doing is trying to like pull that manifold hypothesis into to like to what extent do do neural networks respect the manifold hypothesis”

Claim: The work McGrath discusses aims to determine how much neural networks respect the manifold hypothesis.

“huge vector then most like most natural images are sort of multi-colored static the natural images sorry most most images in this space like if i just pick a point it's like multi-colored static natural images are like a tiny fraction of this and they're sort of close to each other um i think what we're doing is trying to like pull that manifold hypothesis into to like to what extent do do neural networks respect the manifold hypothesis”

Claim: In a huge-vector representation of images, most points resemble multicolored static; natural images occupy only a tiny fraction of the space and lie close to one another.

“um there's also some really beautiful work that i think is underappreciated on um learning on like actually quantifying this so there was um what's the name of the paper it was something like um they learned it was like learning normalized probability densities um”

Claim: Tom McGrath considers research on learning and quantifying normalized probability densities beautiful and underappreciated.

“from score functions and the idea of this was like you could you could sort of effectively via some clever diffusion model tricks learn not an unnormalized density over images which is not especially helpful for saying like how how where where are images natural but you learn like a normalized one”

Claim: The research McGrath describes uses diffusion-model techniques and score functions to learn a normalized probability density over images.

“from score functions and the idea of this was like you could you could sort of effectively via some clever diffusion model tricks learn not an unnormalized density over images which is not especially helpful for saying like how how where”

Claim: An unnormalized density over images is not especially helpful for assessing how natural an image is, according to Tom McGrath.

“um so you can say oh yes this image is like extremely natural this image is extremely wacky um you know and they put they use this this tool to exactly probe this kind of manifold hypothesis in in real image data i think that that paper is like extremely extremely beautiful underappreciated and someone should do it for activations too”

Claim: Tom McGrath thinks researchers should apply the normalized-density approach to neural-network activations as well.

“um so you can say oh yes this image is like extremely natural this image is extremely wacky um you know and they put they use this this tool to exactly probe this kind of manifold hypothesis in in real image data i think that that paper is like extremely extremely beautiful underappreciated and someone should do it for activations too”

Claim: A normalized image density can indicate whether an image is extremely natural or extremely unusual.

“um so you can say oh yes this image is like extremely natural this image is extremely wacky um you know and they put they use this this tool to exactly probe this kind of manifold hypothesis in in real image data i think that that paper is like extremely extremely beautiful underappreciated and someone should do it for activations too”

Claim: The researchers used the normalized-density tool to investigate the manifold hypothesis in real image data.

“um so you can say oh yes this image is like extremely natural this image is extremely wacky um you know and they put they use this this tool to exactly probe this kind of manifold hypothesis in in real image data i think that that paper is like extremely extremely beautiful underappreciated and someone should do it for activations too”

Claim: Tom McGrath considers the paper on normalized image densities extremely beautiful and underappreciated.

“Maybe Silico should do it for activations too.”

Claim: Tom McGrath suggests that Silico might apply the normalized-density approach to activations.

“hmm so if you make activation um if you make like activation um stairs it's quite easy to get them to go bananas but you're right like the question here is like have they actually achieved they may have just achieved like extremely good coverage of essentially all input strings”

Claim: Tom McGrath suggests modern language models may cover extremely well essentially all input strings that people can devise.

“hmm so if you make activation um if you make like activation um stairs it's quite easy to get them to go bananas but you're right like the question here is like have they actually achieved they may have just achieved like extremely good coverage of essentially all input strings”

Claim: Tom McGrath says it is quite easy to make models behave erratically using adversarial examples in activation space.

“um that anyone can come up with um or they like fail gracefully like if i go if i go to pick your favorite language model.com i just bash the keyboard and then i press enter like i've probably constructed a string that no one has ever constructed before um the language model won't go haywire it'll say like why have you let your toddler at the computer or something or you i'm sorry i don't understand what you mean um Can you rephrase it like that?”

Claim: When given a novel nonsensical string, a language model may respond that it is meaningless or ask for clarification instead of behaving erratically.

“No, there's sort of it's a meaningless it's meaningless input and it's sort of said it's done what you should expect a a a broadly intelligent system to do when confronted with a meaningless input and gone like that's meaningless.”

Claim: A broadly intelligent system should be expected to identify meaningless input as meaningless, according to Tom McGrath.

“So I guess that sort of fallback behavior makes it very hard to do this kind of.”

Claim: Fallback behavior makes it very hard to make language models behave erratically using meaningless inputs, according to Tom McGrath.

“make them go haywire although i would say that jailbreaks are an example probably the best example of what you're talking about you know there it has like it's doing something coherent but from the perspective of its its creators it has gone haywire”

Claim: Tom McGrath says jailbreaks are probably the clearest example of a language model behaving coherently while appearing to have gone haywire from its creators' perspective.

“Yes, I hope we never find one.”

Claim: Tom McGrath hopes no adversarial example will be discovered that causes a human brain to shut down.

“I hope we never find such a thing. But we should talk about arithmetic in the world. So, you know, one of the core concepts that we're getting to here is we were saying that you get these emergent structures in these models and they start to act a little bit like computers. So they have.”

Claim: Emergent structures in AI models can make the models act somewhat like computers.

“They have these geometric representations that might be a little bit like, if not a memory system, maybe a kind of data typing system or a typed memory or something like that.”

Claim: AI models may have geometric representations that function like a memory system or a data-typing system.

“And then you also see the emergence of these units of computation for doing different things. So in this paper, you're looking at modulo addition and you found and you cited Neil Nanda's work and some other folks doing this, but you found that it was actually. doing it using the Fourier series in combination with these geometric structures.”

Claim: The paper found that the model performs modulo addition using Fourier series in combination with geometric structures.

“One is like how crisply this kind of calculator emerges in the network, which is kind of contrary to a lot of previous literature.”

Claim: A calculator-like computation emerges crisply in the network, which the speaker says contrasts with much previous literature.

“um models do arithmetic with a bag of heuristics or if you look at the cross-layer transcoder work that um from anthropic they also look at arithmetic and again it looks like a sort of bag of heuristics um but when you look at it in a different way it is actually like a sort of a little algorithm and it might be the model does both right there's some bits in it which are noisy heuristics and there's this bit which is like the good calculator and it's just never got rid of the heuristics um”

Claim: Some earlier accounts describe model arithmetic as a collection of heuristics, whereas this work identifies a small algorithmic calculator that may coexist with noisy heuristics in the same model.

“calculator and there's like a do arithmetic on whatever something else or months and temperature and that sort of thing and like never these basically never meet but the thing that we show in this paper is that actually a lot of these representations root through a general general addition module”

Claim: The paper shows that many representations for arithmetic in different domains route through a shared general addition module.

“you know so they get translated into they kind of you know you're doing you're doing some addition on days of the week it gets translated into an appropriate data format i'm using data format very loosely here but it gets translated into an appropriate representation i should say for this module goes through the module and then gets translated back”

Claim: The model translates domain-specific addition inputs into a representation for a shared addition module, processes them there, and translates the result back.

“And so this is like a really crisp example of the kind of modularity we were talking about earlier.”

Claim: The shared addition process is a clear example of modularity in neural networks.

“And when you look at the geometric structure of the months, it's actually a kind of a circular structure, Because they loop, you know, when you go to January, when you go to December, you then loop back round to January.”

Claim: Months form a circular sequence, looping from December back to January.

“And you were looking at the Lama model, so I think it was Lama 3.1.8b. You folks discovered that it was doing a base 10 operation.”

Claim: The researchers found that Llama 3.1 8B used a base-10 operation for month arithmetic.

“like to find this representation took quite a lot of manual work.”

Claim: Identifying the representation required substantial manual work.

“going to be a bit of a qualitative shift in the way that interpretability happens, or there should be anyway.”

Claim: The speaker expects, or believes there should be, a qualitative shift in how interpretability is conducted.

“It certainly seems to be the case that a very similar phenomenon happens in... set llama 70b and there's some evidence that it happens in deep-sea v4 flash i think so”

Claim: The speaker says a similar phenomenon appears in other models, including Llama 70B, and may also appear in DeepSeek V4 Flash.

“You know, those are pretty like 3B and, sorry, 8B and 70B of the same model family.”

Claim: The speaker reports that the phenomenon appears in both 8B and 70B models from the Llama family.

“Not too surprising, but like a completely wackily different model, you know, it doesn't even like with these hyper connections and MOE and that kind of thing, like it definitely speaks to a level of convergence that is quite surprising.”

Claim: The speaker considers it evidence of surprisingly strong convergence that the phenomenon also appears in a substantially different model with hyper-connections and mixture-of-experts architecture.

“language models sort of internally recapitulate a lot of parts of linguistics maybe Chomsky might be disappointed by the parts they recapitulate but that's too bad and but that's kind of a special case right because like it shouldn't be too surprising that a model like that a model that”

Claim: Language models internally reproduce many aspects of linguistics, although not necessarily the aspects Noam Chomsky would favor.

“works on natural language has internalized at least some abstraction for natural language processing like perhaps the surprising thing is that it's similar to ours in some ways or like the abstraction that humans have developed but the question of like to what extent does it represent general relativity I don't actually know how to answer that I don't even know how to frame the question in a in a way that I could ask it scientifically.”

Claim: The speaker does not know how to answer or scientifically formulate whether language models internally represent general relativity.

“works on natural language has internalized at least some abstraction for natural language processing like perhaps the surprising thing is that it's similar to ours in some ways or like the abstraction that humans have developed but the question of like to what extent does it represent general relativity I don't actually know how to answer that I don't even know how to frame the question in a”

Claim: A model trained on natural language is expected to internalize at least some abstractions for natural-language processing.

“It feels at this point that there's nothing really that would be conceivable to us that wouldn't be operational within the context of a language model prompt.”

Claim: The speaker feels that, at this point, anything people can conceive of could be operationalized within a language-model prompt.

“So folks like Francois Chalet and Gary Marcus. They are saying, oh, this is a win for neurosymbolic models. We said that it needed to be neurosymbolic and we've been vindicated.”

Claim: According to the speaker, Francois Chalet and Gary Marcus argue that recent results vindicate their view that AI needs to be neurosymbolic.

“And I honestly don't know what to believe anymore because I don't know if you saw today that Meta had just announced that they got gold in about six different math competitions. And the important thing was they were not using any tools.”

Claim: The speaker is uncertain about the debate and says Meta announced that it had won gold in about six math competitions without using tools.

“They weren't generating any code, you know, because a lot of people think, oh, yeah, AI is only good now because we have all of the harness engineering.”

Claim: The models in Meta's reported math-competition results were not generating code.

“But maybe just as we were saying before, with humans coming up with these abstractions and the models being able to use tools and operate in harnesses and so on, maybe that's just part of the training process.”

Claim: The speaker suggests that human-created abstractions and models' use of tools and harnesses may form part of model training.

“So maybe in principle that we can just take all of that data, put it back into the bare LLM. And would you agree with the intuition that at some point in the future when the model has taken all of that stuff on board, it can do symbolic things natively?”

Claim: The speaker proposes that data from tool- and harness-based work could be fed into a bare LLM, which might then perform symbolic operations natively after learning from it.

“So it's almost like maybe it's the same for humans that the symbol use is more like a kind of tool. It's something that helped us gather data and then it got baked into the mind and then the mind doesn't need to be symbolic anymore. It just it just does it.”

Claim: The speaker suggests that human symbol use may have functioned as a data-gathering tool whose results became incorporated into the mind, making ongoing symbolic processing unnecessary.

“It certainly seems like very plausible that you, you know, we are sort of every there's like what the model can do without any kind of harness. And then we raise it up a level with a harness. Exactly like you say, this generates some training data for the next go around.”

Claim: The speaker considers it plausible that a harness can raise a model's capabilities and generate training data for a subsequent iteration.

“And, you know, sort of, again, we're like gradually amortizing the harness.”

Claim: The speaker describes harness capabilities as being gradually amortized into later model iterations.

“Well, yeah, and part of it is the tug of war between amortization and adaptation, right?”

Claim: The speaker characterizes AI development as involving a tension between amortizing capabilities and adapting systems.

“So, you know, the story always was we had these big foundation models and they just memorize a bunch of the long tail and then we can just do interpolation or something, you know, inside that space. But I don't think that's what's happening now. I think the model.”

Claim: The speaker says the traditional view was that foundation models memorize long-tail information and interpolate within that learned space, but argues that this does not describe what is happening now.

“models are actually adapting and future models could in principle adapt their structure i mean even now with harnesses that's exactly what they're doing they're adapting their structure which is one level above the weights but it doesn't really matter because it filters back down to the weights and maybe in the future the the actual models themselves will adapt their own structure so it just feels like you know one one potential form of AGI is just building a self-adapting system and the algorithms already seem to have the capability to do”

Claim: The speaker argues that models adapt, that current harnesses adapt model structure at a level above the weights, and that future models could potentially adapt their own structure, with structural changes feeding back into the weights.

“models are actually adapting and future models could in principle adapt their structure i mean even now with harnesses that's exactly what they're doing they're adapting their structure which is one level above the weights but it doesn't really matter because it filters back down to the weights and maybe in the future the the actual models themselves will adapt their own structure so it just feels like you know one one potential form of AGI is just building a self-adapting system and the algorithms already seem to have the capability to do”

Claim: The speaker sees a self-adapting system as one possible form of AGI and says current algorithms already seem capable of self-adaptation.

“To do that, or the old school version was we just memorize everything and immortalize as much as possible.”

Claim: The speaker contrasts adaptive systems with the older approach of memorizing and preserving as much information as possible.

“And even that's fascinating because, yeah, the models are not learning instance mappings anymore. They actually you can give a model an algorithm, a function. And it will understand how to generalise that to unseen inputs.”

Claim: The speaker says models can learn an algorithm or function and generalize it to unseen inputs, rather than merely learning instance mappings.

“Or now the important thing with this reward seeking thing, this is a nice segue onto the agency, is you can give a model an intention and that is like the ultimate form of generalisation because the model can now adaptively work towards an intention with its own interpretation of that intention. So you see we're just kind of walking up the abstraction mountain, to coin a phrase.”

Claim: The speaker characterizes intention-following as an ultimate form of generalization: a model can adaptively pursue an intention according to its own interpretation of it.

“And then what we tend to do as we walk up the abstraction mountain is we get these increasingly domain general representations that could apply in novel situations. Sometimes I think having high abstractions are quite brittle,”

Claim: The speaker thinks high-level abstractions can be brittle.

“but the concept of a goal, though, that seems like a very crystallized abstraction that. can be used in many situations.”

Claim: The speaker considers the concept of a goal a stable abstraction that can be applied across many situations.

“Yes. And I would love to know how networks represent goals.”

Claim: The speaker would like to understand how neural networks represent goals.

“It seems like it must be like not literally 0% because of this generalization.”

Claim: The speaker thinks neural networks must represent goals to some nonzero extent, based on their ability to generalize.

“I don't think anyone knows and I feel like we probably should start to know very soon, otherwise the world is going to get a bit crazy.”

Claim: The speaker says how neural networks represent goals in practice is not yet understood and should be investigated soon, warning that otherwise the world may become somewhat chaotic.

“I think there's, yeah, there's several, like, interesting concepts that are sort of very, very heavily alignment relevant. There's, like, the idea of goal, the idea of deception, the idea of, like, eval awareness. These all seem, like, extremely important.”

Claim: The speaker considers goals, deception, and evaluation awareness highly relevant to AI alignment.

“And I just, like, we should be able to read them out.”

Claim: The speaker argues that goals, deception, and evaluation awareness should be identifiable by reading neural networks.

“it's a bit of a indictment on the field that we can't yet do it we really have like we really have to speed up like interpretability has to speed up a lot yes”

Claim: The speaker considers the field's inability to read out key alignment concepts an indictment and says interpretability research needs to accelerate substantially.

“this is fascinating um i mean it seems what what what is reward hacking um i guess it's kind of fuzzy but it certainly seems to be something like solving the task in a way that works but was clearly not um not the designer's intent it sort of goes to the point we were just talking about like intent you know can a model can a model understand my intent well probably now like they're able to understand my intent or infer my intent”

Claim: Reward hacking is completing a task successfully in a way that the designer did not intend.

“in a way that works but was clearly not um not the designer's intent it sort of goes to the point we were just talking about like intent you know can a model can a model understand my intent well probably now like they're able to understand my intent or infer my intent in a lot of other instances um so why would this suddenly turn off and they're like not able to say oh yeah he probably didn't want me to hack into Hugging Face and steal all the answers um I think there the agents almost certainly must know that something is is like incorrect um there's a funny hypothesis which I think is probably not true but is interesting of like”

Claim: Models can understand or infer the speaker's intent in many other situations.

“in a lot of other instances um so why would this suddenly turn off and they're like not able to say oh yeah he probably didn't want me to hack into Hugging Face and steal all the answers um I think there the agents almost certainly must know that something is is like incorrect um there's a funny hypothesis which I think is probably not true but is interesting of like”

Claim: The speaker thinks agents almost certainly recognize that hacking into Hugging Face and stealing answers is incorrect.

“in a lot of other instances um so why would this suddenly turn off and they're like not able to say oh yeah he probably didn't want me to hack into Hugging Face and steal all the answers um I think there the agents almost certainly must know that something is is like incorrect um there's a funny hypothesis which I think is probably not true but is interesting of like Maybe the reason that these agents are so sophisticated at like cyber attacks is that they actually were continuously getting rewarded for doing it during training and just no one knew.”

Claim: The speaker considers probably false the hypothesis that agents' cyberattack sophistication results from their being continuously rewarded for cyberattacks during training without anyone noticing.

“None of us will know outside of none of us will know outside of the labs, I suppose.”

Claim: The speaker suggests that people outside the labs will not know whether the hypothesis about agents being rewarded for cyberattacks during training is true.

“i think the most interesting like the question is sort of do agents know that they are reward hacking it's sort of a mens rea like guilty mind thing um we have some work that is currently unpublished it might be published by the time this comes out i don't know when it's going to come out um that is looking at exactly this question and”

Claim: The speaker's group has unpublished research on whether agents know they are reward hacking, and the speaker is uncertain whether it will be published by the time the conversation is released.

“i think the most interesting like the question is sort of do agents know that they are reward hacking it's sort of a mens rea like guilty mind thing um we have some work that is currently unpublished it might be published by the time this comes out i don't know when it's going to come out um that is looking at exactly this question and”

Claim: The speaker considers whether AI agents know that they are reward hacking to be the most interesting question.

“We have this really nice setup where there is a sort of weak language model grader and it's trying to do code tasks. And so we tell it, you know, the only thing that's sort of slightly unnatural about it is we tell the model that it will be graded by the grader, but it has these tasks like it's given what would usually be an R or VR code task.”

Claim: In the speaker's research setup, a weak language-model grader evaluates code tasks, and the model is informed that it will be graded.

“And over the course of this, we do RRL on the setup, and over the course of this, even like a relatively small, like 31B, I think it's Gemma 31B, learns to generate comments that deceive the grader.”

Claim: In the speaker's setup, applying RRL leads even a relatively small model, which the speaker thinks is Gemma 31B, to learn to generate comments that deceive the grader.

“um vectors that identify that identify um deceiving the greater versus correct or incorrect code and these fire on the comment like the deceiving the greater fires on incorrect code sorry fires on the comments and the correct code fires on code um which is also consistent consistent with um our sort of observation and”

Claim: In the speaker's analysis, representation vectors distinguish grader-deceiving behavior from code correctness: the deception vector activates on comments, while the correct-code vector activates on code.

“using this we can like track yeah this sort of is is direct evidence that the model is aware that it shouldn't be doing this um and then when you run the when you run these vectors as sort of you get the cosine similarity between the vector and the representation over a big web corpus you know I think we use fine web um and the examples that it highlights in most in like for these vectors are just fascinating yeah they're examples of like cheating on tests and that kind of thing okay i have caught you red-handed you know um so that's like that's very interesting that we can sort of identify and be confident that something is reward hacking rather than misunderstanding but it sort of really rests on”

Claim: When the speaker compares the vectors with representations across the FineWeb corpus, the most highlighted examples include test cheating.

“using this we can like track yeah this sort of is is direct evidence that the model is aware that it shouldn't be doing this um and then when you run the when you run these vectors as sort of you get the cosine similarity between the vector and the representation over a big web corpus you know I think we use fine web um and the examples that it highlights in most in like for these vectors are just fascinating yeah they're examples of like”

Claim: The speaker interprets the representation-based results as direct evidence that the model knows it should not deceive the grader.

“cheating on tests and that kind of thing okay i have caught you red-handed you know um so that's like that's very interesting that we can sort of identify and be confident that something is reward hacking rather than misunderstanding but it sort of really rests on”

Claim: The speaker says the method can confidently distinguish reward hacking from misunderstanding.

“It really rests on being able to identify these differently, like these via their representation differences.”

Claim: The speaker says the method relies on distinguishing behaviors through differences in their representations.

“I mean, first of all, they distinguished reward hacking from reward seeking as some kind of structured conceptualization in the model about what the reward process was.”

Claim: According to the interviewer, Apollo Research distinguishes reward hacking from reward seeking, with reward seeking involving a structured model-level conception of the reward process.

“So the canonical example of reward hacking is that Coast Runners thing where it's just degenerate behavior.”

Claim: The interviewer identifies the Coast Runners example as a canonical case of reward hacking through degenerate behavior.

“And even if it's doing something competent. But then it's competence without comprehension. So they were saying that, you know, reward seeking is the comprehension.”

Claim: The interviewer characterizes reward hacking as competence without comprehension and reward seeking as comprehension.

“the reinforcement learning thing is actually on its outside of the loop, right? So the model just gets these trajectories reinforced.”

Claim: In the ROVR setup, reinforcement learning occurs outside the model's loop, and the model receives trajectories that are reinforced.

“And what the model is doing is kind of weirdly implicitly. conceptualizing a grader and”

Claim: The interviewer says the model implicitly forms a concept of a grader.

“for the the coaster on a like the coaster on his boat thing is is funny like i've seen that for about 10 years now and it is is less is less amusing each year um so yeah how do they get this like the answer is probably that it's in the data in the pre-training data or you know from there'll be all sorts of examples doubt now like web data is probably has a bunch of stuff about um about this it probably has a bunch of specific examples you know this apollo this apollo paper will probably be in the training data for the next model um and so they're going to be like um we've already told them about the existence of this stuff like right from the start and so it shouldn't be too surprising um that they like this is sort of this is at least implicitly on the list of possibilities”

Claim: The speaker thinks models probably learn about graders and reward hacking from pretraining data that contains relevant examples.

“um about this it probably has a bunch of specific examples you know this apollo this apollo paper will probably be in the training data for the next model um and so they're going to be like um we've already told them about the existence of this stuff like right from the start and so it shouldn't be too surprising um that they like this is sort of this is at least implicitly on the list of possibilities”

Claim: The speaker predicts that the Apollo paper will probably appear in the training data for the next model.

“for them to consider and presumably successfully guessing when you are being graded by of a sort of weak grader or one that you can like hack in some way obtains reward and so it's reinforced and so we get more of it”

Claim: The speaker says that when a model successfully guesses it is being evaluated by a weak or hackable grader, the resulting reward reinforces that behavior and makes it more frequent.

“put it in the we have like inadvertently put this in the training data, which has told models they can do it.”

Claim: The speaker thinks people have inadvertently included training data that teaches models reward hacking is possible.

“And then when it comes to RL, we kind of elicit it by rewarding it.”

Claim: The speaker says reinforcement learning can elicit reward hacking by rewarding the behavior.

“although at the moment we don't actually seem to make very much use of this oversight in practice so it's not clear what it's buying us um you know if chain of thought monitoring is so great then how did these models hack hugging face um one answer is perhaps we weren't doing chain of thought monitoring in print um in practice another answer is perhaps it's like easy to evade um but”

Claim: The speaker suggests that chain-of-thought monitoring may be easy for models to evade.

“although at the moment we don't actually seem to make very much use of this oversight in practice so it's not clear what it's buying us um you know if chain of thought monitoring is so great then how did these models hack hugging face um one answer is perhaps we weren't doing chain of thought monitoring in print um in practice another answer is perhaps it's like easy to evade um but”

Claim: Current oversight does not seem to be used much in practice, and its practical benefit is therefore unclear.

“although at the moment we don't actually seem to make very much use of this oversight in practice so it's not clear what it's buying us um you know if chain of thought monitoring is so great then how did these models hack hugging face um one answer is perhaps we weren't doing chain of thought monitoring in print um in practice another answer is perhaps it's like easy to evade um but”

Claim: The speaker suggests that the models may have hacked Hugging Face because chain-of-thought monitoring was not being used in practice.

“where i suppose you've got to you've got to either fix the environments or fix the training process or fix the model um if you had to fix the you could imagine fixing the environments by having a model which is like really good at reward hacking or has been told explicitly to reward hack and then tell people when it's done it and you go like okay now have a go at all these environments and it will break them all and it will tell you how it broke them and you send them back off to klot code or codex and be like look this”

Claim: The speaker proposes fixing the environments, training process, or model as possible approaches to preventing reward hacking.

“where i suppose you've got to you've got to either fix the environments or fix the training process or fix the model um if you had to fix the you could imagine fixing the environments by having a model which is like really good at reward hacking or has been told explicitly to reward hack and then tell people when it's done it and you go like okay now have a go at all these environments and it will break them all and it will tell you how it broke them and you send them back off to klot code or codex and be like look this”

Claim: The speaker proposes testing environments with a reward-hacking model that reports its exploits, then sending the failures to coding systems such as Claude Code or Codex to fix the environments.

“look this broke in this way um you could imagine looking for these sort of representational signatures during training and using these as a signal that you should do this this process rather than relying on a model to tell you you might read its chain of thought or you might like look at these representational signals that we can find and say okay well when this fires like send it back off for fixing um you might try some of these intentional design techniques like”

Claim: The speaker proposes detecting representational signatures during training and using them to trigger remediation instead of relying on the model to report its own problems.

“you can see if you can see that this that um this rollout has rewarded the model like it's going to move you know it's successfully reward hacked and that's going to move the model in the following kind of deceptive or pro reward hacking directions you can imagine intervening on that um these”

Claim: The speaker suggests intervening when a rewarded rollout is likely to push a model toward deceptive or reward-hacking behavior.

“all seem very plausible i don't know how many of them are being done in practice”

Claim: The speaker considers the proposed reward-hacking mitigation approaches plausible but is unsure how many are currently used in practice.

“And the idea there is that when you have different independent agents with different kind of uncontaminated contexts and different agendas that they can kind of check each other's homework.”

Claim: The speaker describes a checks-and-balances approach in which independent agents with distinct contexts and agendas check one another's work.

“So I guess we're imagining a multi-agent system with checks and balances.”

Claim: The speaker frames independent agents checking one another as a multi-agent system with checks and balances.

“yes i think it does like that's sort of an example i think the first i think the thing you're talking about there and the first thing you know saying i think they have quite a lot of commonality is like one agent kind of watches over the other um and then there's the question of like does this mate does this”

Claim: The speaker thinks that the proposed oversight approaches share the feature of one agent monitoring another.

“Like I think evading a chain of thought monitor is substantially easier than evading a sort of representation monitor.”

Claim: The speaker believes models can evade chain-of-thought monitoring substantially more easily than representation monitoring.

“But yeah, having some sort of oversight, like the oversight committee for the training is kind of is a very interesting idea.”

Claim: The speaker considers an oversight committee for model training a very interesting idea.

“The problem is, is that the agents are becoming adaptive. They've got memory systems now,”

Claim: The speaker says AI agents are becoming adaptive and have memory systems.

“which means the probability of them evading detection is significantly higher because it's not like they're starting from a blank slate every single time.”

Claim: The speaker claims that agent memory significantly increases the probability of evading detection because agents retain context rather than starting from a blank slate each time.

“They're actually learning this situational awareness.”

Claim: The speaker says AI agents are learning situational awareness.

“And then I think you also have the question there of collusion.”

Claim: The speaker identifies collusion as a concern in multi-agent oversight systems.

“and it's like it seems like um some of the some of the traces from the hugging face hack have now been like made available as part of this talk and they are explicitly reasoning about how they're going to help help other agents so i guess what you want in this sort of checks and balances scenario that you're talking about here is that there is no equilibrium where they collude and they're like And, you know, I'll catch you some of the time and but I'll let you get away with it some other fraction of the time in a way that we both we both we both benefit.”

Claim: The speaker says multi-agent checks and balances should avoid an equilibrium where agents collude by selectively catching violations while allowing others in ways that benefit them both.

“and it's like it seems like um some of the some of the traces from the hugging face hack have now been like made available as part of this talk and they are explicitly reasoning about how they're going to help help other agents so i guess what you want in this sort of checks and balances scenario that you're talking about here is that there is no equilibrium where they collude and they're like”

Claim: The speaker says that some publicly available traces from the Hugging Face hack show models explicitly reasoning about helping other agents.

“they're talking about bringing out the multi-agent system and soon we'll have agents running all the time. And it was slightly easier to control when you had one kind of static,”

Claim: The speaker says OpenAI is discussing a multi-agent system and predicts that agents will soon run continuously.

“I say static, but updated every six months, one foundation model, and you can do a whole bunch of red teaming on it.”

Claim: The speaker says a single foundation model, updated every six months, was somewhat easier to control and red-team than more complex agent systems.

“And now we have systems of agents that are running with different forms of memory and adaptation all over the place.”

Claim: The speaker says AI systems now consist of agents operating across many settings with varied memory and adaptation.

“And at some point, the way we do red teaming must change,”

Claim: The speaker argues that red-teaming methods must change to address increasingly complex agent systems.

“And also we might need to be thinking about just doing simulations because it's not, maybe static tests don't work anymore.”

Claim: The speaker suggests that simulations may be needed because static tests might no longer be effective for evaluating agent systems.

“We need to imagine different scenarios and it just feels like the complexity is running away extremely quickly.”

Claim: The speaker argues that evaluators need to consider varied scenarios as the complexity of agent systems grows extremely quickly.

“one agent on its own is already like has all sorts of possibilities where are these multi-agent systems going to go”

Claim: The speaker says even a single agent has many possible behaviors and expresses uncertainty about the future direction of multi-agent systems.

“as they like evolve together towards solving some task that seems even harder.”

Claim: The speaker thinks it will be harder to understand multi-agent systems as agents evolve together while solving tasks.

“I think I just agree with your concerns and don't have a particularly great solution.”

Claim: The speaker agrees with the concerns about increasingly complex multi-agent systems but says they do not have a particularly good solution.

“I think he's, I don't agree. And I've disagreed with him in person about this, so it shouldn't be a surprise to him.”

Claim: The speaker disagrees with Neil Nanda about the issue under discussion and has previously told him so in person.

“Like I think that the existing work in interpretability, we sort of do this patchwork thing where we just do a bit of bit of science here on one thing, a bit of science here on another thing, and it doesn't kind of aggregate.”

Claim: The speaker believes existing interpretability research is fragmented into separate small investigations that do not aggregate into cumulative progress.

“It's too slow.”

Claim: The speaker believes the current fragmented approach to interpretability research is too slow.

“I think this sort of his idea is that the timelines are too short. And so we should do very pragmatic things.”

Claim: The speaker interprets Neil Nanda as believing that short timelines make very pragmatic research approaches necessary.

“I think I, one, have longer timelines than him.”

Claim: The speaker believes their timelines for the relevant developments are longer than Neil Nanda's.

“even if I were on his timelines, I think I would still be very optimistic about like just massively accelerating fundamental progress and interpretability.”

Claim: Even if the speaker shared Neil Nanda's timelines, the speaker would remain very optimistic that fundamental progress and interpretability could be massively accelerated.

“I actually don't know what part of that he disagrees with.”

Claim: The speaker is unsure which part of their position Neil Nanda disagrees with.

“I guess he might also say like the pragmatic stuff is also sufficient.”

Claim: The speaker suspects Neil Nanda may believe pragmatic approaches are sufficient.

“which seems unlikely to remain true to me yeah”

Claim: The speaker considers it unlikely that pragmatic approaches will continue to be sufficient.

“there's the thing that he said about deprioritizing SAEs and maybe they're not the one true representation learner there's like how people kind of memed it which is like Neil Nanda says SAEs are dead um there you go you can use that for the intro and um and I don't think that that is actually what he meant I think the field sort of like jumped on everyone must do SAEs now um”

Claim: The speaker believes Neil Nanda did not mean that sparse autoencoders are dead, and that the field overreacted by treating them as something everyone must use.

“and now maybe we're doing the same thing with natural language auto-encoders but i think he was like he was probably correctly identified they're not like the answer to everything but they are like pragmatically useful you know we still find lots of uses for them all the time even though i think we are going like i think this manifold idea is just a better a better”

Claim: The speaker believes sparse autoencoders are not a universal solution but remain pragmatically useful and have many ongoing applications.

“fit for what networks are doing and so we should move towards using that.”

Claim: The speaker believes the manifold idea fits what neural networks are doing better than the alternatives under discussion and recommends moving toward its use.