When AI Research Starts Moving Faster Than Human Research - Zhengyao Jiang

“To me, harness engineering is a more of a cheap way and efficient way to adapt the intelligence to a specific task.”

Claim: Harness engineering can adapt intelligence to a specific task more cheaply and efficiently.

“So back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views, claiming to demonstrate the first evidence of recursive self-improvement.”

Claim: In July, a London-based AI company posted a tweet that received about 1.8 million views and claimed to demonstrate the first evidence of recursive self-improvement.

“The thing is, not everybody bought it though.”

Claim: Not everyone accepted the claim that the tweet demonstrated evidence of recursive self-improvement.

“I'm Zheng Yaojiang. co-founder and the CEO of Wicco AI, where we build self-improving agents.”

Claim: Zheng Yaojiang is the co-founder and CEO of Wicco AI, a company that builds self-improving agents.

“Where we run auto-research on auto-research harness, and manage to discover a better auto-research harness, then we keep tuning in the last two years.”

Claim: Wicco AI ran auto-research on an auto-research harness, discovered a better harness, and continued tuning it over the last two years.

“Now when most people think about recursively self improving intelligence what they conjure up in their mind is a system that can rewire its own brain in a loop.”

Claim: People often imagine recursive self-improvement as a system repeatedly rewiring its own brain.

“But what if the system was only changing the code around the brain? Would that still count?”

Claim: Whether a system that changes code around its underlying model counts as recursive self-improvement is an open question.

“Maybe we care about the outcome, like does it actually improve the capability of the system, not a specific layer.”

Claim: The relevant criterion for self-improvement may be whether the system's capability improves, not which specific layer is changed.

“what about this small matter of Goodhart's Law? which is that when a measure becomes a target, it ceases to be a good measure.”

Claim: Goodhart's Law states that a measure ceases to be a good measure when it becomes a target.

“And more interestingly we find that it design mechanisms prevent itself from reward hacking.”

Claim: The team found design mechanisms intended to prevent its system from reward hacking.

“If you arrive R_S_I_ there is going to be uh technical singu singularity or like intelligence explosion. I don't think that would be the case.”

Claim: Zheng Yaojiang does not expect recursive self-improvement to result in a technological singularity or intelligence explosion.

“This project actually started uh about uh one and a half years ago.”

Claim: The recursive self-improvement project began about one and a half years before the conversation.

“Where we get interested in the concept of a recursive self-improvement, I mean historically all the research have diminishing returns.”

Claim: Research has historically exhibited diminishing returns.

“Historically, it's possible by mythology improvement, better tools, but human researchers, like human researchers' brain, for example, is always the key bottleneck.”

Claim: Human researchers' cognitive capacity is a key bottleneck, although research efficiency can be improved through better tools and other improvements.

“If we now have an autonomous research system, can we point the research topic to itself? So that it can improve its research efficiency.”

Claim: An autonomous research system could improve its research efficiency by directing research toward improving itself.

“And the hope is one day this improvement of efficiency can overcome the diminishing return. the curve of effort output turn from concave to convex.”

Claim: The speaker hopes that future research-efficiency improvements will overcome diminishing returns and make the effort-output curve change from concave to convex.

“They won't know how far we can go, but improving the efficiency of self. Differential research is definitely already on the roadmap.”

Claim: Improving the efficiency of self-directed research is already on the team's roadmap, but the team does not know how far it can go.

“And it's very, very simple. You give it a metric, you give it a problem and it will hill climb towards that problem.”

Claim: The research agent takes a metric and a problem as inputs and hill-climbs toward solving the problem.

“But what they did was they set it up recursively. So they actually like hill climbed towards improving the agent itself. And what that means is improving the harness. So, you know, the prompts. prompts and the tooling and you know all of the skill surface all of the things that the harness can do and they just left it running they just left it running and they saw what happened”

Claim: The team configured the agent to optimize itself by improving its harness, including its prompts and tools.

“There is an interesting thing we observe on self-evolved auto-research harness. We see the code it generated, exactly, it's like alien code, it's spaghetti.”

Claim: The code generated by the self-evolved auto-research harness appears highly unfamiliar and tangled to the speaker.

“But for some reason, it generalize really well.”

Claim: The code generated by the self-evolved auto-research harness generalizes well, according to the speaker.

“Of course, we have a set of benchmarks we're trying to hear time on. After that, we will test it on hold-out benchmarks. Those are public benchmarks like MLEBenchLite, ALEBenchLite. ALEBench is an algorithmic discovery benchmark from Salkana. MLEBench is the... OpenAI's machine learning engineering benchmark and we also tested on a very far out of distribution task called weather bench 2 essentially you're trying to build a physics-based forecasting engine to predict weather And the the weather bench is very far out of distribution from the training or optimization set of but it is still generalized really well.”

Claim: The team tests its system on hold-out benchmarks, including a weather-forecasting task far outside the optimization distribution, and reports that it generalizes well on that task.

“It is generalized better than our. w we would argue more elegant uh hand-tuned harness. We uh a bit torn uh on the experiment results, like uh to what extent We want to stick to more further principled and clean solutions we generated versus the solution that actually works better in practice.”

Claim: The self-evolved harness appeared to generalize better than the team's hand-tuned harness, but the team was conflicted because its generated solution was less principled and clean than a solution that performed better in practice.

“There's definitely space that we can improve over there, I think, like adding certain constraints. To the optimizer, two sets of constraints might be something like you regularize neural networks. How do you make it trying to find the most simple solution to the same problem?”

Claim: The team believes it may improve the optimizer by adding constraints that encourage simpler solutions, analogous to neural-network regularization.

“That's one direction. But another direction might be, okay, you just need to live with it because one of the burden that make people feeling it's a spec. kind of speculated code might be how much work the agent is done in the background, like it uh do one hundred experiments in our case on Auto-Research self-improvement, and that number of experiment is probably similar amount that we did uh in the last uh two years. So”

Claim: The team may need to accept difficult-to-understand agent-generated code because the agent can perform substantial background work, such as 100 experiments, which the speaker estimates is comparable to the team's experiments over the prior two years.

“I think in practice you still need those constraints.”

Claim: The team believes constraints are still needed in practice for its research system.

“Human and AI need to collaborate.”

Claim: Humans and AI need to collaborate in research.

“There are certain aspects of a human. intelligence are still superior than AI, I believe so.”

Claim: Some aspects of human intelligence remain superior to AI, according to the speaker.

“For example, In Aiden, where we send the agent to attend OpenAI's hiring challenge, we found the human is still much better at generating creative primitives. We found it's very important for human to build the first prototype that gives the agent the right search space, basically the initial abstraction.”

Claim: In the OpenAI hiring challenge, humans were much better than the agent at generating creative primitives, and a human-built initial prototype was important for defining the agent's search space.

“We anchor the search space, a bit like how you design a neural network architecture where you introduce inductive biases for the learning process. If the initial code base is based on a search-based scaffold,”

Claim: An initial code base can anchor an agent's search space, analogous to how neural-network architecture introduces inductive biases into learning.

“a bit like how you design a neural network architecture where you introduce inductive biases for the learning process. If the initial code base is based on a search-based scaffold, like eight, then a lot of ideas agent came up are actually... coming from search literature. But if you give the initial code base as a react agent,”

Claim: Using a search-based scaffold as the initial code base leads many agent-generated ideas to draw on search literature.

“react agent basically means you put all the context into the history and give the agent full flexibility, then a lot of the ideas come from From the agent, the outer loop agent, outer loop meaning the agent optimizing another agent.”

Claim: A ReAct agent places all context in its history and has full flexibility; in this setup, many ideas come from the outer-loop agent that optimizes another agent.

“You can run recursive self-improvement loop, but it's not strictly better than human doing R&D.”

Claim: At level zero, running a recursive self-improvement loop is not strictly better than humans performing research and development.

“And we believe all the previous public results are sitting on this level, level zero. And the level one is what we called...”

Claim: The speaker believes that all previous public results fall at level zero of the recursive self-improvement framework.

“not positive where you run the self-improvement loop and you found that the speed it self-improve is actually faster than human doing R&D to improve the system and that's where we are currently sitting at and in level one when we're saying improvement there's always”

Claim: At level one, a self-improvement loop improves the system faster than human R&D, and the speaker says their work is currently at this level.

“like a way to measure it like we measure it on a collection of downstream tasks but we didn't measure how good it is improving its capability of improving itself does the inner loop found by the outer loop actually can become a better outer loop and that's the what we call the level two”

Claim: At level one, improvement is measured using downstream tasks, not by directly measuring whether the inner loop improves the outer loop's self-improvement capability.

“Recursive self-improvement, we call it ignition.”

Claim: The speaker uses “ignition” to refer to recursive self-improvement.

“Because that's the necessary condition to the grand goal of recursive self-improvement, essentially the intelligence explosion scenario. Improved efficiency can overcome the...”

Claim: The speaker considers inner-to-outer-loop generalization necessary for recursive self-improvement, which they describe as the intelligence-explosion scenario.

“increase that difficulty because on level one you will still see diminishing return like the positive feedback loop from inner loop”

Claim: At level one, the positive feedback loop from the inner loop still exhibits diminishing returns.

“We also don't care much about it because we care about the outcome, like does it actually improve the capability of the system, not a specific layer.”

Claim: The speaker evaluates recursive improvement by whether the system's capability improves, rather than by which specific layer changes.

“And to be fair, I think eventual level three recursive self-improvement like it. When we actually reach the inflection point, I think there needs to be this self-referential loop on all different layers.”

Claim: The speaker thinks level-three recursive self-improvement will require self-referential loops across all layers when it reaches its inflection point.

“But for us, we care about harness layer, but at the same time, we don't think only model layer improvements count as RSI.”

Claim: The speaker's work focuses on the harness layer, and they do not define recursive self-improvement as requiring model-layer improvements alone.

“Yeah, the best agent is 8.85, we call it.”

Claim: The speaker says the best agent they found was named 8.85.

“There's a lot of changes, but to boil it down, there are search algorithm improvement, there are better... Context management system and there is a complete rewrite of all the prompts and more interestingly we find the design mechanisms prevent itself from reward hacking like the outer loop trying to prevent the inner loop from cheating the benchmark.”

Claim: The speaker says the best agent incorporated search-algorithm improvements, a context-management system, and a complete prompt rewrite.

“Context management system and there is a complete rewrite of all the prompts and more interestingly we find the design mechanisms prevent itself from reward hacking like the outer loop trying to prevent the inner loop from cheating the benchmark.”

Claim: The speaker says the agent designed mechanisms intended to prevent the inner loop from reward-hacking the benchmark, with the outer loop acting to prevent cheating.

“And it actually designs three-layer system against that, that includes a prompt level anti-cheating. This is like just a territory,”

Claim: The speaker says the agent designed a three-layer anti-cheating system that included a prompt-level layer.

“And there are set of hard-coded rules that... Basically check if there is evidence of cheating from the code, the inner loop agent generated.”

Claim: The speaker says hard-coded rules checked inner-loop-generated code for evidence of cheating.

“The more interesting one is it also trying to filter out those cheating solutions from some statistical property. For example, if one solution is way too far out of the average of its peers, then it will think, this might have some reward hacking happening.”

Claim: The speaker says the agent used statistical outlier detection to filter possible cheating solutions, treating solutions far from their peers' average as potential reward hacking.

“But what's interesting is first that mechanism evolved, like designed in early stages of the process, and it's actually broken in the later stage.”

Claim: The speaker says the agent's statistical anti-cheating mechanism emerged early in the process but stopped working in a later stage.

“I think the later stage of a code changes just uh introduce a a bug. So that layer stops working. Um that's also sort of a the spa spaghetti code base we property we just mentioned.”

Claim: The speaker thinks a bug introduced by later code changes caused the anti-cheating layer to stop working.

“In a loop and outer loop are both optimizing towards a similar but different goals. So in all our benchmarks used for heel climbing, we have a public set and a private set. The inner loop agent will only look at the public set, whereas the outer loop will look at the aggregated score of the from the private set. So those two agents are optimizing at a kind of different level of objective.”

Claim: The speaker says the inner-loop and outer-loop agents optimize toward objectives that are similar but not identical.

“we have a public set and a private set. The inner loop agent will only look at the public set, whereas the outer loop will look at the aggregated score of the from the private set.”

Claim: In the benchmark setup, the inner-loop agent accesses only the public set, whereas the outer loop evaluates the aggregated score on the private set.

“So whenever inner loop agent trying to cheat, our evaluation protocol will show, okay, although you achieve good performance on public set, your private set performance is actually going down.”

Claim: The speaker says the evaluation protocol detects attempted inner-loop cheating by showing that public-set performance may be good while private-set performance falls.

“So the outer loop managed to learn about it and uh trying to reduce the gap between public and private?”

Claim: The speaker says the outer loop learned from the public-private performance gap and attempted to reduce it.

“That's why there's actually two order of generalization we are testing here.”

Claim: The speaker says the experiment evaluates two distinct orders of generalization.

“There's a first order generalization, which is a generalization from public side to private side of the same benchmark,”

Claim: The first order of generalization measures transfer from a benchmark's public split to its private split.

“but in order to evaluate the whole Meta learning system. We need to test it on a hold out set of benchmarks so that the out those benchmarks are tasks that even auto loop never sees and for all of those benchmarks they are still private and a public split so we tested the discovered inner loop agent on those hold out tasks that's the protocol”

Claim: The speaker says they evaluate the meta-learning system by testing the discovered inner-loop agent on held-out benchmarks unseen by the outer loop, each with public and private splits.

“The thing is the more outer loop you have, it will become more expensive.”

Claim: Increasing the number of outer-loop iterations increases the experiment's cost.

“If the inner loop agent being discovered or harness being discovered can be applied back to the outer loop, then you don't need to add another layer.”

Claim: If an inner-loop-discovered agent or harness can be applied to the outer loop, the process can continue without adding another layer.

“Like essentially every few steps, you promote the inner loop agent to the outer loop.”

Claim: The speaker proposes promoting the inner-loop agent to the outer loop every few steps.

“You can run this for infinite number of steps, apply infinite number of iterations without adding. another layer.”

Claim: The speaker says the process can run indefinitely, applying infinitely many iterations without adding another layer.

“We haven't experimented that, but it's a really promising direction we are exploring.”

Claim: The team has not experimented with giving episodes shared memory or access to prior episodes, but is exploring this as a promising direction.

“I think there are some more fundamental problem formulation problem. To address, yeah, currently we frame the problem as a stateless uh optimization problem. Is that the right framing at all? Um probably not. There is a lot of uh more flexible framing.”

Claim: The team currently frames the problem as stateless optimization, but the speaker thinks this is probably not the right formulation and that more flexible formulations exist.

“Yeah, we do see that. Um sometimes the um agent just keep trying all the search algorithms um ideas, even though I mean to some extent uh if you try it uh so many times you probably learn a lesson that this direction is not going to work in general, but it's it's still trying.”

Claim: The agent sometimes repeatedly tries search-algorithm ideas even though repeated attempts could probably teach it that a direction will not work in general.

“the best agent search policy is really complex and it's actually combining multi-arm bandit and a weird strategy of anti-saturation, which is like, oh,”

Claim: The speaker says the best agent's search policy combines a multi-armed bandit algorithm with an anti-saturation strategy.

“It organized the search process with a few lineage, or you can call them islands, and it will allocate budgets between those lineage with multi-arm bandit algorithms. And it also did a bit of customization on that multi-arm bandit algorithm.”

Claim: The agent organizes search into multiple lineages (also called islands) and allocates budgets among them using a customized multi-armed bandit algorithm.

“And whenever one lineage gets saturated, it will create a new lineage, like a new island with a fresh context, but still extending some of the ideas from the old lineage.”

Claim: When a search lineage saturates, the system creates a new lineage with fresh context that carries forward some ideas from the old lineage.

“So it's a really... complex strategy, but it seems like it works.”

Claim: The speaker believes the agent's complex search strategy appears to work.

“Of course there are gray areas. We think It is one step of a recursive self-improvement because you find a better algorithm. In that case, I believe it's like a matrix multiplication algorithm.”

Claim: The speaker considers AlphaVol a possible single step of recursive self-improvement because it finds a better algorithm, which he believes is a matrix multiplication algorithm.

“you can argue it managed to improve itself. Like if you apply that back to the AI system that... actually doing the like inference of the language models like you essentially make it faster so we think You can argue that's like one step of recursive self-improvement, although we don't know because I don't think they actually test it.”

Claim: The speaker says applying the improved algorithm to a language-model inference system would make it faster and could count as one step of recursive self-improvement, although he does not think this was tested.

“We are trying to optimize the whole system on end-to-end testing.”

Claim: The team is optimizing the entire system using end-to-end testing.

“So what they are trying to do is they have a An agent optimizing a coding agent, so there's a meta agent optimizing a coding agent on general coding tasks, and that coding agent is part of the agent doing the optimization.”

Claim: The speaker describes the Darwin-Godel machine as using a meta-agent to optimize a coding agent on general coding tasks, with that coding agent forming part of the optimization system.

“In their case, there's only two levels, like there's an agent optimizing. another agent doing coding whereas in our case there are three levels there's an outer loop agent there's a interloop inner loop agent both are doing auto research and they there's another layer which is the downstream tasks and some of them are actually harness engineering tasks”

Claim: The speaker contrasts the Darwin-Godel machine's two levels with their system's three levels: outer- and inner-loop agents conducting automated research, and downstream tasks that include harness engineering.

“And I think the interesting part of the thread of Darwin-Godel machine is the lower layer, the task layer, actually have a component of the search algorithm. And they say, if I improve this component in the unit test. The performance can actually generalize back to the whole search algorithm.”

Claim: The speaker highlights that the Darwin-Godel machine's task layer includes a search-algorithm component, and that improving it in a unit test can generalize to the full search algorithm.

“Like when we start this project, we're always thinking, okay, we should optimize the end-to-end performance of the whole auto-research system, and we have our auto-loop agent pointing to that goal.”

Claim: The team designed its auto-research project to optimize the whole system's end-to-end performance, with the outer-loop agent pursuing that objective.

“I mean, first of all, we will have like a proper paper later, at least the first PDF version where we are going to give credits to. the academia on this whole thread of meta-harness optimization.”

Claim: The team plans to publish a paper, beginning with a PDF version, that credits academic contributions to meta-harness optimization.

“To us, what we care about is the outcome of the system, not like the conceptual differences, even though I just explained some conceptual differences. To us, we care about the outcome of it actually bending the curve. of recursive self-input, like of the R&D efforts, we see this is the first evidence that the curve actually starting get”

Claim: The team prioritizes whether its system improves the trajectory of recursive self-improvement in research and development over conceptual distinctions.

“Of course, we are not claiming we're inventing this meta. optimization idea which can date back to like 20-30 years.”

Claim: The team does not claim to have invented meta-optimization, which the speaker says may date back 20–30 years.

“I think a search space, like maintaining a good abstraction don't necessarily reduce the search space.”

Claim: The speaker argues that maintaining a good abstraction does not necessarily shrink the search space.

“Like if the agent is smart enough, it should be able to escape from the current abstraction or change it a little bit whereas generating really complex code. I think it would direct the search in a more kind of low level. changes direction, which is not necessarily a good thing.”

Claim: The speaker thinks a sufficiently capable agent should be able to escape or modify its current abstraction; generating very complex code may instead steer search toward lower-level changes, which is not necessarily beneficial.

“But I have my human bias here, like compare the code we manually write versus the generate solution, I always prefer our own code, but to some extent it don't work as well.”

Claim: The speaker says his human bias makes him prefer manually written code to generated solutions, although generated solutions can work better.

“Back then we realized reward hacking is one of the big problem of auto research agents because that's especially severe on tasks like GPU kernel engineering where the agent always figure out of the complex way to”

Claim: Reward hacking is a major problem for automated research agents, particularly in GPU-kernel engineering tasks.

“Back then we realized reward hacking is one of the big problem of auto research agents because that's especially severe on tasks like GPU kernel engineering where the agent always figure out of the complex way to make the unit test perform much better and when you deploy that kernel into the actual end-to-end inference of a model the performance are actually getting worse and specbench is a extrapolation of that to more general purpose software engineering and we extend the protocol we found on.”

Claim: On GPU-kernel tasks, an agent may optimize performance on a unit test while making the kernel perform worse in end-to-end model inference.

“make the unit test perform much better and when you deploy that kernel into the actual end-to-end inference of a model the performance are actually getting worse and specbench is a extrapolation of that to more general purpose software engineering and we extend the protocol we found on. GPU kernels like we maintain a public set which is a more like a unit test for the software system and a holdout set where we test the capability of the agent on capability of the generated software on the composition of those unit tests like simulating real-world usage”

Claim: SpecBench evaluates general-purpose software-engineering agents with a public unit-test-like set and a holdout set that tests generated software on combinations of tests intended to simulate real-world use.

“The first interesting finding is the longer you run for the agent or the more complex the code base is, the larger reward hacking rate the agent will show, because when the code base are becoming more complex, the agent just have more space to reward hack.”

Claim: In the speaker's evaluation, agent reward-hacking rates increase with longer run time and greater codebase complexity.

“because when the code base are becoming more complex, the agent just have more space to reward hack.”

Claim: Greater codebase complexity gives an agent more opportunities to exploit the reward measure.

“And the second interesting thing is the public set performance are rather consistent across different models. So if you have a like smaller models working on the same task with the auto research harness, it can reach the same public score.”

Claim: On the evaluated task, smaller models using the same automated-research harness can achieve the same public-set score as other models, and public-set performance is fairly consistent across models.

“But the larger models. always have in our test always have a smaller reward hacking rate so the solution it generated generalized better”

Claim: In the speaker's tests, larger models had lower reward-hacking rates and produced solutions that generalized better.

“It's because of the mechanism. So for the inner loop agent, we just fixed the model. And the outer loop agent managed to discover a different mechanism to prevent the same. model from reward hacking.”

Claim: The speaker attributes the lower reward-hacking rate to the agent mechanism: the inner-loop model was fixed, while the outer-loop agent discovered another way to prevent it from reward hacking.

“I think the only ice case is quite dramatic, like of the new reward hacking behaviors.”

Claim: The speaker considers the cited case of new reward-hacking behavior quite dramatic.

“So what we see is on the Detectable reward hacking, actually the new models are getting better and better.”

Claim: Newer models are improving at detectable reward-hacking cases.

“But because of the capability frontier also progress so quickly, there are some reward hacking behavior that's not even detectable given previous protocol.”

Claim: Rapid capability progress can produce reward-hacking behaviors that existing detection protocols do not detect.

“I think we just need to keep evolving the detection. protocol in the future to capture those emerging real-world hacking behavior.”

Claim: Detection protocols should continue evolving to capture emerging real-world agent-hacking behaviors.

“And I think the responsibility should mostly be on the model developer side.”

Claim: Responsibility for addressing agent reward-hacking behavior should fall mostly on model developers.

“Yeah, on harness layer, like a developer can also add additional guardrails if they like.”

Claim: Developers can add optional additional guardrails at the agent-harness layer.

“So basically what it did is it write a gigantic monkey patch to our eval script. And at first we think, okay, this must be reward hacking. Why are you attaching the eval script? But eventually we found that it's just fixing a bug.”

Claim: In the speaker's example, an agent made a large monkey patch to the evaluation script that initially looked like reward hacking but was ultimately found to fix a bug.

“to some extent, I think it will become harder and harder for humans to detect those through reward hacking because the agent's behavior is just becoming so sophisticated.”

Claim: The speaker expects human detection of reward hacking to become increasingly difficult as agent behavior grows more sophisticated.

“A lot of the development... bottleneck shift to understanding of what agents generates.”

Claim: A development bottleneck is shifting toward understanding agent-generated outputs.

“I think one of the way to address them might just be defining a good abstraction like instead of trying to understand all the code just like previously we developing neural networks we don't try to understand all the ways although there is like a progress on that as well.”

Claim: One possible approach to understanding agent-generated code is to define useful abstractions instead of trying to understand every line of code.

“But the general principle is we define a good input-output contract, we define a good way to measure its behavior both on generalization and maybe more compacts are out of distribution generalization.”

Claim: A useful approach is to specify an input-output contract and evaluate behavior for generalization, potentially including out-of-distribution generalization.

“I think that might be the way in the future software engineers working with agent-generated code as well.”

Claim: The speaker thinks software engineers may use input-output contracts and behavioral evaluation when working with agent-generated code in the future.

“And basically what that means is if you hill climb on one objective, You blind yourself to interesting stepping stones. So it's an interesting situation,”

Claim: Blindly optimizing a single objective can prevent a search process from noticing interesting intermediate ideas or opportunities.

“I think from the perspective of open-ended learning, I agree that our current system probably don't have that property of collecting stepping stones. And of course, there's a kind of a whole new domain for improvement.”

Claim: Current AI systems probably do not collect useful intermediate ideas or capabilities as stepping stones.

“The behavior of a cracking stepping stone is you need to have a large collection of tasks.”

Claim: Collecting useful stepping stones requires a system to work across a large collection of tasks.

“When you're optimizing for a single task, it's always better to just do greedy hill climbing if you consider the efficiency.”

Claim: For optimization of a single task, greedy hill climbing is more efficient than diversity-seeking approaches.

“Actually, during self-improvement process of AdaSquare, the outer loop try inject all sorts of diversity search approaches, but none of them really improve the efficiency.”

Claim: In AdaSquare's self-improvement process, diversity-search methods introduced in the outer loop did not improve efficiency.

“And I think that's expected because we are only optimizing a single task from scratch.”

Claim: The speaker expected diversity-search approaches not to improve efficiency because AdaSquare was optimizing only one task from scratch.

“But if you have a whole large collection of tasks, Sometimes you cracking interesting pieces like a primitive ideas from one task and although it is not improving the performance on the current task you can potentially bring that idea to future tasks and that's actually my own definition of interestingness or open-endedness which is an idea need to be”

Claim: Across a large collection of tasks, an idea found while working on one task can be useful for future tasks even if it does not improve performance on the task where it was discovered.

“Sometimes you cracking interesting pieces like a primitive ideas from one task and although it is not improving the performance on the current task you can potentially bring that idea to future tasks and that's actually my own definition of interestingness or open-endedness which is an idea need to be or an object or artifact is interesting if it is novel and it also being useful for a larger collection of tasks.”

Claim: An idea or artifact is interesting when it is novel and useful across a broad collection of tasks.

“There are apparently two ways to do it, like you can trying to optimize the hardness against all of the tasks or You can find a task and you optimize harness just for that one task.”

Claim: The speaker describes two approaches to harness optimization: optimize difficulty across all tasks, or optimize it for a single selected task.

“To me, harness engineering, most of the interesting part are actually the later.”

Claim: The speaker considers the later stages of harness engineering to be its most interesting part.

“if you can automatically tune the harness, it's almost a continuation of post-training.”

Claim: Automatically tuning an AI harness is nearly a continuation of model post-training.

“connecting it back to a lot of people's complaints about harness engineering is that, oh, I designed a harness, but later there's a new model and it sort of absorbs all the ideas from it.”

Claim: People complain that a later model can absorb the ideas in a manually designed harness.

“But interestingly enough, no people complain about it for post-training. Like, do you expect post-training of GPT? fossil waste generalized to GPT-5. Nobody asked it.”

Claim: People do not complain when a later model absorbs the results of earlier post-training.

“I think it's because historically harness engineering are pretty manual.”

Claim: Harness engineering has historically been largely manual, according to the speaker.

“And it's not very scalable with compute.”

Claim: Historically, harness engineering has not scaled well with increased compute.

“So people want the harness they spend like a few months designing can generalize to the next model, but it's really not true.”

Claim: People often want a harness designed over several months to generalize to the next model, but the speaker says it generally does not.

“Now I think things change quite a bit that harness can be designed automatically.”

Claim: Harnesses can now be designed automatically, according to the speaker.

“Okay, you spent two, three days tuning your harness based on the new model. It's no longer like that to build with human hours.”

Claim: Adapting a harness to a new model can involve two or three days of tuning and is no longer limited to human labor hours.

“I do have this perspective that a lot of knowledge is path dependent.”

Claim: Much knowledge depends on the path by which it was acquired or developed.

“A lot of people criticize people using LLMs to do research because they say you're taking the information out of its context. You don't understand why it came about.”

Claim: Some critics argue that using LLMs for research removes information from its context and obscures why that information or idea arose.

“So that's part of the reason why some heuristics are required, because it's not like you can just say, oh, use UCB. There has to be some kind of framing to explain why you should use it. And then the LLM becomes coherent in the autonomous setting.”

Claim: In autonomous settings, an LLM needs explanatory framing for a heuristic such as UCB to use it coherently; simply instructing it to use UCB is insufficient.

“It's just the throughput to recombine those ideas and attaching them are extremely high. To some extent, those language models don't have a deep understanding of like how those ideas came from and how can I use the... Like the process those ideas get discovered to design the next generation of algorithm like in a very creative way.”

Claim: Language models lack deep understanding of how ideas were discovered and how those discovery processes can be used creatively to design next-generation algorithms.

“It's just the throughput to recombine those ideas and attaching them are extremely high. To some extent, those language models don't have a deep understanding of like how those ideas came from and how can I use the...”

Claim: Automated research systems can recombine existing ideas and attach them at extremely high throughput.

“What we found though is when you ask the model to come up with something new, it's always to fix a few number of items.”

Claim: When prompted to generate something new, the model tends to focus on fixing only a few items.

“I'm wondering though if it's more of a fact that basically we don't have continuous learning and we only have a single instance of language models like it's very centralized therefore maybe those ideas initially you first prompt it maybe those ideas are interesting but if everyone see the same idea so many times then it just become less interesting.”

Claim: The speaker suspects that without continual learning, a centralized single-instance language-model setup may cause initially interesting ideas to become less interesting through repeated exposure to the same ideas.

“And I do feel like if we don't solve the continual learning, then that problem probably won't be addressed.”

Claim: The speaker believes the problem of repeated, less-interesting ideas probably will not be addressed unless continual learning is solved.

“every human researcher has their unique neural network of ways. And as a collection, human can come up with a lot of different creative ideas.”

Claim: Individual human researchers have distinct networks of thought, and the research community can collectively generate many different creative ideas.

“And that idea of pools is very valuable, I think, as a research community.”

Claim: A diverse pool of ideas is valuable to a research community, according to the speaker.

“OpenAI hosted a hiring challenge where they tried to put a load of researchers into a... competition of a training a small language model under 16 megabytes constraint. Yeah, we send a experimental harness,”

Claim: OpenAI hosted a hiring challenge in which participants competed to train a small language model within a 16-megabyte constraint.

“competition of a training a small language model under 16 megabytes constraint. Yeah, we send a experimental harness, auto research harness to the competition and the agent runs about 22 days and eventually it made seven records get accepted by OpenAI in the best individual contributor only made three.”

Claim: In OpenAI's 16-megabyte model-training challenge, the speaker's team submitted an experimental auto-research harness whose agent ran for about 22 days and produced seven accepted records, compared with three from the best individual contributor.

“So that's also a moment we found out the autonomous system can be so powerful.”

Claim: The competition convinced the speaker that autonomous systems can be very powerful.

“It's not just a hill climbing a benchmark.”

Claim: The autonomous system's work was not merely benchmark hill-climbing, according to the speaker.

“The fact that first the OpenAI accepted those PRs. The second, so many human researchers building on top of those PRs. of AI agent generated is just very fascinating to us because to some extent knowledge AI agent generated only”

Claim: The speaker found it significant that OpenAI accepted the agent-generated pull requests and that many human researchers subsequently built on them.

“of AI agent generated is just very fascinating to us because to some extent knowledge AI agent generated only It's only helpful if it gets into the lineage of human innovation.”

Claim: Knowledge generated by AI agents is useful only when it becomes part of the lineage of human innovation.

“We think this is actually a sandbox of human AI collaboration.”

Claim: The speaker views the competition as a sandbox for human-AI collaboration.

“Yeah, the outer loop is actually fixed.”

Claim: The outer loop is fixed.

“Yeah, the outer loop is actually fixed. And the hope is this inner loop search policy can generalize to the outer loop.”

Claim: In the described system, the outer loop is fixed, and the goal is for the inner-loop search policy to generalize to the outer loop.

“And the hope is this inner loop search policy can generalize to the outer loop.”

Claim: The speaker hopes that the inner-loop search policy will generalize to the outer loop.

“We did try that, we replay the search process from step 15. So in general we run about 100 steps.”

Claim: The team typically runs about 100 search steps and tested replaying the process starting at step 15.

“So in general we run about 100 steps.”

Claim: The speaker's team generally runs about 100 steps in the described process.

“And in step 15 we use the vanilla outer loop and compare that to the best agent we found inner loop agent we found in first 50 steps and what we found is the inner loop step forty seven I think eventually it converge slightly faster than previous art loop but eventually the solution they found have a similar performance so that's also the reason we didn't claim we reached the level two recursive self-improvement which is like the improver managed to improve its own capability of thing”

Claim: In the reported comparison, the inner-loop agent found around step 47 converged slightly faster than the previous outer loop, while both ultimately reached solutions with similar performance.

“forty seven I think eventually it converge slightly faster than previous art loop but eventually the solution they found have a similar performance so that's also the reason we didn't claim we reached the level two recursive self-improvement which is like the improver managed to improve its own capability of thing of improving in the loop.”

Claim: The team did not claim level-two recursive self-improvement, which the speaker describes as an improver improving its own ability to improve.

“forty seven I think eventually it converge slightly faster than previous art loop but eventually the solution they found have a similar performance so that's also the reason we didn't claim we reached the level two recursive self-improvement which is like the improver managed to improve its own capability of thing of improving in the loop.”

Claim: The speaker's team did not claim to have achieved level-two recursive self-improvement, defined as an improver improving its own capability to improve within the loop.

“forty seven I think eventually it converge slightly faster than previous art loop but eventually the solution they found have a similar performance so that's also the reason we didn't claim we reached the level two recursive self-improvement which is like the improver managed to improve its own capability of thing”

Claim: The speaker reports that the inner-loop agent from step 47 eventually converged slightly faster than the previous outer-loop approach, while the resulting solutions had similar performance.

“I don't think that would be the case.”

Claim: The speaker does not think artificial superintelligence is imminent.

“Even if you arrive recursive self-improvement, it will take a long time to there.”

Claim: The speaker expects that, even if recursive self-improvement is achieved, reaching it will take a long time.

“Like, you gradually bend the curve.”

Claim: The speaker expects progress toward recursive self-improvement to bend the development curve gradually.

“It's just like, okay, even if you have a GPT 3.5, it's very smart chatbot, but it's not going to be AGI that can solve every problem.”

Claim: The speaker characterizes GPT-3.5 as a very smart chatbot, but says it is not artificial general intelligence capable of solving every problem.

“There's still a lot of human in the loop part of research, including defining good abstractions.”

Claim: The speaker says that much of research still involves humans in the loop, including the work of defining good abstractions.

“Defining what is good, essentially defining evals and constraints attached to the eval. And the third one is coming up creative primitives.”

Claim: The speaker says that human research work includes defining what counts as good by specifying evaluations and their constraints.

“Aiden parameter golf competition, we still found most of those creative primitives actually coming from human.”

Claim: The speaker reports that most creative primitives in the Aiden parameter golf competition still came from humans.