Alex Zhang

“but there are some exceptions to this, I think, that are coming out.”

Expects: Some harnesses that differ from the broadly similar current designs are emerging. · no date given · not yet resolved

“yeah yeah yeah and there's like there's actually a surprisingly small space of optimizations that people do and there's actually not that many kernels per se that people are interested in optimizing and so we kind of had this thought that like if you had enough data like in the same way that code forces millions of problems if you could do this with gpu code like you could scale and automate kind of gpu kernel development which for researchers is a huge deal because i think one of the bigger bottlenecks like if you look like mamba for example like they released the paper with kernels because otherwise like you can't really use it in any meaningful way you know and not everyone has like a tree dow on their team so we're very interested in this uh kernel bench kind of spawned from that too”

Expects: Alex Zhang and others thought that collecting enough data could enable GPU-kernel development to be scaled and automated, analogously to how Codeforces’ large problem collection supports programming. · no date given · not yet resolved

“you like can sufficiently explore solutions to that problem but oftentimes like you know maybe you can burn like 100 billion or a trillion tokens on something but if you bring in someone who knows something about the problem they can uncover something for the model that would like erase that 1 trillion token spend i mean it's not it's not super clear like what exactly the trends are here but i think like there are so many problems in the wild still right now that we want to solve and like”

Expects: A knowledgeable person may identify an insight that lets a model solve a problem while spending fewer tokens—potentially avoiding as many as 100 billion or a trillion tokens. · no date given · not yet resolved

“an open air anthropic maybe like meta and gdm like you just you got to do something else you know like it's just it's the sad reality but i think i i actually think it's a good thing i'm very happy that like scale works and these companies will just keep doing it because it opens up like potentially new players like if they uncover something really really interesting because”

Expects: Large AI companies will keep pursuing scaling, which may create opportunities for new players if they discover something especially interesting. · no date given · not yet resolved

“given the purpose of harness for a model but in general like i think it's uh yeah i i hope to see very quickly or very soon like all of these games be in by newer models”

Expects: The speaker hopes newer models will be able to play all of the games in the suite very soon. · no date given · not yet resolved

“given the purpose of harness for a model but in general like i think it's uh yeah i i hope to see very quickly or very soon like all of these games be in by newer models”

Expects: Newer models will probably become able to play every game in the benchmark soon. · no date given · not yet resolved

“what the role of the harness is because harnesses can actually greatly increase the generalization capability of your model and the amount of data that it's given and this is not exclusive to RLMs I think there is a wide class of harnesses that are yet to be discovered that actually can yield similar properties and to extend this argument a little bit further I think what you can also extrapolate from this is like if I look at an RLM”

Expects: Many kinds of harnesses beyond RLMs remain to be discovered and may provide similar generalization benefits. · no date given · not yet resolved

“we will see beyond just like i'm going to design a new coding harness that uses a special form of compaction or something i think we can be a lot more creative here like there's there's so much we can do with these language models that i think we are just not doing and i'm like very excited about this because i think like i think we can get serious gains from very very opinionated and good harness design”

Expects: The speaker expects creative, strongly opinionated harness designs to produce serious gains by making greater use of language models, which current harness designs underuse. · no date given · not yet resolved

“yeah like most harness choices don't matter because all of these harnesses are the same but i will say like grock bot for example is actually quite different i think from my understanding than than how some of these other harnesses have been designed and i like that a lot um and i think it's clear from here at least that like i'm pretty sure anthropic or open ai are exclusively training on their harnesses they're probably not training on their competitors harness i mean i'd assume not because i don't know why they would do that but”

Expects: The speaker suspects that Anthropic and OpenAI train only on their own harnesses and probably exclude competitors' harnesses from training. · no date given · not yet resolved

“i think though as models get smarter or like as they get better this distinction becomes uh not that important that important in the sense that like if you take astra and you put it inside of open code like it's not gonna go crazy because i think it's like just sufficiently good and so i say this because the only benefits between these different harnesses is just cost for the most part and and i think like what i'm getting at here is if you plug these models into rlm's though”

Expects: The speaker thinks that as models improve, differences between harnesses will matter less, and that Astra is capable enough to work in OpenCode without behaving badly. · no date given · not yet resolved

“models that do this correctly they will be a lot more efficient as well i mean you can kind of just think through like why this is the case right oh”

Expects: Models that correctly perform RLM-like workflows will be substantially more efficient. · no date given · not yet resolved

“If an RLM breaks down its computation into like a kind of a meta harness of sorts or like a program that involves sub agents that like look at a local problem, every individual language model call over the course of this task is in distribution, even if the entire task itself is out of distribution.”

Expects: The speaker expects compositional harnesses to make each language-model call in distribution, even when the overall task is out of distribution. · no date given · not yet resolved

“i think this is a collaboration with base 10”

Expects: Alex Zhang thinks Harvey's legal RLM work may have been done in collaboration with Base10. · no date given · not yet resolved

“But the reason for this is like, I think we will start to see in the future probably one day and I wrote a blog about this, what we think of as a language model, like the thing that we query might actually be like a swarm or like a scaffold or like some. weird harness design that scales very well but the user just doesn't see it ultimately all the user sees is some front end version of this harness and yeah i i i think it's a relatively safe bet at least to make that this is what we will see”

Expects: Users may eventually be able to query a well-scaling swarm, scaffold, or harness as if it were a language model, while seeing only a front-end interface. · no date given · not yet resolved

“yeah so yeah there's there is this uh someone pointed to me some actually like academics have especially pl like programming languages people have some like very kind of cool ways of doing this and so like you know at some point maybe i'll i'll like work on this guess mostly”

Expects: The speaker may work in the future on advancing tool execution, building on promising approaches developed by academics, particularly programming-languages researchers. · no date given · not yet resolved

“doing some job i think it's like ridiculous that we can't do that and it's part of the reason is like the the format of a language model is not really amenable to that but i think you can shape a harness around it and do it and i think like this in itself is like ignoring the rlm stuff ignoring all the like you know what abstraction should we use like i just think someone can design a harness that can do this like i i i mean that's maybe Do long running but simple tasks and do them reliably.”

Expects: The speaker believes it is possible to design a harness that makes long-running, simple tasks reliable. · no date given · not yet resolved

“people's general workflows where you probably could just vibe code up some specific harness to help you do like automate this thing some examples are like automating finding like research papers and stuff like that but usually people will design like a specialized agent to help them do this kind of thing or like like they'll vibe code a harness like and just run it or like their slack slack bot or something but i almost think there's just like a standard form like just a harness that you just plug in like you don't it doesn't need to be designed for finding papers or like you just kind of tell it find this for me and you like plug it into that setting what i'm getting at is that i think there's a lot of easy things that can be automated and”

Expects: Many easy tasks in people’s workflows could be automated with a reusable general-purpose harness that users can direct, without designing a specialized agent for each task. · no date given · not yet resolved

“And I think we can actually approximate the human.”

Expects: The speaker believes models can be made to approximate human performance substantially more closely. · no date given · not yet resolved

“yeah i mean to me like also this aspect of like what is worth spending and betting my time on now because like maybe i spent a lot of time on this problem and then like in six months like a different solution kind of like maybe a new model comes out and it's like oh it's way better for this and so i do have to be careful i like you know you have to be conscious about like where you think things might be going um so yeah”

Expects: A different solution or a new model might emerge within six months and work much better for a problem Alex Zhang is considering investing time in. · 2 Oct 2026 – 2 Apr 2027 · not yet resolved