⚡ An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio

“Yes. So the thing that I talked about last time was this idea that we had these large collections of screens and that they were big enough environments and counterfactually dense enough that you could take agents and drop them in there and then give them tasks, and they could go do those tasks on a limited budget in this landscape that we had measured completely.”

Claim: Ian Quigley says Leash Bio's collections of screens are sufficiently large and counterfactually dense to let agents perform assigned tasks on a limited budget in a completely measured environment.

“And so the flagship problem that we spent a lot of time over the past couple months on is a structure activity relationship problem. And the idea there is like, okay, I've got some molecules, and I know that they bind, maybe they bind to this protein that I want them to. How do I change them to make them better? This is a very common task for medicinal chemists to do. And SAR, as it's sort of abbreviated to, also includes other stuff like certain drug-like properties, you know, does it get metabolized, all these things.”

Claim: Ian Quigley describes structure-activity relationship work as improving molecules and assessing other drug-like properties, including metabolism.

“And so the flagship problem that we spent a lot of time over the past couple months on is a structure activity relationship problem. And the idea there is like, okay, I've got some molecules, and I know that they bind, maybe they bind to this protein that I want them to.”

Claim: Ian Quigley says Leash Bio's flagship agent task, developed over the preceding couple of months, is a structure-activity relationship problem.

“We're not doing those right now. We're just doing, can we find molecules that bind better to the thing that we're interested in? And so what we do is we take mode-- we take an agent, like a language model, and then we say, "Here's a catalog that you could look at. You have a budget of a hundred molecules. We're not gonna tell you anything except the target, and, uh, go find the best binder you can." And so different agents try different stuff.”

Claim: In Ian Quigley's setup, an agent receives a molecule catalog, a budget of 100 molecules, and the target, and is asked to find the best binder without other information.

“We're not doing those right now. We're just doing, can we find molecules that bind better to the thing that we're interested in? And so what we do is we take mode-- we take an agent, like a language model, and then we say, "Here's a catalog that you could look at.”

Claim: Ian Quigley says Leash Bio's current task focuses on finding molecules that bind better to a target and does not yet address other drug-like properties.

“They have different search policies. They have different ways of wanting to spend their budgets. And what they'll do is they'll put out in readable language, uh, SAR hypothesis about why they think these molecules are working or not working.”

Claim: The agents in Ian Quigley's experiment differ in their search policies and how they allocate their molecule budgets.

“They have different search policies. They have different ways of wanting to spend their budgets. And what they'll do is they'll put out in readable language, uh, SAR hypothesis about why they think these molecules are working or not working.”

Claim: The agents in Ian Quigley's experiment state readable-language structure-activity hypotheses explaining why molecules may or may not work.

“Uh, and then they'll go in, and with this limited experimental budget, they'll test those hypotheses. And because it's all pre-measured, you can roll out a lot of these. We're up to about ninety thousand.”

Claim: Ian Quigley says agents test hypotheses with a limited experimental budget, and because the data are pre-measured, the team can run about 90,000 rollouts.

“Yeah. So the preprint will probably be out at the same time that this episode comes out. It is pretty ready. And, uh, what we, what we show in there is, uh, that we get great separation of agents across, uh, model families and at the levels, that kind of stuff.”

Claim: Ian Quigley says Leash Bio's preprint shows clear performance separation among agents across model families and model levels.

“But another thing that's really interesting is because we have not just the results that you could use as an eval for how the agent's doing the job, we also know the results of the entire measured landscape or world.”

Claim: Ian Quigley says Leash Bio's evaluation includes both agent results and results for the entire measured environment.

“And so then you can start to interrogate how good is the agent at making scientific decisions. Because maybe it only got to, like, the thing that's half as good as the best thing in the entire environment.”

Claim: Because the full environment is measured, Ian Quigley says they can assess agents' scientific decisions by comparing their results with the best result available in that environment.

“And so there we see all kinds of interesting behaviors like agents that give up early, uh, want to buy molecules that don't agree with the rest of their hypotheses, ones that keep retesting the same molecules over and over despite having a limited budget, and also when it's good to explore versus exploit.”

Claim: Ian Quigley says their evaluation reveals agent behaviors including early abandonment, choices inconsistent with stated hypotheses, repeated testing of the same molecules, and varying exploration-exploitation decisions.

“And it turns out that more performant models are much better at abandoning kind of unproductive neighborhoods. And that was actually a really interesting surprise for us. Yeah.”

Claim: Ian Quigley reports that more performant models abandon unproductive search neighborhoods more effectively, a result that surprised the team.

“We don't know that at all yet. All we know is that they're abandoning these crappy neighborhoods more quickly. Uh, and we definitely see a progression of the, like, a, a more willingness to jump around overall, uh- ... for more performant models, but they especially abandon the, the lousy places fast. And then the really poor ones stick around.”

Claim: Ian Quigley says the reason more performant models leave poor search neighborhoods faster is not yet known, although they are observed to switch locations more readily and abandon poor areas sooner.

“Exactly. Exactly. Yeah. And what happens with the, the dumber models is that they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere, right? Like, uh, these two, they have a low score, but I feel good about them, and so I'm gonna keep trying stuff. And then, uh, and then the smarter models will abandon the entire process. We're gonna go somewhere else.”

Claim: Ian Quigley says less capable models persist with small changes to unpromising molecules, whereas smarter models abandon that search process and move elsewhere.

“Yeah. They think about all kinds of stuff, they'll say. You know, when, when we release all these transcripts, like everyone to look at, one of my favorite things to do is just take, you know, large numbers of transcripts and just dump them into LLMs and just have a look at the different failure modes and stuff like that.”

Claim: The speaker likes to analyze large collections of released transcripts with LLMs to inspect different failure modes.

“But you'll see, yeah, I think that this ring is good, and I think that this particular nitrogen is painful or, you know, whatever. And they'll, they'll say these things, uh, and then they're very plausible, you know, medicinal chemistry narratives.”

Claim: The speaker says the models generate plausible medicinal-chemistry narratives about molecular features, including whether a ring is good or a nitrogen is painful.

“... like, you know, not a ton, but because, you know, we have, we have a large number of them. But when we get medicinal chemists to look at it, they're like, "Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign."”

Claim: Medicinal chemists who reviewed the model outputs judged them to be plausible ideas for conducting the relevant part of a structure–activity relationship campaign.

“I, I, I like, I like this plan. We did do a game a little bit like this for Name That Chemist. Um, we would have to probably boil it down to have a smaller number of building blocks because right now to have a chemist go in there, you can imagine that, uh, if you're a medicinal chemist, you might have this suite of things that you might wanna try over time, like put a methyl here, like move this around.”

Claim: Medicinal chemists may want to try a range of molecular modifications over time, such as adding a methyl group or moving a feature to another position.

“I, I, I like, I like this plan. We did do a game a little bit like this for Name That Chemist. Um, we would have to probably boil it down to have a smaller number of building blocks because right now to have a chemist go in there, you can imagine that, uh, if you're a medicinal chemist, you might have this suite of things that you might wanna try over time, like put a methyl here, like move this around.”

Claim: The speaker's team previously made a game called Name That Chemist that was somewhat similar to the proposed medicinal-chemistry game.

“And when we have these pre-measured environments, we are going to be restricted to the things that we did actually measure, and so then we would have to be asking the medicinal chemist, "Oh, well, you can't do whatever you want, you can just do a lot of stuff."”

Claim: The team's pre-measured environments limit users to the interventions that the team has actually measured.

“They have to know what the stuff is, yeah. So we'd have to figure out a way to surface it to them in a meaningful way, but I kinda like where you're going with this.”

Claim: The team would need to present the available measured options meaningfully so medicinal chemists can understand them.

“Like, we could definitely shrink it down to a smaller number of options. So we should do that.”

Claim: The speaker says the proposed medicinal-chemistry game can be reduced to fewer options and recommends doing so.

“Let's see. I was looking at the plot a minute ago. It's, it's kind of embarrassing that I forgot already. Uh, right now, uh, the champion is Astra 6.”

Claim: At the time of the conversation, Astra 6 was the leading model on the speaker's evaluation.

“Not a big shock. Sonnet 5.5 is actually pretty good too.”

Claim: The speaker considers Sonnet 5.5 to perform well on the evaluation.

“Yeah, Sonnet 5 is both very expensive and not very performant.”

Claim: The speaker says Sonnet 5 is very expensive and performs poorly on the evaluation.

“We don't yet, no. That's, that's one thing that actually, has actually been a little infuriating, uh, as you guys can appreciate. So when we talked last two months ago, we were starting these rollouts to see what happened, and then there's been a new frontier model drop, what, like once every 11 days?”

Claim: The speaker says that since the team's rollout work began two months before the conversation, new frontier models had been released roughly every 11 days.

“Yeah. So like, every time I'm, "Oh, we're gonna ship this. We need to drop this public stuff." And like, some other new stupid model shows up, it's like, "Ah."”

Claim: The speaker says that each time the team is preparing to release its public material, another new model appears.

“Yeah. Yeah. Right. Good, good luck, Ian. So yes, so we are excited to put in more in here. Uh, and we also have run a large number of, of the open source ones as well.”

Claim: The speaker's team plans to add more models to its evaluation and has already evaluated many open-source models.

“So we've thrown a bunch of smaller Quinns in there and GLMs and Kitty and DeepSeeks and that kind of stuff. So there's a lot of these.”

Claim: The speaker's team has evaluated smaller Quinn models, GLM models, Kitty, and DeepSeek models.

“They're, uh, DeepSeek V4 Flash was pretty good. Not that bad. Yeah, maybe I should just send you guys, like, the full chart of how things are going, the data.”

Claim: The speaker judged DeepSeek V4 Flash to perform reasonably well on the evaluation.

“... right, right. It does, it does kind of look like that, and that's kind of why we made it like that. And then, I think one thing I mentioned earlier that has been very interesting to us is about this time last summer, we had started an internal project to do, uh, supervised fine-tuning on some of these language models to get them to understand SMILES, because none of the existing frontier models were any good at it.”

Claim: The speaker's team began an internal supervised fine-tuning project around the previous summer to teach language models SMILES, because existing frontier models performed poorly at understanding SMILES.

“So like, none of the Anthropic or, or OpenAI models at the time could even convert SMILES to non-canonical SMILES very effectively. And so we're like-”

Claim: At the time described, Anthropic and OpenAI models were not effective at converting SMILES strings into non-canonical SMILES strings.

“Yes. Yes, that's right. Uh, uh, I think we mentioned it a little earlier when we recorded last, but it's a string, and molecules can be forked or, or circles or, or, or both put together, right?”

Claim: The speaker describes SMILES as a string representation of molecules, whose structures can include branches, rings, or both.

“These different things. And so a SMILES is basically a graph traversal across the molecule written down into a one-dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule.”

Claim: A molecular graph traversal used to write a SMILES string can begin at different points in the molecule.

“These different things. And so a SMILES is basically a graph traversal across the molecule written down into a one-dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule.”

Claim: A SMILES representation encodes a traversal of a molecule's graph as a one-dimensional string.

“And so there's multiple SMILES ways of representing the same molecule. And so you, it's nice to be fluent in transferring from one to the other. It's a pretty easy, uh, there's a couple open source packages people use, like RDKit to do this.”

Claim: The same molecule can have multiple SMILES representations, so converting between those representations is useful.

“And so there's multiple SMILES ways of representing the same molecule. And so you, it's nice to be fluent in transferring from one to the other. It's a pretty easy, uh, there's a couple open source packages people use, like RDKit to do this.”

Claim: RDKit is an open-source package used to convert between SMILES representations.

“Uh, but we wanted the language models to do it just 'cause it would go faster, and they weren't as good at using tools on their own back then, and nobody could do it. And then around February, like a lot of other things, uh-”

Claim: The team pursued language-model SMILES conversion because it was expected to be faster, while models at that time were less capable of using tools independently.

“... it became clear that they could start to actually work with chemistry in a meaningful way, which is a reason we started going in this direction.”

Claim: Around February, the speaker's team concluded that models had begun to work meaningfully with chemistry, prompting the team to pursue this direction.

“I think a little bit. Yep.”

Claim: The speaker thinks the models have internalized some, but only a little, understanding of molecules' physical nature.

“But these are the tasks that we're having these things do right now. Like, they are iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next.”

Claim: In the iterative molecule-design tasks, agents use information about examined molecules to choose better molecules in later steps.

“And when we remove the information about the molecules, like, these decisions really suffer. So they're, they're learning something. This is very interesting. Or at least they're using it.”

Claim: Removing information about molecules worsens agents' decisions in the molecule-design tasks.

“I guess that's a better way. We're not, we're not changing weights here, but they're definitely using that information as they make future choices.”

Claim: Although the agents do not update their weights, they use molecule information when making later choices.

“We think it's too spiky of a landscape, and it's very high dimensional. We're not really, we don't do that, but we mostly think it's not gonna work. I think it's-”

Claim: The speaker's team thinks Bayesian hyperparameter algorithms probably will not work for its problem because the search landscape is too spiky and very high-dimensional.

“That's it. Yeah, I think, I think that's the big thing, and that's probably something closer to, like- I guess, I guess what I would probably say is we have built this thing where we've-- if you've got scientific agents and you wanna know how they're behaving, what mistakes they might make, what... how they might learn from physical feedback, you know, if a new training recipe improves scientific decision-making.”

Claim: The team has built a system to measure scientific agents' behavior and mistakes, their learning from physical feedback, and the effects of new training recipes on scientific decision-making.

“We've got the capabilities to measure these things at scale.”

Claim: The team can measure scientific-agent behavior, learning from physical feedback, and training effects at scale.

“We do, yes. We have... Broadly, the way that we think about this is it is a lab in the loop with amortized latency.”

Claim: The team characterizes its approach as a lab-in-the-loop system with amortized latency.

“So it's like, go out, you record the thing once, and then you can have a huge number of roll-outs through that thing, effectively infinite roll-outs. And it-- we are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments.”

Claim: The team's approach records an environment once and supports a huge, effectively infinite number of subsequent rollouts through it.

“So it's like, go out, you record the thing once, and then you can have a huge number of roll-outs through that thing, effectively infinite roll-outs. And it-- we are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments.”

Claim: The team is considering whether other biological or scientific questions can be converted into reusable rollout environments.

“Because if you're doing conventional lab in the loop, it does take a long time, and, and it, and be- and because those things themselves have limited budgets, it's also not counterfactually dense.”

Claim: Conventional lab-in-the-loop experimentation is slow, and limited experimental budgets prevent it from being counterfactually dense.

“Right. Right. Yeah, yeah. Doing it, doing it in the roll-out one step at a time offers a lot more freedom because you can choose almost anything, and here you're restricted to the number of choices that we made when we pre-measured the environment.”

Claim: Live, step-by-step rollouts allow more freedom of choice than the team's pre-measured environments, which limit agents to the choices included during measurement.

“That's definitely true. Um, but it's like, it's still a pretty big, uh, number of choices.”

Claim: The team's pre-measured environment still offers a large number of choices despite restricting agents to the choices measured in advance.

“And, uh, yeah. And then we have, right now we're, we have, we've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that.”

Claim: Since Leash began, the team has screened about 1,000 proteins against a couple of chemical libraries.

“I think we're gonna try to get to maybe 20 of these new environments a day. That's a plan for us to try to get to beginning of maybe '26 or '27.”

Claim: The team plans to try to scale production to about 20 new environments per day by early 2026 or 2027.

“We c- we're doing two a day right now. Uh, so yeah, I think we could probably get to 20 a day without too much lifting. So we'll, we're, we're pushing. We'll see if we can do it.”

Claim: The team currently produces two environments per day and thinks it could probably reach 20 per day without much additional effort, but has not established that it can do so.

“Yeah. Yeah. It seemed like, it seemed like the thing to do. I mean, kind of informally, it seemed like where a lot of these agents are making the best and the fastest progress is where there's instant feedback that's very scalable.”

Claim: The speaker informally observes that many agents make their best and fastest progress in settings with instant, highly scalable feedback.

“And it's really hard to do that in the physical world, um, but we think that there's probably a handful of biological questions that you could re-rebuild in this way, uh, and go after them.”

Claim: Providing instant, scalable feedback is difficult in the physical world.

“And it's really hard to do that in the physical world, um, but we think that there's probably a handful of biological questions that you could re-rebuild in this way, uh, and go after them.”

Claim: The team thinks a handful of biological questions could probably be reconstructed as scalable environments and investigated using this approach.

“And even at... Yeah, and then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move on to new biology yet, uh, but you know, like, we're trying. We're gonna find out. It's not long.”

Claim: The team has not yet established whether reinforcement-learning progress in its environments transfers to new biological problems, and is investigating that possibility.

“Yep, yep. And then also one thing that I need to do is, uh, talk to some more decision people because a lot of this is about how do you spend a limited experimental budget, and then what can you do to do better at that?”

Claim: A central problem is how to allocate a limited experimental budget more effectively.

“Yeah. Beyond just medicinal chemistry, right? Just like, yeah, when do I stay? When do I go? How do I, how do I get better at that?”

Claim: The speaker wants to consult more decision-makers about improving choices beyond medicinal chemistry, including when to continue or move on.

“Yes. Yes. So we are, we're incorporating more and more structural tool calls and things like that with this as well, so it's, uh, it's, it's a pretty fun time right now.”

Claim: The team is adding an increasing number of structural tool calls to its system.