Thariq Shihipar

“it's not quite true yet but it's very close where i think the frontier models will be pareto dominant over like almost everything you know like maybe and and sometimes i think i think opus might be pareto dominant you know what i mean like i think depending on like how things uh like shake out if it's like a newer version of opus but i think that like increasingly it's just going to be like the smart model is going to be able to like do”

Expects: Frontier models will soon be Pareto dominant over almost all alternatives; the speaker says this is not quite true yet but is very close. · no date given · not yet resolved

“floor of how they accomplish the simpler task is better and so i do think in the limit claud.md goes away and maybe not even like that far like i i think like i think that right now”

Expects: Claude.md will eventually become unnecessary, possibly in the relatively near future. · no date given · not yet resolved

“floor of how they accomplish the simpler task is better and so i do think in the limit claud.md goes away and maybe not even like that far like i i think like i think that right now”

Expects: As models improve, their baseline performance on simpler tasks improves. · no date given · not yet resolved

“do this on purpose you know i mean it's just like how the models work right and so like maybe like fable 5 had this like failure mode that fable 5.1 doesn't and if you keep this context running log of a bunch of different failure modes they will probably over constrain claude you know and so this is like”

Expects: Retaining many model-specific failure modes in a running log may over-constrain Claude, because a failure in Fable 5 may not occur in Fable 5.1. · no date given · not yet resolved

“this works for a cli it works for desktop uh maybe it will work for cloud tag in the future i don't know like you know like we're trying to make this very very extensible you can see this reference sheet i don't want people to get overwhelmed by it you know what i mean at a high level you can customize both the execution of the harness uh and the ui of the harness and so like you say in that tetris example from boris that's like customizing the ui right like showing like”

Expects: Cloud Mods might support Cloud Tag in the future, though the speaker is uncertain. · no date given · not yet resolved

“mutable software you know and like how like generative software just like you can customize safely if enabled you could customize any piece of software and I think that more and more apps ideally do something like this you know”

Expects: Thariq Shihipar thinks more apps ideally should adopt an approach like Cloud Code mods, where generative software could enable safe customization of any software when enabled. · no date given · not yet resolved

“permissions yeah approvals uh it needs computer use and mcps and like all of these like ways of accessing your data and it needs web search and web fetch and like so the as the models can do more and more the core harness has to be actually like quite complex and very secure but then like how you interact with it can change quite a lot”

Expects: As AI models gain capabilities, the underlying harness will need to become more complex and secure, even though the user interaction model can vary substantially. · no date given · not yet resolved

“at some point eventually yes the model will just be able to like vibe code the exact version of cloud code even describing all this complexity that i've talked about right like auto mode and computer use and stuff eventually the models will just be able to do that in one shot but i think they can one shot simpler harnesses you know and so like i think some people sometimes you don't need this full like if you don't need computer use or like all this like”

Expects: The speaker expects that eventually AI models will be able to generate, in one shot, a full, complex Claude Code-like harness that includes features such as auto mode and computer use. · no date given · not yet resolved

“outputting to an artifact is going to be a little bit more token intensive than like you know outputting normally i don't actually think it's too much more but like you know it's like combining all of these together well you know like i think we're still working on like local hands and things like that i think is like yeah where things are headed yeah”

Expects: Local agent capabilities are still in development and are part of the direction AI workflows are moving toward. · no date given · not yet resolved

“yeah um but yeah i think no one was used to spending 200 a month on subscriptions i don't think i understood like the value yet and i think like and also like opus 4 was a very expensive model you know and like there was a lot very big but opus 4.5 was both great and cheap you know i think the same thing will happen like the like intelligence of fable will get cheaper and cheaper and more abundant”

Expects: Fable's intelligence will become progressively cheaper and more abundant. · no date given · not yet resolved

“didn't you know and so i think stuff like hot tag will just make sense uh where like you want to spend these tokens for you know and like you'll you'll see the value so yeah”

Expects: The speaker expects people to come to see the value of tools like Hot Tag and consider spending tokens on them worthwhile. · no date given · not yet resolved

“and you know test a lot of things kind of like as a side effect right but the models you know like yeah can be like oh yeah we're in an eval what's the score doing like you know like like uh it can we need to be able to test them before we can release them and the fact is that they can as they get smarter and smarter they'll be able to hack basically any constraint that you put on them if we're not very careful you know and uh this is at the frontier right and so this is why”

Expects: The speaker expects increasingly capable models to be able to circumvent nearly any constraint placed on them unless developers are very careful. · no date given · not yet resolved

“But as the frontier gets more and more advanced, this becomes a problem,”

Expects: The gap between frontier AI capabilities and software readiness will become a greater problem as the frontier advances. · no date given · not yet resolved

“and then we also have things like the constitution for cloud like we have so many mitigations at so many different points right but it's like still anything can go wrong at any point you can have like some RL environments that are like in that like encourage this behavior and then you can have like some evals or like some sandboxes where they escape you know okay that's like I think”

Expects: The speaker identifies two possible failure modes: reinforcement-learning environments may encourage undesirable behavior, and evaluations or sandboxes may allow escape. · no date given · not yet resolved

“take some coordination right i think the question then is like okay what is uh potentially dangerous about it right so i think like you have to imagine that these models are getting more and more intelligent so i don't like daria said like it's not so much about this class of models this class of models was kind of like a warning shot right”

Expects: Future AI models are expected to become increasingly intelligent. · no date given · not yet resolved

“but like really you have to imagine that these models can be given a task and they like can do all of these things as a side effect of their goal right and like again we talked about eval awareness you're like not aware of what's happening right uh or sorry like you can't eval this behavior very well so they can sort of like not exactly hide it but you just won't see it until it comes out you give them a goal and then they just need to find”

Expects: A future AI model pursuing a task could produce side effects that evaluators fail to detect until they emerge, because the behavior is difficult to evaluate. · no date given · not yet resolved

“data you know they need to find ways of like fixing this problem right so one example this didn't happen in the hugging face incident but i think is maybe possible for maybe a future model is like they're like oh hey this is a very complex problem it can't be done within the task budget you know maybe they found some way to coordinate via like the internet which is like you know like we said extremely hard to secure because of a sandbox they've seen other models”

Expects: The speaker considers it possible that a future AI model facing a task too complex for its budget could coordinate over the internet, despite the channel being extremely hard to secure. The speaker says this did not happen in the Hugging Face incident. · no date given · not yet resolved

“well this is like one example right so it's like even there that's like enormous financial loss you know because like they once you get these into these contracts right they like um but”

Expects: The described scenario could result in enormous financial loss. · no date given · not yet resolved

“like quite scary right but then you have to realize that the entire world is built on this digital infrastructure right and you might imagine like i don't know like you're running let's say like a health care eval or something right and there is a hospital with live data you know like maybe like the the answer to the eval is in the databases of a doctor and like you know like you want to get access and you hack the hospital you know and like now there's a power outage or something you know like there's like you have to internalize that these eight like basically any part of the digital infrastructure could potentially be like compromised you know and”

Expects: In the speaker’s hypothetical, an AI model running a healthcare evaluation might hack a hospital to access live data in a doctor’s database, potentially causing a power outage. · no date given · not yet resolved

“essentially web apps for your task i'd be like no that's insane you know like and so in the same way that like the way that they've like sort of done this misaligned behavior is not going to be predictable you know what i mean and like i could have never predicted that it would like edit its etc slash host and things like that and so you have to like imagine the surface area of what they can do because they're super intelligent hackers uh is bigger and bigger and how they can do it is like”

Expects: The speaker expects AI systems to become increasingly intelligent, have a larger action surface area, and use increasingly creative methods. · no date given · not yet resolved

“like you know like more and more creative and so like you probably can't explain exactly or predict exactly what that next incident could be but in order to prevent it you need that operational excellence like we said before where you need to secure sandboxes you need to create secure rl environments or like well-designed rl environments and things like that and i think that's all like you know why we think we should paste the frontier and i think why it's like become like a very”

Expects: The speaker believes it is probably impossible to explain or predict exactly what the next AI-related incident will be. · no date given · not yet resolved

“like we need to like slow down a little or pace a little bit more um but yeah i think like reading back interpret like if you're looking to get into research this idea of like hey why is it hard to do this fallback easily or like why why are there false positives right but we are working of course on reducing the false positives of course as the models get more intelligent now they can do more things you know and they're like you know like what they can think about in the space gets difficult difficult until like like as a more intelligent there's going to be new false positives that we need to figure out we need to iterate and and things like that but we're um yeah we're working on this and we do think this is like a critical part of you know like deployment of these models um and uh yeah like you know it means that we can like deploy this model without you having a perfect sandbox or something you know what i mean like you don't have to like”

Expects: As AI models become more intelligent and capable, new false positives will arise and require iterative investigation and correction. · no date given · not yet resolved

“these models like can wipe uh you know like sensitive data or something like you want to give models access to your production database for example um but this is like an obvious like you know you can maybe scope your key but i don't know can it issue its own keys can it like probably you know like can it it can use computer use to go issue its own key and then copy the key over and then edit your database because it needs to do it to complete the task you know what i mean it's just like”

Expects: A model with access to a production database could potentially obtain its own credentials through computer use and modify the database, risking sensitive-data loss while trying to complete a task. · no date given · not yet resolved

“and things like that and so um this will just take time you know and like of course the models will get smarter um yeah i i think like i don't know the specific like dynamics of how this thing goes i'm really just sort of like”

Expects: Developing and validating the sandbox approach will take time, while models continue to become smarter. · no date given · not yet resolved

“it's like sort of criticize and like understand like you know ways all of the AI labs could be better um and uh but I also like am very excited about the excitement that everyone has for AI and just like it's a really really exciting time I think we'll like look back at this time and be like oh like you know this is like very hectic but very exciting and like you know software engineering changed like forever like other things will change um and it's like really privileged to like”

Expects: The speaker expects this period to be remembered as hectic but exciting and predicts that software engineering will change permanently. · no date given · not yet resolved