The speaker argues that a capability is best measured in very difficult real-world environments created at the frontier.

“the distillation from like methos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so that you can distill like kind of these behaviors and things in i think the prediction from this is that the frontier labs don't necessarily have much of advantage if at all in oral environments now because yes like user distribution matters for like general like behavior and and so on but like the best measure of a capability is the very very”

“very hard oral environments you've made at the frontier and so if you have access to those oral environments as anthropic and you have access to logic disflation and you've still made a worse model then maybe”