The speaker believes current models generally cannot solve the benchmark games meaningfully, while noting that Astra has been able to solve the Kirby game.

“but honestly like most models still can't really do or i don't actually think any models can solve these games very meaningfully like there are some games that they can i think i've seen astra be able to solve the kirby game and we also for this benchmark we intentionally designed a really minimal harness and i will get to this point about harnesses because i think there's a whole conversation to be had about like what is the value of a harness like what is even the”