The speaker argues that, because execution details are difficult, researchers may rationally do more work at smaller scale and accept a final-performance penalty to iterate faster, train more models, and learn more effectively.

“training runs that people have done that were a bit of a bust and part of it is that i think there's just a bunch of details and actually getting that right and so um it makes sense to just do more of the work at smaller scale and just eat the fact that you're taking a hit on final performance in order to like be able to like quickly iterate and you know train more models faster and therefore better learn and also better”

“or better be able to just have like a, you know, a smarter ultimate production model.”