The speaker says there is still substantial headroom for better pre-training and that gains in compute efficiency have not slowed down.

“I don't see there hasn't been an asymptote of how much how much juice you can get out of better pre-training like we're not seeing a slowdown in compute efficiency gains you keep seeing stuff there's a lot a lot of headroom and reinforcement learning I think that that one is probably there's a lot of stuff to be kind of discovered there but it's again it feels more like a engineering discovery than”