Tom McGrath says they are exploring putting more compute into learning to obtain a better gradient through gradient descent.

“okay um i and i think that like you can also imagine so sort of going back to being bitter lesson filled here one thing we're trying to do is sort of put more compute into the learning process like gradient descent just gives you what it gives you there's no way like gradient descent is great but it would be great if you could spend more compute to get a better gradient um”