Andrej Karpathy predicts that in ten years, giant neural networks will likely still be trained with forward and backward passes and gradient-descent updates, although the approach may look somewhat different.

“So maybe some of the things that we can bet on, I think, in 10 years by translational sort of covariance is we're still training giant neural networks with forward backward pass and update through gradient descent.”

“but maybe it looks a little bit different and”