In his modernization of Yann LeCun’s 1989 convolutional network, Andrej Karpathy found that algorithmic changes alone could roughly halve the error, while further gains required substantially more data, computational optimizations, longer training, dropout, and other regularization.

“And I was able to very quickly like half the learning rate just knowing by time travel by 33 years.”

“So if I time traveled by algorithms to 33 years,”

“I could adjust what Yann LeCun did in 1989 and I could basically half the learning, half the error.”

“I had to add a lot more data. I had to like 10x the training set and then I had to actually add more computational optimizations,”

“had to basically train for much longer with dropout and other regularization techniques. And so it's almost like.”