Alexander Mattick thinks the better scaling behavior of deep neural networks is probably why practitioners choose them over very wide shallow networks.

“nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”