Alexander Mattick thinks architectural choice makes no difference in the large-scale limit, though he is not referring to an infinite limit.

“nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”