Alexander Mattick gives the example that one architecture might require a billion parameters while another requires 100 billion because the second architecture scales less favorably.

“Like one architecture needs a billion parameters,”

“the other one needs 100 billion parameters just because the other architecture is just less nice.”

“nicely scaling. And it's that's probably the reason why we do is why we choose deep neural networks and not just a shallow one which is really really really wide. But I don't think that in the limit and I don't mean like infinite limit, but in the like large scale limit it makes a difference. And I think it's much more uh kind of”