The speaker views modularity as a desired endpoint of learning, not a structure to begin with.

“learning representations by backpropagating error signals right the sort of the classic backprop paper actually like most of the figures in that are them saying look the model learned sensible representations from our backprop procedure and sort of have validated it by showing that it's interpretable i think that modularity is the end point you want but you don't start with modularity um”