Tensor cores are designated circuits optimized for matrix multiplication, which accounts for the overwhelming majority of the floating-point operation count in the models being run.

“You conceive of things like the tensor core,”

“which is a designated circuit dreamt up by an architect that is exceptionally good at matrix multiplications because you have this insight that matrix multiplications are the overwhelming majority of the kind of flop count in these models that you're running.”