For the same intelligence level, increasing MoE sparsity from 1-in-16 to 1-in-128 or 1-in-256 can save substantial FLOPs.

“uh we get better and better performance the more flops we pour into these models at train time and test time um and if you look at just like the known landscape of ideas today we already see some scaling laws for bandwidth so one is uh these mixture of expert models it's pretty well known that actually ideally we would make them sparser and sparser and sparser so for like iso intelligence you will save a ton of flop”