For a fixed level of intelligence, increasing sparsity in mixture-of-experts models can substantially reduce FLOP requirements.

“uh we get better and better performance the more flops we pour into these models at train time and test time um and if you look at just like the known landscape of ideas today we already see some scaling laws for bandwidth so one is uh these mixture of expert models it's pretty well known that actually ideally we would make them sparser and sparser and sparser so for like iso intelligence you will save a ton of flop”