Today's HBM-based GPUs and XPUs often cannot efficiently serve highly sparse MoE models because of bandwidth bottlenecks, resulting in low FLOP utilization.

“flops if you go from being like one in 16 sparse on your moe to one in 128 one in 256 um but one of the challenges one of the headwinds to doing that is actually that it becomes incredibly prohibitive on today's hbm-based gpus xpus to serve those models efficiently you end up often bandwidth bottlenecks you end up serving it really really low mfu low flops utilization”