The speaker's rough calculation suggests that a sparse model such as DeepSeek V3 has an optimal inference batch size of more than 2,400 concurrent sequences being generated at once.

“but if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching. Back of the envelope math suggests that the optimal inference batch size for a sparse model like say DeepSeekV3 is more than 2400 concurrent sequences being generated at once.”