When company-specific instructions require full weight updates instead of low-rank adapters, batching can provide substantial inference advantages.

“but if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching. Back of the envelope math suggests that the optimal inference batch size for a sparse model like say DeepSeekV3 is more than 2400 concurrent sequences being generated at once.”