Among frontier models, attention mechanisms and sparsity levels in MoEs and attention can change every couple of weeks.

“unified in kind of crying out for and memory bandwidth is one of them um but then you know as you say there are these evolutions and especially if you look at the kind of frontier of open source chinese models you know the exact nature of an attention mechanism will churn every couple of weeks in terms”

“kind of at the frontier the sparsity the sparsity level of the moes that you have the sparsity on the attention itself um and you know we may be in for like you”