The speaker considers it evidence of surprisingly strong convergence that the phenomenon also appears in a substantially different model with hyper-connections and mixture-of-experts architecture.

“Not too surprising, but like a completely wackily different model,”

“you know, it doesn't even like with these hyper connections and MOE and that kind of thing, like it definitely speaks to a level of convergence that is quite surprising.”