Architecture changes can enable qualitatively new operating regimes that previous architectures could not reach, so describing their benefit only as a percentage efficiency gain can be misleading.

“architect i say like combining like saying just like an x percent efficiency gain is kind of misleading because like what an architect does is like letting you reach like a qualitatively new regime which you couldn't reach with the old architecture and then within that regime obviously the data is like the primary thing determining it but like you know if we say it didn't have like even like gqa we're doing like full attention all day we wouldn't be able to do like a million like ridiculous expenses do a million context and then like because of that we couldn't we can never use the data which is like actually at a million context”