The speaker thinks economic pressures may favor adding specialized modules, such as law-specific modules, over updating one large shared model with deployment data.

“the economics of this will pressure not necessarily weight updates to one big like common shared model but like kind of like modules that get subbed in so a very obvious example this is a law but it might be something else like you know there's been a lot of work to try and fit like an arbitrary context length into a fixed size like this is all the linear attention stuff and all that sort of stuff and like cartridges which are essentially kv caches trained to be very very compressed kv caches to fit in a lot of information”