Current deployments of fast inference chips do not run long-context attention on those chips and instead switch to a GPU for that workload.

“And if you look at the sort of technical detail of how they actually get employed and deployed today.”

“they're not running long context attention you still flip back to a gpu to do that and so there's this sort of tantalizing mismatch where precisely the area where we are almost able to go and take these things and make them much faster we also don't have that technical capability and so in a sense this is you know like many technical challenges it boils down to a slightly mundane technical observation which is we need chips that have this kind of”