While at Snapchat, Alex Zhang explored whether specialized GPU kernels could be written for the Infinite Attention paper.

“so um they had a project where like they were interested in writing it was this paper called infinite attention it was like a google paper yes”

“yeah so i was interested in in whether or not you could write specialized kernels for it at snapchat um it didn't nothing really came of it but i joined gpu mode at the time it was called cuda mode uh i think for like legal reasons or something they changed the name”