The speaker says SWE-bench was initially dismissed as an impossible benchmark, but interest in improving performance on it increased after Devin came out.

“and i think it's like when you get a reaction like that it's almost like a good sign in the sense that like it's clear that people aren't thinking about what the purpose of this is and i will give another example of like sweetbench when sweetbench came out i mean oofier loves to tell this story when it came out like nobody cared like everybody was like this is an impossible task like why would we ever even consider this as a benchmark and it wasn't until devin came out that everyone was like well”