AI systems can perform well on coding tasks that are impressive indicators of human coding ability, such as competitive programming.

“I think they are good at things that maybe impress us about human coders, like if you want to see like, oh, like what makes a person a really impressive coder, you might look at their competitor programming performance.”