The coding benchmark tasks are substantial programming tasks that could take a human four to eight hours, rather than simple multiple-choice questions.

“something to scoff at they're not just multiple choice questions they're like the sort of tasks that would take a human like four hours to do or eight hours to do but that's not the same thing as complete automated encoding so first we extrapolate so when they saturate the benchmarks and then we try to make our guess as to what the gap is between the first system that can completely saturate these benchmarks and the first system that can actually automate the coding and that's probably the more speculative part”