AI benchmark tasks often use compact problems with clear evaluation metrics, such as loss minimization or recovering performance from a scrambled embedding matrix, but the speaker says these are unlike actual AI R&D work.

“what are the skills you need to actually automate the process of being a researcher then what fraction of those skills do the AI systems actually have even in coding like a lot of coding is you have a very large code base you have to work with the instructions are very kind of vague there isn't for example you mentioned a meter eval in which because they needed to”

“make it an eval all the tasks have to be kind of compact and close and have clear evaluation metrics like here's a model like get its loss on this uh you know data set as low as possible uh or whatever or like here's another model and it's like it's embedding matrix has been scrambled just uh fix it to recover like most of his original performance etc those are not problems that you actually work on”

“in AI R&D. They're like very artificial problems.”