The agent competition will score generalization using performance on a private test set and weight that private-set performance more heavily because participants are expected to overfit to public material.

“be competitive on this and what are the types of scores you can get for it so the way that we're going to judge the agent competition is we will have a general like a generalization score which is like how well do you do on the private test set we know people are going to overfit to the public stuff but we'll place a higher weighting on the private side but that's a good point and one of the things i'm actually really curious about is what role does scaffolding play in future agi”