The speaker’s team uses question-contextualized image descriptions and frontier-model task solving as an auxiliary reward to assess a model’s global image understanding and vision encoder.

“And so one thing we do as like a shortcut to help models is, uh, have them generate descriptions of the image that are contextualized on the question, and then we ask frontier models to solve the task using just the image description as an example of an auxiliary type reward that really is just testing the model's, uh, global understanding of the image and how well that vision encoder is actually working.”