In 2021, evaluating training data was simpler because model responses to straightforward prompts, such as finding the best TVs, could be cross-checked.

“Yeah. So I've been working on labs for the last five, six years. When I started in 2021, it was like, it was much simpler because the kinds of things that the models had to do were, like, a prompt would be, uh, "Fi-- Help me find the best TVs on the market."”

“And then you can pretty easily cross-check that response and understand if the, the data that we are training the model on is good or not. Now it's, uh, really complex environments, so a lot of the, um, the way we test the data in labs has also become agentified, where we would write, uh, a set of, um, criteria that the data needs to meet, both on individual datasets, but as well as on the diversity of datasets.”