A reward model can be trained for human taste by having people compare pairs of outputs and choose the better one.

“And so how do you train on human taste? It's-- I think the technical answer is quite simple. I think the like practical answer obviously is more complicated. Um, the practical answer is you just train a reward model, right?”

“You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. Um, so like, um, fun fact, like we did this for slides, right?”