In the early days of reinforcement learning from human feedback, training primarily used human comparative judgments of which of two options was better.

“Yeah. Uh, so it's funny. Like very early on in the, like early RLHF days, the only thing we trained models on w- were essentially taste, right?”

“Like there, there was reinforcement learning through human feedback, and the human feedback was always a parallel signal. It was like, "Given two options, which one's better?" And so the early days of reinforcement learning were built on the notion of training on human taste.”