The speaker recalls that the cart-pole reward was a binary future signal indicating whether the pole was balanced, requiring actions before the signal was received.

“and there were people in his lab at Cambridge doing all sorts of control theories. So, for example, there's this cartpole balancing problem and you can just solve that analytically. But this reward thing was different because it's basically in this particular case, I think it was a binary signal.”

“signal so it's either balanced or it's not and and it's a signal from the future so you have to do a lot of stuff before that thing says yes and and at the time they thought that this was an interesting problem to solve because this is much more kind of like amenable to you know maybe it's a better representation of animal learning”