The speaker says online reinforcement learning faces a substantial variance-reduction problem when feedback consists of a single user's signal and a single rollout rather than groups of examples.

“they it was it's hard because when you do online reinforcement learning you don't have groups right you just have one user saying one thing and then you get one roller”

“roll out and so like you have a big variance reduction problem”