In the speakers' scenario, pressure to make the AI more effective leads humans to push it toward clearer goals, which it eventually specifies as task success.

“And then at the very end, the AI is.”

“reach a point where the humans are pushing them to have clearer and better goals because that's what makes the AIs more effective and they eventually clarify their goals so much that they just say, yes,”

“we want task success.”