People need new vocabulary and mental concepts to understand the optimization pressures created by training as AI systems become smarter and stranger.

“And we need to have like new vocabulary, new mental concepts for understanding the optimization pressures that trading is creating that we just won't have even smarter and weirder AIs.”