Research-oriented engineering starts with a defined goal and evaluation, then diagnoses failures by checking the evaluation, environment, and model.

“You're defining upfront what your, uh, goal is, trying to formalize it as an eval, and then, uh, try to identify, okay, why, why is it failing? And sometimes your eval is wrong. Totally.”

“Actually, a lot of the times your eval is wrong. Yes. Sometimes your environment's wrong, and then sometimes the model's wrong. Yeah.”