Public transparency into AI companies' development practices is currently insufficient to establish how they address reward hacking or whether their solutions overfit.

“you think you've solved it but you haven't actually solved it and i think that in that case like the thing we need is like a really good scientific understanding of like did we actually solve it and unfortunately um i think that currently the amount of public transparency into the development practices of ai companies are not sufficient to answer very basic questions about you know are how are they solving issues with reward hacking are they overfitting what's going on there um and so i think we would just need like a better and i think this like the current situation is like”