Clinicians achieved about 40% F1 on the ICD-10 coding evaluation, lower than the team expected.

“a comma recurrent comma left right like that's doctors didn't go to medical school to do that and they're not that great at it so we did is we we recruited about 18 physicians in the wild we created this like you know eval set annotated by four expert annotators each you know to kind of make sure that the quality is extremely high and and the clinicians were using f1 score we're scoring like let's say around 40 percent right on the f1 score which was surprisingly low lower than we thought”