Changing a model's output space can enable very fast inference; when a task such as binary classification can use a prior, invoking a language model at 400 times the cost may be unnecessary.

“to give you a trade-off which is i will do very very fast inference over like if you have some prior about this problem like let's say i only need to make a binary classification am i going to ask my language model to do this and pay like a 400x cost no that's like silly right um and i think for the longest time because the labs are the only places that control you're never going to use something other than like gpt6 or”