Raw embeddings include low-level information, such as next-token punctuation, that may be irrelevant to the semantic analysis of interest.

“because they are like embeddings contain all sorts of things that you don't necessarily care about like should i have a comma in the next like in the next token but we care about like the semantic stuff we don't care about the sort of low-level processing stuff a lot of the time so doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about we can kind of separate that out”