Why embeddings are useful and easy to overtrust

How does a computer know that "refund" and "money back" mean roughly the same thing, when they share almost no letters?

It stops comparing spellings and starts comparing positions.

Meaning gets an address

An embedding converts text into a long list of numbers: coordinates in a space arranged so that distance corresponds to similarity of meaning. "Refund", "money back", and "return my payment" land near each other. "Refund" and "refuel" do not, despite looking alike.

That is the machinery under semantic search, and it is genuinely powerful: users can ask in their own words and still reach documents written in yours.

Similarity is not relevance

Here is the gap that causes production problems. Embeddings measure similarity. Teams use them as if they measured relevance. Those two overlap often enough to be convincing and diverge often enough to hurt.

"What is our refund policy?" and "customers furious about refunds" sit close together in embedding space. One is a policy question; the other is a complaints analysis. Semantic closeness gets you to the right neighbourhood, not the right house, and the scoring function will not tell you which one you got.

The sharpest version of this: "eligible" and "not eligible" embed remarkably close together. Negation barely moves the coordinates, and the consequences of getting it backwards are enormous. Any domain where a "not" flips the outcome (eligibility, compliance, safety, permissions) needs a signal that is not purely semantic.

Three ways teams overtrust them

Domain vocabulary. General-purpose embedding models are trained on general-purpose text. Your internal product names, acronyms, and jargon may be positioned badly or arbitrarily. Test with the vocabulary your users actually type, not with tidy examples.
Version lock-in. The embedding model defines the geography of your index. Change the model and every stored vector becomes meaningless relative to new queries. You must re-embed the entire corpus. If that migration is not planned from day one, it eventually happens as an emergency.
Confidence theatre. A similarity score of 0.87 looks like a probability of correctness. It is not. It is a distance in a space you did not design, and the threshold that seemed reasonable on sample data may be meaningless on real queries.

What to actually do

Use embeddings for what they are excellent at: broad recall, catching paraphrases, surfacing candidates you would never have matched by keyword — then let other components decide precision: exact-match signals for identifiers, metadata filters for permissions and dates, reranking for final ordering.

The practical test: find queries in your domain where the semantically closest text is the wrong answer. Every domain has them. If you cannot find any, you have not looked hard enough, and your system is currently trusting similarity as though it were truth.