Day 16: Why keyword search and semantic search both matter
Why exact words and semantic meaning both matter in production retrieval.
Search error code E-4012 with semantic search and you may get a thoughtful paragraph about error handling in general.
Search why do customers stop paying with keyword search and you get nothing, because no document in your corpus contains that exact sentence.
Two failures, two causes, one lesson: these are different tools for different jobs, and "which is better" is the wrong question.
What each one is actually doing
Keyword search matches the literal string, usually with statistical weighting so rare terms count for more. It is fast, precise, explainable, and completely indifferent to the fact that "cancel" and "terminate" mean the same thing to your users.
Semantic search compares meaning through embeddings. It handles paraphrase beautifully and is oddly unreliable on the exact tokens (part numbers, error codes, policy names) where a single character changes everything.
| Retrieval signal | Strong at | Weak at |
|---|---|---|
| Keyword | Exact terms, IDs, SKUs, error codes, quoted phrases, rare jargon | Paraphrase, synonyms, questions phrased in user language |
| Semantic | Intent, related concepts, natural-language questions, vocabulary mismatch | Exact identifiers, negation, recency, precise policy names |
| Reranking | Choosing the best few from many candidates | Finding anything that first-stage retrieval never returned |
That last row matters more than it looks. Reranking is often proposed as the fix for retrieval problems, and it cannot promote a document that never made it into the candidate set.
Stop debating, read your logs
The useful move here is not architectural. It is empirical, and it takes about an hour.
Pull a day of real production queries. Sort them into two piles: the ones that hinge on an exact string, and the ones that describe an intent. Nearly every real query stream has substantial volume in both piles. The ratio is almost never what the team assumed in the design meeting.
Now check which pile your retrieval currently serves. Choosing a single method is choosing which of those users to quietly fail, and because a missed result looks like one unlucky answer rather than a systematic gap, you can fail them for months without noticing.
The failure is invisible per query
This is why the problem persists. When keyword-only search misses a paraphrased question, the user sees "no results" and rephrases or leaves. When semantic-only search mangles an invoice number, the user sees a plausible-but-wrong document and may not realise it is wrong.
Neither event looks like a retrieval architecture problem. Both look like one bad day. The pattern only becomes visible in aggregate, which is precisely why you have to look in aggregate.
Closing thought
Know which detective you hired, and know which cases that detective structurally cannot solve.
Tomorrow: what happens when you put both of them on the same case.
Day 16 of 60 Days of Production AI Systems.
Previous: Day 15: Why vector stores are infrastructure, not magic memory
Next: Day 17: Why hybrid search is often the practical default
Day 16 of 60 · Retrieval Foundations (chapter 3 of 10)
