Day 19: Why reranking is where retrieval becomes useful
How reranking turns broad retrieval candidates into evidence the answer can depend on.
The top result from your retriever is often not the best evidence. It is the nearest vector. Those are not the same thing, and the gap between them is where a lot of RAG quality quietly leaks away.
Why first-stage retrieval settles for "near"
Vector search has to scan millions of chunks in milliseconds, so it compares compressed representations, each document embedded once, in advance, with no knowledge of the question it will eventually be asked to answer.
That is enough to gather fifty plausible candidates. It is not enough to pick the three that should shape the answer.
A reranker does the expensive thing instead: it looks at the question and each candidate together and scores how well that specific passage answers that specific question. Slower by orders of magnitude, far more accurate, and completely impractical across a full corpus. Which is exactly why it belongs in second position.
The funnel is the whole design
Cheap and wide first. Expensive and narrow second.
Break the funnel in either direction and you pay for it. Rerank too many candidates and latency and cost balloon for diminishing returns. Skip reranking and raw vector order decides what the model reads, which matters because the model largely trusts whatever you put in front of it. It has no way to know that passage two was a better match than passage one. It will build a confident answer on whatever arrived, in the order it arrived.
Feed a strong model "nearby but wrong" and you get fluent, well-structured, confidently incorrect output. That is a ranking failure wearing a hallucination costume, and teams routinely misdiagnose it as one, then spend a sprint on prompt engineering for a problem no prompt can reach.
Before you add one
Measure the uplift on your own queries. Reranker performance varies sharply by domain. The benchmark number on the model card was not produced on your corpus, your chunk sizes, or your users' phrasing.
Check what reranking cannot fix. If the right document rarely appears in the candidate set at all, reordering is irrelevant. You have a recall problem upstream, and a reranker will simply sort fifty wrong answers very precisely. Diagnose that first: pull twenty failed queries and check whether the correct passage was anywhere in the top fifty. The answer tells you which component to work on.
Closing thought
Retrieval finds. Reranking chooses. Most systems invest heavily in the finding and leave the choosing to a distance metric that never saw the question.
Day 19 of 60 Days of Production AI Systems.
Previous: Day 18: Why relevant data can still be unauthorized data
Next: Day 20: Why every important AI claim needs provenance
Day 19 of 60 · Evidence-Driven RAG Patterns (chapter 4 of 10)
