Day 30: Why RAG must be evaluated in parts
Why RAG evaluation has to measure retrieval, grounding, generation, and citations separately.
"The RAG system is bad."
Fine, which part? A single end-to-end quality score is a diagnosis-free diagnostic. It tells you something is wrong and nothing about where, so teams default to fixing the layer they can see, which is the prompt. Weeks of prompt iteration later, retrieval was broken the whole time.
It fails in parts, so measure it in parts
A RAG pipeline is at least four systems wearing one interface, and they fail independently.
| Layer | The question it answers | Example metric |
|---|---|---|
| Ingestion | Did the right knowledge ever enter the index? | source coverage |
| Retrieval | Did the correct passage make the candidate set? | recall@k |
| Ranking | Did it make the cut the model actually reads? | MRR / precision@k |
| Grounding | Did the answer stick to that evidence? | supported-claim rate |
| Citation | Can a reviewer verify each claim? | valid citation rate |
Diagnose in order, because failures masquerade
This is the part that saves the most time. Upstream failures disguise themselves as downstream ones.
If the right passage was never retrieved, everything after is irrelevant, and the symptom will look exactly like hallucination, because the model had nothing correct to work from and produced something plausible instead. A large share of "the model is making things up" tickets resolve to "recall@20 was 0.4 and nobody was measuring it."
So work upstream to downstream. Was it in the index? Was it retrieved? Was it ranked highly enough to be read? Did the answer use it? Only when all four pass is a generation problem actually a generation problem.
The minimum viable setup
Fifty labelled questions. For each: the question, the passage that should answer it, and what a good answer contains.
That is it. Fifty questions, per-layer metrics, run on every meaningful change, chunking strategy, embedding model, retrieval config, prompt. It takes an afternoon to build and it converts every subsequent argument into a measurement.
The specific thing it buys you: re-chunking stops being a gamble. Change the strategy, run the set, read the diff. Without it, every structural change to your pipeline is a leap of faith you find out about from users.
Closing thought
You cannot improve what you do not measure, and you cannot fix what you cannot locate. Component metrics tell you where the fault is. The end-to-end score only tells you that there is one.
Day 30 of 60 Days of Production AI Systems.
Previous: Day 29: Why reflection only matters when tied to evidence
Next: Day 31: Why agents need outcomes, boundaries, and stop conditions
Day 30 of 60 · Production RAG Operations (chapter 5 of 10)
