Day 21: Why RAG needs both broad context and precise evidence
Why good RAG often needs both precise snippets and enough surrounding context.
There is a structural tension in every retrieval system, and no chunk size resolves it.
Small chunks retrieve well and read badly. The embedding is focused, so search finds them precisely, but pulled out alone, a two-sentence fragment has lost the definitions, conditions, and qualifiers that gave it meaning.
Large chunks read well and retrieve badly. All the context is there, but the embedding now averages six different topics into one vague point in space, and precise queries stop matching it.
Pick either extreme and you lose something real.
Search small, answer big
Parent-child retrieval resolves the tension by refusing to use one unit for both jobs.
Index the small chunks, because that is what makes search precise. But when a child chunk matches, do not hand the model the child, hand it the parent: the section that chunk came from, with the surrounding paragraphs intact.
The match happens at street level. The evidence arrives as the whole neighbourhood.
| Retrieval unit | Optimised for | What it costs you |
|---|---|---|
| Child chunk only | Precise matching, tight embeddings | Fragments that lost their qualifiers |
| Parent section only | Coherent, self-explanatory evidence | Diluted embeddings, weaker matching |
| Child match → parent context | Precision in search, coherence in the answer | Larger contexts, dedup work, a sizing decision |
Where this earns its keep
The clearest signal you need it: your system keeps quoting the rule and missing the exception directly beneath it.
That failure has a specific shape. Retrieval was correct. It found the passage stating the rule. The exception lived two paragraphs down, in a different chunk, which did not match the query because it never restated the rule it was modifying. The model answered faithfully from what it received and was still wrong.
Policies, contracts, technical documentation, medical and legal content (anywhere meaning is carried by conditions and carve-outs) this pattern is close to mandatory.
Three details that decide whether it works
- Size the parent sensibly. A parent spanning an entire 40-page policy defeats the purpose; you have reintroduced noise at the answer stage. Sections and subsections, not whole documents.
- Deduplicate. Three child matches inside one parent should produce one parent, not the same text three times. Skip this and you burn context window on repetition and skew the model toward over-weighting that source.
- Budget the context. Expanded parents are large. Decide how many you can afford per answer before latency and token cost decide for you.
Closing thought
Retrieve broadly, answer precisely, or rather, match precisely and answer with enough surrounding context that the match still means what it meant in the document.
The right sentence, stripped of the paragraph that qualified it, is not evidence. It is a quote out of context, and the model has no way to tell the difference.
Day 21 of 60 Days of Production AI Systems.
Which of your answers are built on passages that lost their qualifiers on the way in?
Previous: Day 20: Why every important AI claim needs provenance
Next: Day 22: Why one user question may need many searches
Day 21 of 60 · Evidence-Driven RAG Patterns (chapter 4 of 10)
