Day 13: Why chunk boundaries shape answer quality
How chunk boundaries decide what evidence the system can actually retrieve.
Split a document in the wrong place and you can make its answer unfindable.
That is chunking: nobody's favourite topic, and a bigger lever on RAG quality than most model decisions. It also gets made exactly once, early, by whoever wrote the ingestion script, and then silently caps everything built on top of it.
What a bad boundary actually does
The damage is not that a chunk looks untidy. It is that meaning gets separated from the thing that qualifies it.
A rule gets split from its exception, so the system confidently states the rule. A number gets split from its unit or its date. A table gets split from the header row that says what the columns are, which turns a grid of figures into a grid of figures about nothing. A "however" ends up in a different chunk from the sentence it was correcting, and the correction simply ceases to exist as far as retrieval is concerned.
Retrieval then returns pieces that are individually plausible and collectively misleading. The model has no way to know something is missing, an incomplete chunk does not announce itself.
Fixed-size is a default, not a decision
Splitting every document into 500-token blocks is fast, uniform, and completely indifferent to what the documents say. It will cut mid-sentence, mid-table, mid-clause, and it will do so at scale.
It is a reasonable starting point. It is not a strategy, and the tell is that nobody can explain why it is 500 rather than 300 or 800.
Cutting where the meaning cuts (sections, headings, paragraphs, list items, whole table-with-caption units) costs more work at ingestion and pays back on every query afterwards. Keep a policy with its exceptions. Keep a definition with what it defines.
A usable bar: a chunk should make sense to a person reading it alone, with no surrounding document. If it does not, the model has no better chance than the person did.
Re-chunking is a release, not a config change
This is the operational habit most teams are missing.
Changing your chunking strategy changes what is findable across your entire corpus. Answers that worked can break; answers that were broken can start working. That is the same blast radius as swapping the embedding model, and it deserves the same treatment: a fixed set of evaluation questions, run before and after, with the diff actually reviewed.
Teams that treat re-chunking as a parameter tweak find out about the regression from a user.
Closing thought
When a RAG answer quotes the rule and misses the exception sitting directly beneath it, the instinct is to blame the model. Check the boundary first. The exception was probably in the part you never retrieved.
Day 13 of 60 Days of Production AI Systems.
Previous: Day 12: Why retrieval quality starts before search
Next: Day 14: Why embeddings are useful and easy to overtrust
Day 13 of 60 · Retrieval Foundations (chapter 3 of 10)
