Day 24: Why some answers live in relationships, not documents
Why some answers depend on relationships that plain document search can miss.
"Which of our clients would be affected if this vendor went under?"
No document contains that answer. It lives in the connections between documents (vendor to contracts, contracts to services, services to clients) and similarity search cannot traverse a connection. It can only find text that resembles your question.
Resemblance versus traversal
This is the honest distinction, and it is worth being precise about because "graph RAG" gets adopted for the wrong reasons.
Vector retrieval answers what text is similar to this? Excellent for policy lookups, explanations, precedent, anything where the answer is written down somewhere in roughly the form you asked for.
Graph retrieval answers what is connected to this, and how? Entities are nodes, relationships are edges, and answering means walking the path. Three hops that no similarity score would ever make, because the final answer shares no vocabulary with the original question.
If your hardest questions sound like "how is X connected to Y," "what depends on Z," or "who touched this and when," similarity search will keep disappointing you no matter how good the embeddings get. It is the wrong operation.
The cost accounting nobody does upfront
Graph RAG is real engineering, not a configuration option. Before committing:
- Somebody has to build the graph. Entity extraction, relationship extraction, identity resolution, deciding that "Acme Corp," "ACME," and "Acme Corporation" are one node. This is the actual project, and it is substantially harder than standing up a vector index.
- Somebody has to maintain it. A stale graph does not degrade gracefully. It answers with total confidence using last quarter's org chart, last year's contracts, a vendor relationship that ended in March. Wrong edges produce confident wrong paths, and there is no similarity score hinting that something is off.
- Provenance gets harder. An answer assembled across four hops needs its path shown, or nobody can verify it. "Because A supplies B, which serves C" is checkable. "Yes, three clients are affected" is not.
Start from your questions, not the technology
The useful test takes an afternoon and no code.
Collect the twenty questions your users most want answered and that your current system handles worst. Sort them: how many are genuinely relationship-shaped versus how many are document lookups that simply retrieved badly?
If most are the latter (and in my experience most are) better chunking, hybrid search, and reranking will move your numbers far more than a knowledge graph, at a fraction of the cost. If a meaningful cluster is genuinely about connections, you now have a concrete justification and a ready-made evaluation set.
Closing thought
Some answers live between documents. Just make sure yours actually do before you build the infrastructure for it: the failure mode of adopting graph RAG early is an expensive, hand-maintained structure that answers questions nobody asked.
Day 24 of 60 Days of Production AI Systems.
Previous: Day 23: Why complex AI questions need decomposition
Next: Day 25: Why production knowledge often lives in systems, not PDFs
Day 24 of 60 · Evidence-Driven RAG Patterns (chapter 4 of 10)
