Day 23: Why complex AI questions need decomposition
Why complex questions become more reliable when the system answers them in parts.
"Compare our Q3 churn against our top two competitors and suggest what to fix."
That is not a question. It is five questions in a trench coat, and no single retrieval pass answers it well.
What a bundled question does to retrieval
Embed that request and you get a point in vector space that sits near everything and matches nothing well, an average of churn, competitors, and remediation that corresponds to no document you own.
So retrieval returns a blend: mostly-relevant material for the loudest part of the question, thin coverage of the rest. The model then writes a fluent paragraph across all five topics, with solid grounding on one and improvisation on four.
The output reads as complete. That is the whole problem. Nothing in the answer flags that "our competitors' numbers" came from nowhere.
Splitting it makes the work inspectable
Decomposition identifies the sub-questions first, retrieves for each independently, and merges only once each part has its own support.
The retrieval gain is obvious, each search finally has one clear target. The operational gain matters more: every sub-answer becomes a checkpoint. When the final output is wrong, you can point at the step that failed rather than staring at one opaque paragraph wondering which third of it to distrust.
One broad request, turned into inspectable work
User asks: "Can we safely automate this support refund workflow?"
- What does the refund policy actually permit?
- What is this customer's and order's current state?
- Which actions are within our authority to take automatically?
- Draft the response.
- Record evidence, decision, and owner.
Each step has its own evidence, its own success criteria, and its own failure mode. Step 2 failing looks nothing like step 1 failing, and in a bundled answer, both look identical.
The two ways it goes wrong
Decomposing everything. Simple questions do not deserve a five-step plan; they deserve an answer. A system that fans out "what is your phone number" into sub-questions has traded latency and cost for nothing. Route by complexity, and make that routing decision explicit rather than emergent.
Unbounded depth. Sub-questions that spawn their own sub-questions need a depth limit and a cost ceiling. Without them, one genuinely hard query can quietly consume an afternoon of compute, and the trace will show it was reasonable at every individual step.
The real skill is not decomposition itself. It is judging when a question is actually a project.
Encode that judgement in the routing layer. Applying it uniformly is just a more expensive way to answer easy questions.
Day 23 of 60 Days of Production AI Systems.
Which request in your product is really several questions hiding inside one sentence?
Previous: Day 22: Why one user question may need many searches
Next: Day 24: Why some answers live in relationships, not documents
Day 23 of 60 · Evidence-Driven RAG Patterns (chapter 4 of 10)
