Skip to main content
Production AI Field NotesProduction AI Field Notes

Command Palette

Search for a command to run...

Day 25: Why production knowledge often lives in systems, not PDFs

Why production knowledge often lives in systems of record, not only in documents.

Updated
•3 min read•View as Markdown
Day 25: Why production knowledge often lives in systems, not PDFs
V
Vamsi Krishna
I’m Vamsi, a builder focused on production AI systems. I write Production AI Field Notes to break down the architecture behind reliable LLM apps, RAG, agents, multi-agent workflows, evaluation, observability, safety, and governance. My goal is simple: help builders move beyond impressive AI demos and design systems that can be tested, operated, trusted, and improved in the real world.
Part of seriesProduction RAG Operations

Asking a document index "how many open tickets do we have right now?" is like asking a library for the current time. Wrong building, and it will helpfully give you an answer anyway: probably from a status report written in April.

Live facts do not live in PDFs. Inventory, order status, account balances, ticket counts, entitlements: these live in databases and business systems, and they change faster than any index can track.

Two retrieval modes, one chat box

Document retrieval answers what does the policy say. Structured retrieval answers what is currently true.

Production assistants need both, and the moment you add the second one, three problems appear that document-only RAG never had.

Routing. Something must decide, per question, whether the answer lives in prose or in a table. Get it wrong and the system quotes a stale PDF about a number the database knows exactly. This routing decision deserves to be explicit and logged, when answers are wrong, "it asked the wrong system" and "it asked the right system badly" need different fixes.

Query safety. Model-generated SQL running against production with write access is an incident generator with a natural-language interface. The floor is: read-only credentials, parameterised templates or narrow typed tools rather than free-form SQL, allowlisted views instead of raw table access, row limits, and query timeouts. Prefer exposing five specific tools (get_order_status, get_open_ticket_count) over one general run_query tool. Narrow tools are easier to validate, easier to log, and dramatically harder to misuse.

Freshness versus cost. Live calls are current but slower, rate-limited, and occasionally down. Decide per question class whether staleness or latency is the greater sin, and design the fallback for when the system of record is unavailable. "The database timed out" should not silently become "answered from a cached document."

Evidence has provenance here too

An API result is evidence like any other, and it needs the same treatment: what was queried, against which system, at what time, returning what.

Without that, a wrong answer is unfalsifiable after the fact — you cannot tell whether the query was wrong, the data was wrong, or the data was right at the time and has since changed. All three happen; only logging distinguishes them.

The demo question is "what is our refund policy?" The production question is "where is my refund?"

They arrive in the same chat box, phrased almost identically. They are answered by completely different infrastructure, and a system that only built one of them will answer the other confidently and wrongly.

Day 25 of 60 Days of Production AI Systems.

Which of your answers is being served from a stale document when a live system knows the real number?

Previous: Day 24: Why some answers live in relationships, not documents
Next: Day 26: Why evidence is becoming multimodal

Day 25 of 60 · Production RAG Operations (chapter 5 of 10)

#sql-rag#api-rag#rag#ai-engineering#production-ai

Comments

Join the discussion

No comments yet. Be the first to comment.

Production RAG Operations

Part 1 of 6

A practical operations layer for RAG systems. This series covers systems of record, SQL and API evidence, multimodal sources, freshness, corrective retrieval, evidence-tied reflection, and evaluation panels for production RAG quality.

Up next

Day 26: Why evidence is becoming multimodal

Why evidence quality changes when the source is visual, tabular, audio, or multimodal.

More from this blog

P

Production AI Field Notes

60 posts

Production AI Field Notes is a practical publication for builders moving from AI demos to reliable production systems. It covers the real engineering decisions behind LLM apps, RAG, agents, multi-agent workflows, evaluation, guardrails, observability, and governance. The focus is simple: less hype, more production judgment. Each note breaks down one AI system concept with practical framing, common failure modes, and builder-focused checklists.