# Day 26: Why evidence is becoming multimodal

The answer to "why did revenue dip in March?" might be a chart. The answer to "which part is broken?" might be a photo. A text-only pipeline walks past both without noticing.

Audit what your organisation actually knows and very little of it is clean prose. It is tables trapped inside PDFs, numbers on dashboards, diagrams in slide decks, screenshots attached to three-year-old tickets.

## Flattening is where meaning dies

The failure here is specific and easy to miss: most pipelines *do* process these formats. They just process them badly, and produce plausible text either way.

A table flattened into a paragraph loses the relationship between rows and columns. `Q1 2.4 Q2 3.1 Q3 1.8` is technically the content and conveys nothing about which metric, which unit, which segment. A chart described as "a bar chart showing quarterly performance" has discarded the actual numbers. A form loses which value belonged to which field.

None of this throws an error. The text is extracted, embedded, indexed, and retrieved with total confidence, and the model builds an answer on a version of your data that has had its structure removed.

## Rank formats by value, not by novelty

Image search is the exciting part of multimodal RAG. Tables are the profitable part.

For most teams the highest-value work is unglamorous: getting tables out of PDFs as actual structured tables, with headers attached and units preserved. That single fix usually unlocks more real questions than any vision model, because tables are where the numbers people ask about actually live.

So pick the one non-text format your users genuinely need and do it properly before adding a second. Half-parsed tables across five formats is worse than one format handled well, more surface area, same confident nonsense.

## Keep the pointer to the original

A model-generated description of a chart is a *retrieval aid*, not evidence. It helps the system find the chart; it should not be what a reviewer sees when verifying the claim.

Always store the reference back to the source artifact (page, region, file) and surface it. When someone asks "where did this number come from," the answer should be the chart itself, not a paraphrase of it written by another model.

## Closing thought

Evidence comes in more than text, and the pipeline decides how much of it survives contact with your index.

The test is simple and slightly depressing: take a question whose answer lives in a table or an image, and read exactly what your ingestion stored for that source. Most teams find the answer was never really in there.

*Day 26 of 60 Days of Production AI Systems.*

Which format holds the most trapped knowledge at your company, and does your pipeline actually preserve it?

---

**Previous:** [Day 25: Why production knowledge often lives in systems, not PDFs](https://blog.vamsiannamreddy.com/day-25-why-production-knowledge-often-lives-in-systems-not-pdfs)  
**Next:** [Day 27: Why freshness is part of correctness](https://blog.vamsiannamreddy.com/day-27-why-freshness-is-part-of-correctness)  

*Day 26 of 60 · Production RAG Operations (chapter 5 of 10)*