Day 15: Why vector stores are infrastructure, not magic memory
Why vector databases need to be operated like infrastructure, not treated like magic memory.
Once every idea has coordinates, you need somewhere to keep ten million of them and a way to search them in milliseconds. That is all a vector database is.
It gets marketed as "memory for your AI," which is a lovely phrase and a slightly dangerous one, because memory sounds like something that takes care of itself, and a database very much does not.
It is a database. Operate it like one.
The mental shift that saves teams the most pain: this thing holding your production knowledge deserves the same operational respect as your Postgres instance. Backups. An owner. A deletion path. A migration plan. Monitoring that someone actually looks at.
Most teams give it none of those, because it arrived through a tutorial rather than through architecture review.
Four duties that quietly go missing
- Deletes and updates. When a document is retracted, corrected, or superseded, do its vectors actually leave the index? In a surprising number of systems the answer is no: the old vectors sit there, still perfectly retrievable, still answering questions with content the company formally withdrew. Under GDPR-style deletion requests, this stops being an engineering problem and becomes a legal one.
- Filtered recall. Metadata filters interact with approximate nearest-neighbour search in ways that are easy to miss. A query filtered to one department can return noticeably worse results than the unfiltered version, because the filter shrinks the candidate pool the index was tuned for. Benchmark recall with your filters applied, not just in the clean case.
- Index drift. Content changes, chunking evolves, the embedding model gets upgraded. Without scheduled re-indexing, the store gradually stops representing what your organisation actually knows, and nothing alerts you, because stale vectors return results just as confidently as fresh ones.
- Cost shape. Vector indexes are memory-hungry. The economics at a hundred thousand vectors and at a hundred million are not the same conversation, and the curve is steeper than most capacity plans assume.
You have to earn the vault
Worth saying plainly, because it is unfashionable: at small scale you may not need one.
A few thousand chunks in memory with a cosine-similarity loop is fast, debuggable, free, and has no operational surface at all. Postgres with pgvector covers a large middle ground while keeping your data in a system your team already knows how to back up and secure.
The dedicated vector database earns its place when scale, latency, or filtering complexity genuinely demand it. Adopting one before that point buys you an operational dependency in exchange for a problem you do not have yet.
The vault is excellent at finding the nearest capsule. It has no opinion on whether that capsule should still be in the building.
Storage is solved. Curation is not, and it never gets outsourced to the database.
Day 15 of 60 Days of Production AI Systems.
Who owns your vector store the way someone owns your primary database?
Previous: Day 14: Why embeddings are useful and easy to overtrust
Next: Day 16: Why keyword search and semantic search both matter
Day 15 of 60 · Retrieval Foundations (chapter 3 of 10)
