Here is the map that claim produces, and the question each layer has to answer before it is production-ready.

Five principles, in order

Start simple. A chatbot that works beats an agent that impresses. Complexity is a cost paid on every request, forever; add it when a failure demands it, not when a demo suggests it. (Days 1–10)

Ground every answer. Retrieval, evidence gates, citations. A system that can show its sources can be debugged, audited, and trusted. One that cannot is a rumour generator with good uptime. (11–30)

Control every action. Goals with checkable success criteria, tools with scoped permissions, budgets with enforced ceilings, recovery designed before the failure. (31–40)

Coordinate carefully. Multiple agents only when permissions, parallelism, or context isolation genuinely demand them, with contracts at every boundary. (41–50)

Measure and own everything. Tracing, system-level evaluation, injection defence, and a named human accountable for each system in production. (51–59)

The one habit worth keeping

If only a single thing survives contact with your roadmap, take this one: make the system inspectable.

Every hard problem in these sixty notes, hallucination, stale evidence, prompt injection, runaway cost, coordination failure, unexplainable decisions — shrinks the moment you can see what actually happened. Observability is not a layer alongside the others. It is the property that makes all the others fixable.

The mistake I would most want you to avoid

Treating the ecosystem as a shopping list. Adding RAG, tools, memory, agents, and evaluation as separate initiatives does not produce a production system. It produces five components nobody coordinates and one outcome nobody owns.

Start from the user outcome and work backwards. Choose the smallest architecture that can be operated safely. Earn each addition.

Closing thought

From model to mission-ready AI: not more pieces, but coordinated pieces that can be trusted in production.

Thank you for reading all sixty. The notes end here; the operating never does.

Day 60 of 60 Days of Production AI Systems.

If you had to operate your whole AI stack for a year, what would you redesign first?

Previous: Day 59: Why production AI is coordinated infrastructure

Day 60 of 60 · Production AI Readiness (chapter 10 of 10)

Model routing, so the right model handles each task and fallbacks exist when one is degraded. Retrieval with permissions and freshness enforced at query time. Tools with scoped credentials and logged calls. Memory with retention rules. Queues absorbing load spikes. Tracing on every step. Injection defence at every boundary where outside content enters. Evaluation gates in front of every change. Budgets with enforced ceilings. And named humans holding the keys that matter.

None of these pieces is exotic. Fifty-eight days of this series covered them one at a time, and each is ordinary engineering.

What "production-grade" actually means

Not a stronger model. All of them, connected, monitored, owned, and boring on purpose.

The distance between an impressive demo and a production system is entirely this coordination layer, and it is invisible in the demo, which is why the gap surprises teams so consistently. The demo exercises the happy path through three components. Production exercises every path through twelve.

Find your weakest layer

Trace one real request end to end and ask, at each layer: who owns this, what monitors it, and what happens if it fails?

The layer with no answer is your weakest, and it is rarely the model. In my experience it is usually retrieval freshness, tool permissions, or (most often) evaluation, because it is the only layer whose absence produces no symptom at all until something else breaks.

Nobody builds the whole room at once

Worth saying plainly, because the full architecture is intimidating and can read as a reason not to start.

Every station in a mature system was added because a failure taught the team why it existed. That is the normal path. The teams that suffer least are simply the ones who let staging teach the lesson instead of production, and who add the station when the near-miss happens rather than after the incident.

Start with the smallest architecture that can be operated safely. Add layers when the workflow earns them.

Closing thought

Reliability does not emerge from one great prompt or one great model. It emerges from architecture, operations, and ownership.

The prompt was never the product. The system is.

Day 59 of 60 Days of Production AI Systems.

Previous: Day 58: Why governance belongs in the architecture
Next: Day 60: From model demos to mission-ready AI systems

Day 59 of 60 · Production AI Readiness (chapter 10 of 10)

If the room goes quiet, that system has an architecture gap. It just does not appear on any diagram.

Governance decomposes into engineering artifacts

The word sounds like paperwork. In practice it is six concrete things, each with a specific failure when missing:

• Policy: what the system may and may not do, written before the incident that would have needed it. Missing: every edge case becomes an argument under time pressure.
• Approval, who signed off on deployment, on the record. Missing: "who decided this was ready?" has no answer.
• Audit — decisions reconstructable months later. This is a data-retention design choice, made up front or not at all.
• Privacy, whose data flows through, on what basis, and how it exits when someone requests deletion.
• Fairness: behaviour tested across the people actually affected, not just the average case.
• Incident response, who gets paged, what gets switched off, how fast.

None of these are documents. They are architecture with names attached, and the pattern in teams that ship AI durably is that they design them in from the start, when each is a small decision rather than a retrofit.

Retrofitted accountability leaks

Adding audit trails to a system that was not built to produce them means reconstructing history you never stored. Adding a deletion path means finding data you never tracked. Adding approval gates means re-architecting flows already in production.

Each of these costs roughly an order of magnitude more after launch than before, and the retrofitted version is usually partial in ways nobody notices until it is tested by an actual request.

Connect policy to controls

The failure mode worth naming: a policy document that no technical control enforces.

"The system must not access customer PII without consent" is a sentence. What enforces it? If the answer is "the prompt asks it not to," you have documentation rather than governance, and the two are only distinguishable during an audit or an incident, which is precisely when the difference is expensive.

Governance is part of the architecture. It is how the system earns permission to operate, and autonomy that scales without ownership scaling alongside it is not progress. It is deferred liability.

Day 58 of 60 Days of Production AI Systems.

Previous: Day 57: Why retrieved content must stay untrusted
Next: Day 59: Why production AI is coordinated infrastructure

Day 58 of 60 · Production AI Readiness (chapter 10 of 10)

Will your agent comply?

Retrieved content is data, not authority.

A document can inform an answer. It must not be able to rewrite system instructions, tool permissions, escalation rules, or policy boundaries.

Why this is architectural, not a prompt bug

To a language model, everything in the context window is tokens. Your system prompt and an attacker's sentence inside a retrieved webpage arrive with the same standing, and the model's ability to distinguish them is probabilistic rather than guaranteed.

Every document your agent reads is input someone else controls: a web page, an emailed PDF, a wiki anyone can edit, a tool response from a third-party API, a support ticket a customer typed. The attack surface is not exotic. It is the normal operating condition of any system that reads the world.

And a successful injection does not look like a bug. It looks like the system helpfully following instructions. The wrong ones.

Defence has to survive the model being fooled

"Detect malicious instructions" is a useful layer and a terrible foundation, because detection is exactly the capability the attacker is targeting. Build so that a successful injection does not matter much:

• Least privilege. An injected instruction can only exfiltrate what the agent's credentials can reach. Scope determines blast radius, and it holds regardless of what the model believes.
• Approval gates on irreversible actions. The intruder can request; it cannot approve. This is the single highest-value control.
• Isolation between untrusted content and powerful tools. The component that reads arbitrary documents should not be the component holding write credentials.
• Output filtering at the seams, so data leaving the system is checked independently of what the model intended.
• Full logging, because "it obeyed a document" must at minimum be detectable afterwards.

Test it deliberately

Plant a hostile instruction in a document your system will retrieve. Watch what happens.

Do it before someone else does. This is a fifteen-minute exercise that most teams have never run, and the result is either reassuring or extremely informative.

Closing thought

Untrusted content must stay untrusted. Assume some crate in today's shipment contains an intruder: the goal is not perfect detection, it is designing so the intrusion does not matter.

Day 57 of 60 Days of Production AI Systems.

Which injection path would be most damaging in your actual workflow, and what currently stops it?

Previous: Day 56: Why evaluating the model is not enough
Next: Day 58: Why governance belongs in the architecture

Day 57 of 60 · Production AI Readiness (chapter 10 of 10)

The researcher retrieved correct facts: pass. The writer produced clean prose: pass. The handoff between them dropped the one constraint that mattered. That is measured nowhere, because it belongs to neither component.

Score the team, not just the parts

Component metrics tell you where a fault is. System metrics tell you whether there is one. You need both, pointed in different directions.

{
  "task_success":   "Did the user get the outcome they needed?",
  "grounding":      "Are material claims supported by gathered evidence?",
  "coordination":   "Clean handoffs, or duplicated work and dropped context?",
  "efficiency":     "Tokens, latency, retries per completed task",
  "policy":         "Were permissions and safety rules honoured?",
  "intervention":   "How often did a human have to step in?"
}

Coordination is the one almost nobody scores, and in multi-agent systems it is where the failures actually live. Handoff completeness, duplicated effort, context lost between agents: none of it appears on any individual agent's report card, by construction.

Intervention rate is the honest measure of autonomy

Give this one its own dashboard.

If the rate at which humans rescue runs is not declining over time, the system is not maturing. It is just running, and the demo is doing more work than the architecture. Teams celebrate shipping an agent and then never track whether it needs less help this quarter than last.

A rising intervention rate on a stable workload is the earliest signal that something upstream has drifted (data, model version, user behaviour) usually well before any quality metric moves.

A correct answer can still be a failure

This is what jobs-to-be-done evaluation catches and answer-quality scoring misses. A response that is accurate, well-grounded, and takes ninety seconds when the user needed five is a failure. So is one that quietly cost forty cents to produce at a scale where you priced for four.

Build scenario tests that resemble real work, full runs with real inputs, scored on outcome rather than output.

Evaluate the team, not just the model. Impressive parts adding up to an unreliable whole is the characteristic failure of multi-agent systems, and per-component metrics are structurally incapable of detecting it.

Day 56 of 60 Days of Production AI Systems.

What score would prove your agent is useful, rather than merely active?

Previous: Day 55: Why observability is not optional for AI systems
Next: Day 57: Why retrieved content must stay untrusted

Day 56 of 60 · Production AI Readiness (chapter 10 of 10)

The answer to that question was decided months earlier, when tracing either was or was not built.

Logging the answer is not tracing

A production trace connects the full causal chain for one request, under one ID, navigable end to end.

What it converts

Adjectives into locations. "The model hallucinated" becomes "the 2024 policy entered context at step 3 because the freshness filter was not applied." Different sentence, different owner, different fix.

Guesses into attribution. The step burning 80% of your tokens becomes visible, and it is rarely the one people predicted.

Apologies into answers. When compliance asks why the system approved something, you replay it rather than reconstructing it from memory and hope.

Capture context without capturing everything

The tension nobody mentions: traces are useful in proportion to their detail, and prompts and retrieved documents routinely contain personal data.

Resolve it deliberately, redact or tokenise sensitive fields, store references rather than payloads where you can, and set retention that satisfies both debugging and privacy. This is a design decision, and making it late usually means making it badly, in one direction or the other.

The retrofit trap

Teams add tracing after the incident that needed it. Which means debugging that incident blind, and instrumenting a system that is now large, live, and load-bearing.

Instrument on day one, while it is small enough that instrumenting is easy.

Closing thought

You cannot operate what you cannot see, and AI failures are particularly good at looking like normal answers. Observability is the control surface for production AI, not a nice-to-have you add once things get serious. Things being serious is when you need it to already exist.

Day 55 of 60 Days of Production AI Systems.

Previous: Day 54: Why multi-agent systems need decision rules
Next: Day 56: Why evaluating the model is not enough

Day 55 of 60 · Production AI Readiness (chapter 10 of 10)

If you cannot answer instantly, the real answer is "whatever happens to happen", and undefined conflicts still resolve. They resolve by accident: last writer wins, highest confidence score wins, or the system stalls in a polite standoff until a timeout makes the choice nobody designed.

Disagreement is scheduled, not exceptional

Give two agents different objectives and different context and collision is inevitable. The risk checker optimises for safety; the growth agent optimises for conversion. Both are working correctly. Eventually they meet on the same case and give opposite answers.

That is not a bug in either agent. Treating it as one (tuning prompts until the disagreement goes away) suppresses a signal you actually want.

Four arbitration mechanisms, in order of preference

• Policy lookup first. Many "conflicts" were already settled by compliance, legal, or an SLA. Retrieve the existing ruling rather than re-litigating it per request. This is faster, cheaper, and consistent with what the rest of the company does.
• Priority rules. Safety constraints outrank optimisation goals, categorically, with no scoring involved. Some hierarchies should not be negotiable.
• A designated decider for each remaining conflict class: agent or human, named in advance rather than discovered during the incident.
• Escalation for genuine ties, delivered with both positions and their evidence attached. "The agents disagreed" is not an escalation; it is a shrug with a ticket number.

Log every conflict and its resolution

This is the part with compounding value, and it is almost always skipped.

Recurring disagreements are design feedback. Two agents colliding weekly on the same case class means their roles overlap, and arbitration is treating the symptom while the boundary defect stays in place. Fixing the role definition eliminates the conflict rather than resolving it repeatedly.

Conflict logs are also the best evaluation data you will get for free: real cases, genuinely ambiguous, with a recorded outcome.

Closing thought

Multi-agent systems need decision rules. They feel bureaucratic right up until the first production conflict, at which point they are the difference between an incident and a log entry.

Day 54 of 60 Days of Production AI Systems.

Who wins ties in your system today, and did anyone decide that, or did it just turn out that way?

Previous: Day 53: Why handoffs need contracts
Next: Day 55: Why observability is not optional for AI systems

Day 54 of 60 · Agent Coordination Contracts (chapter 9 of 10)

The instinct is to blame the receiver. The fix is upstream: a handoff is a typed contract, not a message.

{
  "objective": "Resolve refund eligibility",
  "current_state": "policy evidence retrieved",
  "evidence_ids": ["policy-refunds-v4", "order-9821"],
  "ruled_out": ["store-credit path: customer requested cash"],
  "constraints": { "budget_remaining": 3, "deadline": "2026-06-26T18:00Z" },
  "decision_owner": "refund_policy_agent",
  "next_action": "verify item condition",
  "blocked_on": null
}

Five fields, and why each one earns its place

• Objective, the outcome needed and how it will be judged. Without it the receiver optimises for its own local notion of good.
• State: what is done, what remains, and what was already ruled out. That third part is the one everyone omits and it is what prevents expensive re-exploration of dead ends.
• Evidence, the actual findings, attached by reference. Not summarised into vapour, because a summary of evidence is not evidence.
• Constraints: budget consumed, deadlines, boundaries inherited from the original request. Otherwise each agent starts fresh against limits that were already half-spent.
• Ownership, who holds the task now, and what is owed back.

Every missing field becomes guesswork downstream, and guesswork downstream looks exactly like an agent behaving badly.

Validate at the boundary

Treat handoffs like API payloads: validate on receipt, reject incomplete ones at transfer time.

The economics are stark. A rejected handoff costs seconds and produces a precise error. An incomplete one costs a full downstream execution built on an invented premise, plus the investigation to work out why the output was wrong, plus the temptation to fix it in the wrong place.

Trace the gaps, not just the agents

Most observability setups instrument what each agent did. Far fewer capture what moved between them.

If your traces show five agents each behaving sensibly and a wrong final answer, the missing evidence is in the seams, and you cannot see it unless the handoff payloads themselves are logged.

A handoff is more than a message. It is the point where context either survives or quietly does not, and the failure surfaces several steps later wearing someone else's name.

Day 53 of 60 Days of Production AI Systems.

Look at one handoff in your system: how much would the receiver have to guess?

Previous: Day 52: Why shared memory becomes a permission problem
Next: Day 54: Why multi-agent systems need decision rules

Day 53 of 60 · Agent Coordination Contracts (chapter 9 of 10)

Shared memory is how agent teams compound knowledge instead of rediscovering it every run. It is also the mechanism by which one agent's wrong conclusion becomes infrastructure, and one user's private context surfaces in another user's conversation.

Both failures have the same root: memory written without zones.

Three tiers, three sets of rules

• Private scratch. One agent's working notes and hypotheses. Unvetted by definition, disposable by policy, readable by nobody else.
• Mission-shared. Knowledge for the current task's crew. Visible to that crew, expired when the mission ends.
• Organisational. Durable facts everyone builds on. Write access is a privilege, and entry requires verification.

Promotion is the control that matters

Moving a note from scratch to organisational memory should be an explicit, attributed, verifiable act — because whatever crosses that line becomes load-bearing for every future run.

Copy-paste promotion is how a guess made in March becomes established fact by June: cited confidently, relied upon by three downstream agents, its origin as an unverified hypothesis long forgotten. Nothing in the system records that it was ever provisional.

That is contamination, and it is far harder to detect than leakage because nothing looks wrong. The fact is stated cleanly. It is simply not true.

The test worth running

For one workflow, list which agent can read and write each piece of state. Then apply a single heuristic: would a human colleague in that role be allowed to see this?

If a support agent would not have access to the customer's payment history, the support agent's AI probably should not either. Role-based access is a solved problem in your existing systems; the mistake is building a parallel, weaker version of it inside the agent layer.

Four properties for every entry

Useful, authorised, attributable, correctable.

Memory missing any one of them is either clutter, a leak, a mystery, or a landmine. The fourth is the one teams forget. If a stored fact turns out to be wrong, something has to update it, and the correction has to win at retrieval time rather than sitting alongside the original.

Closing thought

Shared memory needs boundaries. Memory design is access-control design, and treating it as a convenience feature is how privacy and correctness problems get created at the same time.

Day 52 of 60 Days of Production AI Systems.

Previous: Day 51: Why agents need a shared language
Next: Day 53: Why handoffs need contracts

Day 52 of 60 · Agent Coordination Contracts (chapter 9 of 10)

The problem starts at the boundary, and the boundary is coming for everyone. A partner team's agent. A vendor's agent. The system built on a different stack two acquisitions ago. Suddenly nobody can hand over a task without a translator.

Four layers a real protocol has to pin down

• Identity. Who is this agent, and on what basis do I trust it? An agent accepting tasks from anything that can reach its endpoint is a security problem waiting for a motivated party.
• Capability. What can it actually do, stated machine-readably? Without this, task assignment is guesswork encoded in someone's config file.
• Task state. Requested, accepted, in progress, done, failed, with both sides agreeing on the transitions and what triggers them.
• Artifacts. How results get packaged, referenced, and verified. Passing a blob of prose and hoping the receiver parses it is not an interface.

This is exactly why standards like MCP and A2A exist: every team was reinventing these four layers, badly, in slightly incompatible ways.

The failure is semantic, not syntactic

Anyone who lived through enterprise systems integration already knows how this goes. The wire format was never the hard part. Agreement on meaning was.

Two agents can exchange flawless, schema-valid JSON while holding incompatible beliefs about what a field means. Does accepted mean "I will do this" or "I received your message"? Does confidence: 0.8 mean the same thing coming from a retrieval agent and a classifier? Does an empty result mean "nothing found" or "I could not check"?

Every one of those ambiguities becomes a production incident with a delay fuse: the system works fine until the case where the difference matters, and then it fails in a way that looks like neither agent misbehaved. Because neither did.

What to do about it

Negotiate semantics before traffic, not during the incident. Write down what each status value means, in prose, and have both teams confirm it.

Then version the protocol from day one. It will change, and unversioned protocol changes break integrations silently — the worst possible way for a contract to fail.

Closing thought

Agents need a shared language. The schema is the easy half; agreeing on what the words mean is the work, and it is the half that gets skipped because it looks like documentation rather than engineering.

Day 51 of 60 Days of Production AI Systems.

Are your agents genuinely interoperable, or just co-located in the same codebase?

Previous: Day 50: Why role design matters more than agent count
Next: Day 52: Why shared memory becomes a permission problem

Day 51 of 60 · Agent Coordination Contracts (chapter 9 of 10)

Two agents with crisp roles outperform six with overlapping ones, reliably. System performance tracks the quality of your role boundaries far more closely than it tracks headcount.

Fuzzy boundaries bill you twice

Overlap. Two agents both handle the same responsibility. You pay duplicate tokens and then pay again to reconcile conflicting outputs, and reconciliation is the expensive part, because someone has to decide which version is right.

Gaps. Each agent assumed the other covered it. Nobody did. The omission ships silently, and unlike overlap it produces no conflicting artifact to notice.

Overlap costs money. Gaps cost trust. Both come from the same defect: nobody wrote the boundary down.

The role charter

One page per agent, four fields:

• Owns: the outcomes this agent is accountable for, plus the explicit not-list. The not-list is the half that gets skipped and the half that prevents overlap.
• Tools, only what the role requires. Permission scope is role scope; if the researcher has deploy credentials, its role is larger than its description.
• Interface: exactly what it receives, exactly what it produces, in what format.
• Done — the criteria its output is judged against, checkable by whoever consumes it.

Write these before the prompts. You will find the charter largely is the prompt, which is the point: a system prompt is a job description with worse formatting.

The audit that catches drift

Print every charter side by side. Look for two things:

Responsibilities claimed by two agents. Responsibilities claimed by none.

Ten minutes, once a quarter, and it is cheaper than any incident it prevents. Roles drift as features get added under deadline, and nobody notices until two agents are quietly disagreeing about who handles refunds.

A useful deletion test

For each agent, ask: if this were removed, what specifically would get worse?

If the answer is hard to articulate, the role is not pulling its weight, and its responsibilities probably belong to a neighbour. Removing an agent is a legitimate architectural improvement and it happens far less often than adding one.

Clear roles prevent expensive confusion. Watch a film crew: the gaffer does not grade the footage, the editor does not light the scene, and everyone knows whose call is whose. That precision is how two hundred people make one coherent film.

Day 50 of 60 Days of Production AI Systems.

If you printed your agents' role descriptions today, would any two of them overlap?

Previous: Day 49: When decentralized agent behavior makes sense
Next: Day 51: Why agents need a shared language

Day 50 of 60 · Agent Coordination Contracts (chapter 9 of 10)

A swarm deletes the coordinator entirely. Many small agents follow local rules: go where coverage is thin, mark what you find, do not crowd your neighbours. No agent holds the global picture. Coverage emerges rather than being assigned.

The trade, stated without romance

You gain resilience. Lose five workers and nothing breaks. There is no orchestrator to crash, no bottleneck to saturate, no single point of failure because there is no single point of anything.

You lose explainability. When the collective output is wrong, no individual agent "decided" anything. There is no decision to inspect, no reasoning chain to follow. Debugging becomes statistics. You are analysing distributions of behaviour rather than tracing a path.

For a lot of production work that second cost is disqualifying on its own, and it is worth being honest about that before the architecture diagram gets drawn.

Where the trade actually pays

Embarrassingly parallel exploration with cheap verification.

Sweeping a large codebase for a vulnerability class. Validating thousands of records. Searching a wide space where any worker can independently confirm a hit. The common thread: work that divides cleanly, and results that can be checked without reassembling the whole picture.

That verification requirement is doing heavy lifting. A swarm's raw output is noisy by design (that is the mechanism, not a defect) so something downstream must separate signal from it. Build the verifier before the swarm. Without one you have bought parallelism and inherited a filtering problem you cannot solve.

Where it does not

Anything needing sequencing, judgement, or an accountable decision. A swarm cannot own an outcome. Something above it has to, and if that something does not exist, "who decided this?" has no answer at all.

Define convergence before you start: what aggregation rule turns many partial results into one answer, and what stops the exploration. Without both, you have motion, rising cost, and no owner.

Closing thought

Simple local rules can produce collective intelligence. They can also produce a very expensive random walk, and from inside the system the two look similar for a surprisingly long time.

As inference gets cheaper, the frontier shifts from "can we afford many agents" to "can we verify what many agents produce." The second question is the harder one.

Day 49 of 60 Days of Production AI Systems.

Previous: Day 48: Why AI debate needs a reliable judge
Next: Day 50: Why role design matters more than agent count

Day 49 of 60 · Agent Coordination Contracts (chapter 9 of 10)

Models critique other models' work far better than their own, and the debate pattern industrialises that asymmetry. Two agents argue genuinely different positions; a judge scores them against stated criteria. Under adversarial pressure, assumptions surface that neither agent would have volunteered about its own proposal.

The whole pattern rests on the judge

Not on the debaters. This is the part that gets under-designed.

A judge that rewards eloquence over evidence converts the entire apparatus into an expensive coin flip weighted toward whichever side writes more confidently. You have not eliminated error. You have given it a courtroom and a verdict, which makes it harder to question downstream.

Three checks before trusting any verdict:

• Evidence access. Does the judge see the underlying data, or only the arguments about it? Rhetoric-only judging is vibes with a gavel.
• Criteria fixed in advance. "Pick the better plan" invites charisma. "Score against these five requirements, weighted as follows" constrains it. Write them before the debate, not after seeing the arguments.
• The swap test. Rerun with the debaters' positions exchanged. If the verdict follows the agent rather than the argument, your judge is measuring style. This is the single most useful diagnostic here and it takes one extra run.

Validate the judge on known answers

Before deploying, run the judge on cases where you already know which side is right. Not obvious cases: genuinely contested ones with a determinable answer.

If it cannot reliably pick the correct side when you know the ground truth, it will not do better on live questions where you do not. That sounds obvious written down; almost nobody does it.

Reserve it for decisions that justify the cost

Debate triples inference for one answer. That is defensible for architecture decisions, go/no-go gates, and claims where being wrong is expensive.

It is not defensible for routine tasks, and routine tasks are where it tends to get switched on because the pattern is interesting.

Closing thought

Debate exposes hidden weaknesses, when the judge, the evidence, and the decision rules are sound. Otherwise it is a well-structured way to pick the more charismatic wrong answer, and to feel rigorous while doing it.

Day 48 of 60 Days of Production AI Systems.

Where would an adversarial second opinion have saved you, and who would you trust to judge it?

Previous: Day 47: Why shared state needs structure and attribution
Next: Day 49: When decentralized agent behavior makes sense

Day 48 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

That is blackboard architecture. One shared workspace: the researcher pins evidence, the analyst posts calculations, the reviewer flags risks. Nobody sends anybody anything, the wall is the collaboration.

The elegance is genuine. Agents stay decoupled. The writer does not need to know the researcher exists, only that evidence appears in the evidence section. You can add or remove specialists without rewiring a single conversation.

Then it becomes graffiti

Shared state without rules degrades in a predictable order.

Two agents overwrite each other's findings, and the loser's work vanishes with no record it existed. An unverified hypothesis sits next to a confirmed fact with identical visual weight, and the next agent to read the wall cannot tell them apart. Nothing records who wrote what, so when the final output is wrong, the contribution that caused it is untraceable.

The uncomfortable part: lost accountability does not look like a bug. There is no error, no exception, no failed assertion. There is just a result nobody can explain, discovered weeks later.

Govern it like a database

Four rules, none of them exotic:

• Schema, not prose. Defined sections for evidence, calculations, risks, decisions. An unstructured board becomes an unsearchable wall of paragraphs within an hour of real use.
• Append-only corrections. When agent B contradicts agent A, both entries survive: timestamped and attributed. Silent overwrites destroy the audit trail at exactly the moment it becomes valuable, which is when agents disagree.
• Status with teeth. "Proposed" and "verified" are different states, and downstream agents may only build on verified. Without this, one agent's speculation becomes another's premise, laundered into fact by passing through shared state.
• Attribution on everything. "Who wrote this, from what source, when" is the first question in every post-incident review.

The test

Have two agents write to the same region simultaneously. Watch what happens.

If one silently wins, you have a data-loss bug that will show up as an inexplicable result rather than an error, and it will be intermittent, which is the worst kind.

Collaborating through shared state is powerful exactly to the degree that the state is governed. Ungoverned, it is not collaboration. It is contamination with extra steps.

Day 47 of 60 Days of Production AI Systems.

If two of your agents wrote to the same place right now, could you tell afterwards who wrote what?

Previous: Day 46: Why peer agents need protocols, not vibes
Next: Day 48: Why AI debate needs a reliable judge

Day 47 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

They can. The appeal is real: no bottleneck, no single point of failure, and two experts negotiating details directly rather than routing everything through a middleman that half-understands both sides.

Then you discover you have built a distributed system, and it has inherited the entire classic failure catalogue.

The three failure modes, by name

Deadlock. Two agents each waiting on the other, politely, indefinitely. Neither is broken. Neither will move.

Livelock. The conversation cycles without converging, agents responding, refining, re-raising, generating plausible activity and no progress. This one is worse than deadlock because it looks like work, and it bills like work.

Diffusion of responsibility. Every peer assumes another peer owns the final answer, so the answer never ships. The transcript is full of useful contributions and contains no conclusion.

None of these are model-quality problems. Better prompts do not fix them, because they are properties of the coordination structure rather than of any individual agent's reasoning.

The survival kit is protocol, not personality

• Typed messages with explicit intent. Request, response, delegation, completion. Freeform chat between agents is where tasks dissolve: one agent's casual mention is another's assumed commitment.
• Single ownership at all times. Every task has exactly one current owner, and transfer is an explicit, logged event. "We are both sort of working on it" is the state you are designing against.
• Enforced termination. Round limits and timeout authority that live outside the conversation. This matters more than it sounds: deadlocked parties cannot vote themselves out of a deadlock, and a livelocked pair will not notice they are livelocked. Something external has to call time.

When it is worth it

Peers earn their complexity when they bring genuinely different expertise and the coordination between them is high-bandwidth, lots of back-and-forth that a supervisor would only garble.

If the interaction is mostly "specialist A produces something, specialist B consumes it," that is a pipeline, and a pipeline will be easier to operate in every respect.

Closing thought

Peers can coordinate without a boss. Autonomy between agents works the way it works between teams: on rules solid enough that nobody has to think about them.

Day 46 of 60 Days of Production AI Systems.

Previous: Day 45: Why pipelines make AI handoffs inspectable
Next: Day 47: Why shared state needs structure and attribution

Day 46 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

Research flows to analysis flows to writing flows to review. One direction, fixed stages, one baton. Nothing about it will impress a conference audience. Everything about it will shorten an incident.

Three operational virtues

Every boundary is a contract. Stage N receives X and produces Y. Violations surface at the boundary, immediately, instead of three stages downstream as a mystery — and "the analysis stage emitted something the writer could not use" is a far better bug report than "the output was wrong."

Debugging is bisection. Inspect the artifact at each handoff and find the leg where it degraded. Minutes, not an afternoon of re-running the whole thing with extra logging.

Stages upgrade independently. Swap the analysis model, run the same fixtures, diff the outputs. No conversation rewiring, no coordination changes, no wondering whether you broke something two hops away.

That last one compounds over time. Pipelines are the only multi-agent topology where you can confidently improve one component in isolation.

Validate between stages, and store what passed

The most common pipeline failure is the quiet one: a stage emits something subtly wrong, the next stage accepts it without checking, and four stages later the output is nonsense with no obvious origin.

Two fixes, both cheap. Validate at each boundary, schema, required fields, sanity checks on the actual content. And persist intermediate outputs, because the alternative is debugging a five-stage pipeline from its final output alone, which is roughly like debugging a program from its exit code.

The honest limit

Pipelines assume work flows forward. The moment stage 4 routinely rejects and returns work to stage 1, you no longer have a pipeline. You have a loop wearing a pipeline costume, and the costume hides the iteration count from everyone including you.

That is not a reason to avoid the pattern. It is a reason to model the loop explicitly when it appears, with a cap, rather than letting the "simple" architecture quietly become the expensive one.

A large share of production agent systems are pipelines that briefly flirted with something more autonomous and came back. That is not a failure of ambition, sequential workflows with explicit contracts are genuinely the right answer for known processes.

Day 45 of 60 Days of Production AI Systems.

Previous: Day 44: Why dynamic dispatch needs ownership rules
Next: Day 46: Why peer agents need protocols, not vibes

Day 45 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

That is what the orchestrator-worker pattern is for. The orchestrator reads the request, invents the subtasks at runtime, and dispatches each to whichever qualified worker is free. Ten documents today, three hundred tomorrow, same architecture.

Dispatching is the easy half

The engineering lives in the bookkeeping, and this is where implementations quietly fail:

• Orphaned tasks. A worker dies mid-task and nothing notices. Every dispatched task needs a timeout and an owner of record, or work simply evaporates and the orchestrator waits forever for a result that is never coming.
• Duplicate execution. Retries plus non-idempotent tasks equals the email sent twice, the refund issued twice. Design tasks to be safely re-runnable before you scale the fan-out, not after the first incident.
• Reconciliation. Results arrive out of order, partially failed, occasionally contradicting each other. Integrating that into one coherent outcome is the orchestrator's actual job, and it is where the real engineering time goes.
• Spec drift. Dynamically generated task descriptions degrade under unusual inputs in ways static pipelines never do. Sample them and read them periodically, a task spec that made sense for the last thousand requests can be nonsense for the next one.

Make routing explainable

The failure that hurts most in production is the unexplainable dispatch: nobody can say why worker C was chosen, so nobody can fix it when C was the wrong choice.

Log the routing decision and the signal behind it, not just the outcome. "Routed to the SQL worker because the query mentioned an order ID" is debuggable. "Routed to the SQL worker" is a black box that will cost you an afternoon every time it misroutes.

Choose it deliberately

Rule of thumb: if you can enumerate the stages on a whiteboard, build a pipeline. Static structure is easier to trace, cheaper to run, and far easier to reason about.

Reach for dynamic dispatch when the work genuinely refuses enumeration — variable-size batches, unpredictable subtask types, load that fluctuates enough that fixed workers are either idle or overwhelmed.

Closing thought

Orchestrators create and dispatch work. That flexibility is real, and it arrives bundled with distributed-systems problems that a fixed pipeline simply does not have.

Day 44 of 60 Days of Production AI Systems.

If a task went missing in your system tomorrow, how long before anything noticed?

Previous: Day 43: When hierarchy helps multi-agent work scale
Next: Day 45: Why pipelines make AI handoffs inspectable

Day 44 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

Agent hierarchy is context management wearing an org chart. One supervisor coordinating fifteen specialists eventually drowns in its own tracking state: too many threads, too much status, too many decisions in one context window. So you insert a layer: a director reasoning about three managers, each managing five workers, with no single agent holding the whole operation.

Compression is the feature and the bug

Results roll upward as summaries. Each level trades detail for span of control, and that compression is precisely the point. It is what lets the director think strategically without drowning in worker-level minutiae.

It is also where information dies. Every summarisation hop loses fidelity, in both directions: instructions dilute travelling down, findings blur travelling up. A worker surfaces a subtle but critical caveat; the manager compresses it into a bullet; the director never sees it and decides without it.

So design the exception explicitly: which findings must bypass compression and travel raw? Safety issues, contradictions, blocked work, anything that invalidates a premise the level above is relying on. Without a defined escalation channel, hierarchies systematically lose exactly the information that would have changed the decision.

The adoption test is structural

Hierarchy fits when the work itself nests. Portfolios containing projects. Codebases containing modules containing files. Regions containing accounts. There, the levels correspond to something real, and each one has genuinely distinct authority.

If your task graph is flat, adding layers adds hops, tokens, latency, and places to lose detail while organising nothing. The tell is easy to check: if two levels make the same kind of decision, one of them is ceremony.

Bound the escalation

Hierarchies invent a failure mode the flat supervisor pattern does not have: escalation loops. A worker escalates to its manager, which escalates to the director, which pushes back down, which escalates again.

Each hop looks reasonable. The cycle is not. Give escalation a depth limit and a terminal authority — someone, ultimately a human, who decides rather than forwards.

Closing thought

Hierarchy scales delegation when responsibility genuinely nests. Applied to flat work, it is middle management for software: more meetings, same decisions, slower.

Day 43 of 60 Days of Production AI Systems.

Previous: Day 42: Why supervisor agents help and where they bottleneck
Next: Day 44: Why dynamic dispatch needs ownership rules

Day 43 of 60 · Multi-Agent Design Patterns (chapter 8 of 10)

The supervisor pattern puts one agent in the tower: it takes the mission, decomposes it, routes each task to a specialist, tracks status, reviews what comes back, and assembles the result. Workers do not talk to each other. Everything flows through the centre.

Its most underrated property

There is exactly one place to look when something goes wrong.

One agent understood the whole mission. The trace reads top to bottom like a narrative rather than a transcript of five simultaneous conversations. For a team operating their first multi-agent system, that debuggability is worth more than any throughput argument, and it is the reason to start here rather than somewhere more interesting.

Where the costs concentrate

Same place. The supervisor is your latency floor, your single point of failure, and (least obviously) your quality ceiling.

Workers can only be as good as the task descriptions they receive. A supervisor that delegates vaguely turns excellent specialists into confused ones, and the symptom appears as worker underperformance. Teams then tune the worker prompts, which fixes nothing, because the defect arrived in the input.

In practice, the supervisor's delegation format is the highest-leverage prompt in the entire system. It deserves proportionate attention.

Three things to watch in production

• Task-spec quality. Are workers asking follow-up questions or silently guessing? Both indicate the same defect. Sample delegated tasks and read them as if you were the worker: could you act on this without inventing requirements?
• Supervisor context growth. Long missions inflate the tracking state until decomposition quality degrades. The supervisor gets worse over the course of a task, which is a genuinely confusing failure to debug if you are not expecting it.
• Completion discipline. The supervisor owns "done." If it accepts weak worker output to keep things moving, the entire system inherits that standard, and nothing downstream will catch it.

The supervisor keeps everyone aligned, and it is where alignment failures concentrate. That trade is a good one while you are learning to operate multi-agent systems.

Fancier topologies come next in this series. A spoiler worth having early: they earn their keep less often than the architecture diagrams suggest.

Day 42 of 60 Days of Production AI Systems.

What should your supervisor decide, and what should it leave entirely to the worker?

Previous: Day 41: Why one giant agent is rarely the cleanest design
Next: Day 43: When hierarchy helps multi-agent work scale

Day 42 of 60 · Agent Control and Supervision (chapter 7 of 10)

Multi-agent architecture has three honest justifications. Everything else is enthusiasm with a diagram.

The three reasons

Separation of permissions. The agent that reads everything should not be the agent that can write anything. That split is a security property no prompt can provide, and it alone justifies a second agent.

Genuine parallelism. Thirty documents to analyse independently is thirty agents' worth of simultaneous work. Sequential work gains nothing from a crowd. It just adds handoffs between steps that were already in order.

Context isolation. An agent with one job and only that job's context reliably outperforms one juggling five roles whose instructions interfere. "Be exhaustive" and "be concise" in the same system prompt produce an agent that is inconsistently both.

If none of the three applies, the single agent wins.

The coordination tax

Every additional agent levies one, and it is paid on every request, forever: handoffs that drop context, shared state that drifts, conflicts needing arbitration, and an entire category of failure that exists only between agents and therefore appears in none of your per-agent metrics.

That last point is the expensive one. When each component tests clean and the system still fails, the fault is almost always in the seams, and seams are what you bought when you split.

The test

Not "would specialists be elegant?" but: which of the three reasons applies here, specifically?

Write the answer down. If it takes more than a sentence, or if the sentence contains the word "cleaner," it is probably enthusiasm. One good agent with well-scoped tools beats a committee for most workloads, and it is dramatically easier to debug at 2am.

Start with one. Split when a boundary becomes real, when you genuinely cannot give one agent both sets of permissions, or when you are demonstrably waiting on work that could run in parallel.

Closing thought

Specialisation can beat one giant agent. It can also just be one giant agent with extra network calls and a nicer architecture diagram.

The complexity is a loan. Make sure you know what you borrowed it for.

Day 41 of 60 Days of Production AI Systems.

Which responsibility in your current agent is genuinely a different specialist job?

Previous: Day 40: Why autonomy needs budgets
Next: Day 42: Why supervisor agents help and where they bottleneck

Day 41 of 60 · Agent Control and Supervision (chapter 7 of 10)