Day 39: Why production agents need recovery design
Why retries, fallbacks, checkpoints, rollback, and escalation belong in the first design.
The most dangerous agent state is not "failed." It is failed halfway.
Three of seven records updated. The notification sent but the database write lost. A workflow abandoned mid-transaction with no record of how far it got. A clean failure you can simply rerun; a partial one you have to untangle by hand, usually while someone waits.
Demo agents are built for the happy path. Production agents are built for the Tuesday when the API times out at step four.
The recovery vocabulary
Five distinct mechanisms, each answering a different failure class. Teams tend to implement one and call it recovery:
- Retry, for transient failures only. With backoff and a hard cap. Uncapped retries against a struggling dependency turn your outage into their outage.
- Fallback, a simpler tool, an older data source, a degraded answer that is honest about being degraded.
- Checkpoint: durable progress markers, so resuming means continuing rather than restarting. This is what turns "failed halfway" from an incident into an inconvenience.
- Rollback, compensating actions for anything externally visible. If step 5 fails, steps 1 through 4 need an undo story, and "we'll fix it manually" is that story until you write a better one.
- Escalate: hand to a human with state attached. "Task failed" is a notification. An escalation includes what was attempted, what succeeded, what is now half-done, and what decision is needed.
The distinction that matters most: retry is for transient failures, not for wrong assumptions. An agent that retries a bad plan three times has not recovered. It has failed three times more expensively, and the trace will show three reasonable-looking attempts.
Test the paths deliberately
This is the discipline that separates teams who have recovery from teams who have recovery code.
Kill the API mid-task in staging. Return a malformed response. Time out the third of five tool calls. Then watch exactly what the agent does.
Whatever it did just now is what it will do in production, at 2am, on a real customer's request. Most teams discover on their first attempt that the agent cheerfully continued as though the failed step had succeeded, because nothing in the loop was checking.
Reliability is not the absence of failure. It is designed recovery — decided before the failure, tested before it matters.
An agent that fails loudly, safely, and resumably is worth ten that only work when everything else does.
Day 39 of 60 Days of Production AI Systems.
Which of these five paths did your last incident actually exercise?
Previous: Day 38: Why guardrails should enable safe action
Next: Day 40: Why autonomy needs budgets
Day 39 of 60 · Agent Control and Supervision (chapter 7 of 10)
