Day 40: Why autonomy needs budgets
Why autonomous systems need limits on cost, time, actions, retries, and risk.
Every team running agents eventually has the invoice conversation. The only variable is whether it happens beforehand, calmly, as a design decision, or afterwards, on a Monday, following a weekend where one stuck agent discovered it had no reason to stop.
Autonomy without budgets is not autonomy. It is an open tab.
| Budget | Example limit | What it prevents |
|---|---|---|
| Time | 90 seconds | Hidden latency users silently absorb |
| Tool calls | 8 per task | Operational blast radius |
| Cost | $0.25 per task | Unpredictable unit economics |
| Retries | 2 attempts | Loops that look like productive work |
| Risk | human review above threshold | Unaccountable high-impact actions |
Define the behaviour at the threshold
Setting a limit is the easy half. The half that gets skipped: what happens when the agent reaches it?
Hitting a budget should produce an outcome, not an exception. At 80% consumed, the agent should be wrapping up, summarising progress, returning partial results, or handing to a human with state attached. An agent that hits a hard cap mid-action and dies leaves you exactly the "failed halfway" mess from yesterday, except now you caused it deliberately.
That distinction is worth designing explicitly: a graceful wind-down at the soft limit, a hard stop at the ceiling.
Scarcity improves planning
The benefit that surprises people is not cost control. It is behaviour.
An agent that knows it has five tool calls remaining plans differently (and usually better) than one operating as though resources are infinite. It prioritises. It stops exploring branches with low expected value. You can watch trace quality change when the remaining budget is made visible to the agent itself rather than merely enforced around it.
Unbounded agents are not more capable. They are less deliberate, because nothing forces a choice about what matters.
The retry trap
Worth isolating, because it is the most common way budgets get consumed invisibly: retries look like work.
Each individual retry is defensible. The trace shows an agent diligently attempting to complete its task. What it does not show, unless you are counting, is that attempts three through eleven were the same failing approach with cosmetic variation.
Cap retries separately from total actions. They are a different failure mode and they need their own ceiling.
Closing thought
Budgets are how autonomy becomes governable, and how it becomes something finance will approve at scale rather than tolerate as an experiment.
Day 40 of 60 Days of Production AI Systems.
Previous: Day 39: Why production agents need recovery design
Next: Day 41: Why one giant agent is rarely the cleanest design
Day 40 of 60 · Agent Control and Supervision (chapter 7 of 10)
