Why evaluating the model is not enough

Every component passed its tests. The system still failed the user.

The researcher retrieved correct facts: pass. The writer produced clean prose: pass. The handoff between them dropped the one constraint that mattered. That is measured nowhere, because it belongs to neither component.

Score the team, not just the parts

Component metrics tell you where a fault is. System metrics tell you whether there is one. You need both, pointed in different directions.

{
  "task_success":   "Did the user get the outcome they needed?",
  "grounding":      "Are material claims supported by gathered evidence?",
  "coordination":   "Clean handoffs, or duplicated work and dropped context?",
  "efficiency":     "Tokens, latency, retries per completed task",
  "policy":         "Were permissions and safety rules honoured?",
  "intervention":   "How often did a human have to step in?"
}

Coordination is the one almost nobody scores, and in multi-agent systems it is where the failures actually live. Handoff completeness, duplicated effort, context lost between agents: none of it appears on any individual agent's report card, by construction.

Intervention rate is the honest measure of autonomy

Give this one its own dashboard.

If the rate at which humans rescue runs is not declining over time, the system is not maturing. It is just running, and the demo is doing more work than the architecture. Teams celebrate shipping an agent and then never track whether it needs less help this quarter than last.

A rising intervention rate on a stable workload is the earliest signal that something upstream has drifted (data, model version, user behaviour) usually well before any quality metric moves.

A correct answer can still be a failure

This is what jobs-to-be-done evaluation catches and answer-quality scoring misses. A response that is accurate, well-grounded, and takes ninety seconds when the user needed five is a failure. So is one that quietly cost forty cents to produce at a scale where you priced for four.

Build scenario tests that resemble real work, full runs with real inputs, scored on outcome rather than output.

Evaluate the team, not just the model. Impressive parts adding up to an unreliable whole is the characteristic failure of multi-agent systems, and per-component metrics are structurally incapable of detecting it.

Day 56 of 60 Days of Production AI Systems.

What score would prove your agent is useful, rather than merely active?

Previous: Day 55: Why observability is not optional for AI systems
Next: Day 57: Why retrieved content must stay untrusted

Day 56 of 60 · Production AI Readiness (chapter 10 of 10)