Contextual Agentic

Ten Ways Multi-Agent Systems Fail, and How to Find Them

In a multi-agent system the symptom and the cause sit in different layers. A fixed triage order finds the fault, and most fixes belong in code and contracts, not prompts.

·6 min read

Written as at 24 August 2026.

agentic-aimulti-agent-systemsreliabilitytroubleshootingai-operations

A multi-agent system seldom fails where the fault is. At Poly, the studio I am building, a producer notices that a lesson never finishes; the cause is a line in an evaluation rubric written weeks earlier. A team sees its model bill double after a release, yet every token count is unchanged. The symptom and the cause sit in different layers, and often in different agents.

So the first skill in running these systems is triage: a fixed order of questions that takes you from what someone noticed to the span, the setting or the contract that caused it.

Work top-down, then confirm bottom-up

Animated monitor for triaging one failed agent run. A heartbeat line runs across the top, with the status: run for lesson 3.2, 14 dollars 20 against a 60 cent median, 118 turns. Six triage checks are scanned in order and stamped. One, ledger: task T4, the script, has been retried 27 times and is not done, treat. Two, trace: the evaluator and scriptwriter span pair repeats 27 times, treat. Three, stop reason: end turn on every call, healthy. Four, window: the scriptwriter window grows every turn, watch. Five, tool: no errors and no policy denials, healthy. Six, platform: no rate-limit errors and normal time to first token, healthy. The diagnosis at the end reads: runaway evaluator loop; make the rubric measurable, cap revisions at two, then escalate, and enforce the budget per task in the ledger.
Figure 1. Six checks, in order, for one failed run. The run and its values are examples. Animated; shown as a still image if your device is set to reduce motion.
  1. Ledger. Which task is not done? The task ledger records what should happen: owner, dependencies, acceptance criteria, budget and status for every sub-task. Compare it with what did.
  2. Trace. Which span failed, repeated or ran long? With one trace per run and a span for each agent, model call and tool call, this is a query rather than an investigation (see One Trace per Run).
  3. Stop reason. Did the model end its turn, hit max_tokens, request a tool or refuse?
  4. Window. How large was the context at the failing turn?
  5. Tool. What came back: the HTTP status, the policy decision, the arguments as received?
  6. Platform. Only now look at rate-limit errors, overload responses, time to first token and regional capacity.

The cheapest, most informative checks come first; the platform comes last because it is the easiest thing to blame. Then confirm by working back up: the lowest finding should explain every layer above it.

In Figure 1 the run cost $14.20 against a $0.60 median and took 118 turns. The ledger shows the script task retried 27 times, the trace shows the evaluator and scriptwriter trading the same pair of spans, and everything below looks healthy. That is a runaway loop, and its fix lives in the rubric and the budget, not the model.

Every failure has a symptom, a cause and a fix

These ten cases come from the failure lab in my Agent Pattern Atlas, the working reference behind this series. The Poly figures are illustrative.

# Failure What you see Root cause Fix
1 Runaway evaluator loop One lesson costs $14.20 against a $0.60 median Unmeasurable rubric line, no revision cap, budget never passed down Measurable criteria; two revisions, then a person; budget checked before each delegation
2 Prompt injection Policy denials for send_email from the researcher Hidden text in a fetched page steered the researcher Keep the tool allowlist; tag untrusted results; quarantine the domain
3 429 storm at fan-out 9 of 16 workers fail within 4 seconds, and retries fail too Burst past the tokens-per-minute limit; retries without jitter Concurrency limit sized to the rate limit; backoff with full jitter; honour retry-after
4 Context overflow Prose instead of JSON, or “prompt is too long” 30 raw pages in one window, 186k tokens Summarise in the tool wrapper; clear old results; restate the output contract at the end
5 Confident wrong fact A lesson says the cache lasts 24 hours The summary dropped its citation; QA checked style only Citations required in the result schema; claim-by-claim source check
6 Duplicate side effect The lesson appears twice in the draft area Publish timed out after the server committed; no idempotency key Key from the ledger to the API; check before acting; human-approved publish
7 Orchestrator crash “Running” for 40 minutes with no progress Plan and progress lived only in process memory Durable ledger with leases and heartbeats; resume from the last completed task
8 Cache miss after a release Cost up 2.4× with the same token counts A timestamp in line 1 of the system prompt; tools listed in random order Dynamic values after the cache breakpoint; sorted tools; hash the prefix in CI
9 Slow first token p95 of 9 seconds against a 1.5-second baseline Workers handed the whole transcript, so prefill is huge Pass briefs, not transcripts; cache the stable prefix; stream status
10 Over-delegation “Fix a typo” starts six agents No rule for scaling effort; vague task boundaries Effort rules by request class; a single-agent path first

Case 4 is the context rot that Hong, Troynikov and Huber measured: performance falls as input grows, even on simple tasks. Case 10 is why Anthropic’s multi-agent research write-up publishes effort-scaling rules, from one agent with 3–10 tool calls for simple fact-finding to more than ten subagents for complex research.

Now count how many fixes touch a prompt. Two do: the rubric and the restated output contract. The other eight are limits, keys, schemas, allowlists and durable state. That is the Deterministic Shell / Probabilistic Core, my model for this split: let the model reason and propose inside the core, and make identity, authority, schemas, state and commitment deterministic in the shell around it (more here).

Three of the fixes are older than the models

Backoff with jitter. Marc Brooker’s Builders’ Library article explains why retries on a fixed schedule arrive together and deepen an overload, and why randomised delays spread them out. Case 3 is that thundering herd, with sixteen workers in place of sixteen clients.

Circuit breakers. Michael Nygard’s Release It! describes a breaker that stops calling a failing dependency for a while. In the worker failure scenario of the agent simulator, the diagram renderer times out twice, the breaker opens, and the worker returns a typed failure (status: failed, reason: tool_unavailable). The orchestrator re-briefs visuals to a fallback renderer, in code.

Compensation. Some effects cannot be retried away. Garcia-Molina and Salem’s sagas split a long transaction into steps, each with a compensating action. A lesson published twice needs a compensating withdrawal, which is why case 6 treats publishing as a Commit Boundary, my term for the point where a proposal becomes an externally observable effect. Checkpointing Is Not Rollback covers the recovery side.

Loops need stop conditions the model cannot argue with

Case 1 is the most expensive because nothing stops it. Loop Engineering, my model for treating an agent’s iteration as a control loop with progress signals, budgets and stop conditions enforced outside the model, gives three settings to check. Criteria must be measurable: runtime under eight minutes, a reading grade, every claim cited. Revisions are capped, then a person decides. The budget is checked in the ledger before each delegation, not only at the edge. Alert on turns per task above the 95th percentile (Loop Engineering).

Rehearse failures before production does

Call every write tool twice in a contract test. Kill the orchestrator mid-run in a chaos test. Load-test the fan-out ceiling. Keep a golden set of known-tricky facts and a poisoned-page corpus in CI. Cemri and colleagues’ taxonomy of multi-agent failures is a useful research companion.

Triage has limits. It finds faults that leave marks in a ledger and a trace, so without those it has nothing to read. It will not catch a lesson that passes every check and is still wrong in a way no rubric measures, and it cannot decide whether a failure is acceptable. That is a product decision.

Executives: ask for cost per successful task and escalation rate per agent, and fund failure rehearsal before go-live rather than after the first incident.

Architects: make the ledger, trace IDs, idempotency keys and budgets part of the reference design, so every new agent inherits them.

Engineers: put the triage order in the runbook, turn the ten cases into tests where you can, and replay the simulator’s failure scenario before your first on-call shift.

This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.

References

  1. The Contextual Agentic Enterprise.
  2. Brooker, M. "Timeouts, retries, and backoff with jitter". Amazon Builders' Library. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
  3. Nygard, M. (2018) Release It! Design and Deploy Production-Ready Software. 2nd edn. Pragmatic Bookshelf.
  4. Garcia-Molina, H. and Salem, K. (1987) "Sagas", ACM SIGMOD Record 16(3). https://doi.org/10.1145/38714.38742
  5. Hong, K., Troynikov, A. and Huber, J. (2025) "Context Rot: How Increasing Input Tokens Impacts LLM Performance". Chroma. https://research.trychroma.com/context-rot
  6. Anthropic (13 June 2025) "How we built our multi-agent research system". https://www.anthropic.com/engineering/multi-agent-research-system
  7. Cemri, M., Pan, M. Z., Yang, S. et al. (2025) "Why Do Multi-Agent LLM Systems Fail?", arXiv:2503.13657. https://arxiv.org/abs/2503.13657

← All articles

Related