One Trace per Run: Observability for Multi-Agent Systems
Logs show what ran, not why a run went wrong. Give every run one trace, keep a ledger of intent, watch seven golden signals and gate every change on offline evals.
Written as at 23 August 2026.
agentic-aimulti-agent-systemsobservabilityopentelemetryai-evaluation
A multi-agent run fails in a way a single program rarely does. Every call succeeds, every log line says 200 OK, and the result is still wrong, late or ten times the usual price. Nothing crashed. The system simply did something other than what you intended, and the logs have no way to say so.
At Poly, the studio I am building, a lesson build fans out to a researcher, a scriptwriter, a visuals agent, a quiz writer and an evaluator. Each makes its own model and tool calls. Without a thread through them, the question “why did this lesson cost what it did?” has no answer.
Logs tell you what ran, not why the run went wrong
Consider an illustrative runaway loop. One lesson costs $14.20 against a $0.60 median and takes 118 turns without ever being marked done. The logs show 118 successful model calls, spread across two agents, interleaved with every other run that day.
A trace shows the story at once. The evaluator and scriptwriter spans repeat as a pair 27 times, each model call slightly larger than the last, and the evaluator’s reasons alternate between “not engaging enough” and “too informal”. The cause is a rubric criterion nobody could measure and no cap on revisions. You can only see that shape if every span in the run shares one identifier and knows its parent.
Give every run one trace, and every step a span
Assign a trace_id at the edge gateway when the request arrives, and never accept one from the client. Pass it, with the parent span, inside every brief the orchestrator sends, so each worker’s calls join the same tree. If a worker starts its own trace, you will see that a run was expensive but not which agent or turn made it so.
Name the spans after the OpenTelemetry semantic conventions for generative AI: invoke_agent for each agent, chat for each model call and execute_tool for each tool call. The conventions are still evolving, so pin the version you emit. The payoff is portability: any OpenTelemetry backend can read the result.
invoke_agent course_builder
├─ invoke_agent orchestrator
│ ├─ chat (plan)
│ ├─ invoke_agent researcher
│ │ ├─ chat
│ │ └─ execute_tool web_search
│ └─ invoke_agent evaluator …
└─ guardrail.output
Put the same attributes on every span: gen_ai.usage.input_tokens and output_tokens, cache read and write tokens, the model, the agent name and the task_id. Add the stop reason on model calls, and the tool name, an argument hash and the policy decision on tool calls. Compute cost at export, not in the agent. Sample prompts and completions, and redact them; personal data does not belong in a trace store.
The ledger states intent; the trace records what happened
A trace on its own shows behaviour. To call it a failure you need a statement of what should have happened. That is the task ledger: every sub-task with an owner, dependencies, acceptance criteria, a budget, an idempotency key and a status. In Poly’s build the ledger says research, diagrams and quiz run in parallel, the script waits for research, and quality review runs once, twice at most.
Comparing the two is the core of Agentic Flow Mining, Contextual Agentic’s discipline of extending process mining (van der Aalst) and observability to discover, check and improve how agents, tools, humans and policies actually executed. The ledger allows two review passes and the trace shows 27; that gap is the finding. The ledger earns its keep a second way, too: if the orchestrator crashes, a new one reads it and resumes from the last completed task.
Seven golden signals, each pointing into the trace
Traces explain single runs. Across hundreds of runs you need a handful of signals that say whether the system is drifting, and each should tell you which spans to open next.
| Signal | What it catches | Where to look in the trace |
|---|---|---|
| Task success rate, against acceptance criteria | Quality regressions | The evaluator’s verdict spans |
| Cost per successful task | Waste, and cheap runs that fail | Usage attributes grouped by agent |
| Turns per task | Loops starting to form | Repeating span pairs |
| p95 time to first token | Oversized windows, capacity limits | Model calls with large uncached input |
| Tool error rate, with policy denials | Rate limits, broken tools, injection attempts | Tool-call status and policy decision |
| Peak window per agent | Early warning of context rot | Input tokens on each model call |
| Human escalation rate | Limits being hit, or quietly bypassed | Escalation spans and their reasons |
Read Figure 1 as a triage list. The researcher’s 118,000-token peak is the treat: fetched pages are probably arriving verbatim, so the fix is a per-page cap in the tool wrapper. Rising turns per task is a watch that may be the same loop forming. Cost per successful task is covered in The Arithmetic of Agent Cost. To see a trace waterfall build as a run plays, open the agent simulator.
Offline evals catch what live signals cannot
Live signals tell you something changed after it shipped. To catch a regression before release, keep a golden set: lesson briefs with known-good outputs and a few known-tricky facts. Picture a lesson that states the prompt cache lasts 24 hours and still passes review, because the research summary dropped its citation and the evaluator never opened the source file. That case belongs in the golden set.
Run the set in CI on every change to a prompt, a model or a tool, score it against the same rubric the evaluator uses, and block the merge on a drop. Sample a slice of live runs for the same scoring. The quarterly agent check-up then reads the trend rather than a single snapshot.
What observability doesn’t solve
A trace explains; it doesn’t enforce. A complete trace of a harmful action is an excellent record of the harm, which is why authority checks belong on the execution path, not in the dashboard. Observability also can’t define success. Someone has to write acceptance criteria a machine can check, and a vague rubric will produce clean telemetry of a system that never finishes.
Who does what
Executives: ask for task success rate and cost per successful task for each agent workflow, and for the date the golden set last blocked a release.
Architects: make the trace_id and parent span mandatory fields in every brief and result envelope, and design the ledger so intent can be compared with the trace.
Engineers: emit OpenTelemetry GenAI spans with usage and policy attributes, redact prompts before export, and wire the golden-set eval into CI as a blocking check.
This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.
References
- The Contextual Agentic Enterprise.
- OpenTelemetry, "Semantic conventions for generative AI systems" (accessed 24 September 2026). https://opentelemetry.io/docs/specs/semconv/gen-ai/
- van der Aalst, W. M. P. (2016) Process Mining: Data Science in Action, 2nd edn. Springer. https://doi.org/10.1007/978-3-662-49851-4
Related
The Arithmetic of Agent Cost
Agent cost is mostly input re-read on every turn. Multi-agent designs may read more tokens yet cost less per result, so budget each task and measure cost per successful task.
Agentic Flow Mining: Discovering How Agents Actually Work
Traces show that each step succeeded. They don't show whether the steps formed an acceptable process. Mining agent runs against invariants does.
The Agent Check-up: A Quarterly Health Review for Agents That Can Act
Agents drift. Their models change, their tools multiply, their grants linger and their context goes stale. A quarterly check-up with the same eight vital signs catches it before an incident does.
Evaluation Is Not a Test Phase
Most agentic programmes pick a model, write a prompt, add retrieval, and write the evaluations last — by which point the evaluations can no longer change the design. Reversing the order gives an architect something better than an opinion: gates that show whether a model swap, a prompt change or a new retrieval strategy measurably improved the system.