Contextual Agentic

The Orchestrator Should Never Read the Raw Material

Every turn re-reads the whole window, so the agent that plans should reason over summaries and pointers while workers read raw material in windows they throw away.

·5 min read

Written as at 20 August 2026.

agentic-aimulti-agent-systemscontext-engineeringprompt-cachingcontext-window

At Poly, the studio I am building, a team of agents produces course lessons. For Lesson 3.2, on prompt caching, a researcher pulls about 38,000 tokens of web pages, a quiz writer retrieves nearly 10,000 tokens of syllabus and a visuals agent renders three diagrams. Together they gather about 50,000 tokens of raw material. The orchestrator that plans the lesson and decides when it is finished sees about 1,450 of them.

That gap is deliberate. It is the most important context decision in a multi-agent system, and the one most often got wrong.

Every turn re-reads the whole window

A model keeps no memory between calls. On every turn, the agent runtime sends the whole message list again: system prompt, tool definitions, history and the latest tool result. So the size of the window drives three things at once.

It drives cost, because input tokens are billed on every turn they are re-sent. It drives latency, because every uncached token has to be processed before the first token of the answer appears. And it drives quality. Liu and colleagues found that models use information at the start and end of a long input better than information buried in the middle. Chroma’s context rot study found that performance becomes less reliable as input grows, even on simple tasks. Anthropic’s context engineering guidance puts the conclusion plainly: treat context as a finite resource.

In the Lesson 3.2 build, a single agent doing the whole job would carry about 86,000 tokens by the end and re-read most of them on every turn. The orchestrator in the multi-agent version peaks at about 10,000.

Workers read; the orchestrator reasons over conclusions

The move that keeps the orchestrator small is context isolation. Each worker gets its own window, holding only its brief, its tools and its results. The researcher reads its pages there, writes a 7,900-token research.md to the artifact store, and returns an envelope with an 850-token summary and the file’s URI. Then its window is discarded.

When the scriptwriter needs the research, it fetches research.md by that pointer, just in time. It reads 7,900 tokens of curated material rather than 38,000 tokens of raw pages, because the researcher has already done the filtering.

The same shape works outside a studio. Picture a motor claims team where a document agent reads a 60-page assessor’s report and three repair quotes. It hands the coordinating agent a few lines: third-party liability accepted, assessed repair $14,200 against a highest quote of $18,900, variance flagged, evidence on pages 12 and 41, and a pointer to the full extract. The coordinator decides the next step without ever holding the report.

Animated monitor of peak context windows for one multi-agent lesson build. A heartbeat line runs across the top. Below it, eight readings are scanned one at a time and stamped healthy, watch or treat. The orchestrator peaks at 10.2 thousand tokens because it holds only summaries and pointers: healthy. The researcher peaks at 45.6 thousand tokens with 38 thousand tokens of raw web pages in its window: watch. The scriptwriter peaks at 15.6 thousand, fetching research by pointer: healthy. The quiz writer peaks at 14.7 thousand: healthy. The visuals agent peaks at 6.8 thousand: healthy. The evaluator peaks at 7.9 thousand, holding a rubric and three pointers: healthy. Fan-in adds 1.45 thousand tokens to the orchestrator instead of 50 thousand: healthy. The single-agent equivalent would carry 86.6 thousand tokens and re-read most of them every turn: treat. The plan reads: trim each page in the tool wrapper, clear old tool results, split research past 100 thousand tokens, and alert on any window above 100 thousand.
Figure 1. Peak context window per agent for one lesson build, against the single-agent equivalent. Example values. Animated; shown as a still image if your device is set to reduce motion.

Six habits keep every window lean

These are the working rules I use at Poly. None of them is exotic; the value is in applying all six to every agent.

  1. Isolate by role. One window per worker, with only that worker’s brief, tools and results. Never pass a worker the orchestrator’s transcript.
  2. Return a summary and a pointer. Full output goes to the artifact store; the envelope carries a bounded, schema-checked summary and a URI. Next week’s article covers the envelope itself.
  3. Keep the prefix stable. Put the system prompt, tool definitions and fixed documents such as a style guide first, and never change them mid-run. Prompt caching matches an exact prefix, and cache reads bill at a fraction of the base input price (writes cost a premium), so a timestamp on line one quietly turns every call into a cache miss.
  4. Retrieve just in time. Keep identifiers and paths in the window, and fetch content only into the window that needs it, at the moment it needs it.
  5. Clear or compact. Drop stale tool results once they have been used, and summarise older turns when a window passes a threshold. Trim inside the tool wrapper too: a page capped at 2,000 tokens never floods a window.
  6. Load tools on demand. Fifty tool schemas can cost ten thousand tokens before any work starts. Give each agent only its own tools.

This is also where a construct from the book fits. A Context Contract, a CAE model, is a declared, versioned and testable specification of what context a given agent purpose may receive. The six habits are how you honour one at run time; Context Is Not a Prompt explains the contract itself.

Measure the peak window of every agent

Figure 1 treats each agent’s peak window as a vital sign. The researcher is marked watch, not because 45,600 tokens is too many, but because 38,000 of them are raw pages held verbatim. That is the agent that fails first.

In the failure cases I use for this course, it looks like this. On a long topic, the researcher fetches 30 pages, its input reaches 186,000 tokens, and it starts returning prose instead of the JSON its brief asked for. The instructions at the top of the window have lost their hold. The fix is a combination of the habits above: trim each page in the tool wrapper, clear old results, split the research across two workers, and restate the output contract at the end of the brief. Then alert on any agent whose peak window passes a threshold you set per model, such as 100,000 tokens.

You can watch the windows fill and empty in the fan-out scenario of the agent simulator.

A bigger window postpones the problem

The tempting fix is a model with a million-token window. It helps less than it seems. Every token in that window is still re-read on every turn, still paid for, and still competing for the model’s attention. Anthropic’s report on its research system found that multi-agent runs used about 15 times the tokens of a chat. Isolation is what stops those tokens from all landing in one place.

Isolation does not make a summary true. It moves the risk from a crowded window to a thin handover. In another of those cases, a research summary dropped its citation, the evaluator never opened the artifact, and a lesson went out saying the prompt cache lasts 24 hours. Require sources in the summary schema, and have the evaluator check claims against the artifact, not against the summary. Compaction loses detail in the same way, so decide in advance what must never be compacted.

Who does what

Executives: ask for the peak window of each agent alongside its cost. A system whose planner holds the raw material is paying for it on every turn.

Architects: design the handover before the agents. Decide what each worker returns, where the full output lives, and which content forms the stable, cached prefix.

Engineers: keep volatile values after the cache breakpoint, sort tool definitions deterministically, cap tool results in the wrapper, and emit the window size on every model-call span so the peak is visible.

This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.

References

  1. The Contextual Agentic Enterprise.
  2. Liu, N. F. et al. (2024) "Lost in the Middle: How Language Models Use Long Contexts". Transactions of the Association for Computational Linguistics. https://arxiv.org/abs/2307.03172
  3. Hong, K., Troynikov, A. and Huber, J. (2025) "Context Rot: How Increasing Input Tokens Impacts LLM Performance". Chroma. https://research.trychroma.com/context-rot
  4. Anthropic (September 2025) "Effective context engineering for AI agents". https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
  5. Anthropic (June 2025) "How we built our multi-agent research system". https://www.anthropic.com/engineering/multi-agent-research-system
  6. Anthropic documentation, "Prompt caching". https://docs.claude.com/en/docs/build-with-claude/prompt-caching

← All articles

Related