Contextual Agentic

Capstone: Designing a Course Factory That Builds 400 Lessons

One agent-built lesson is a demo. Four hundred overnight is an architecture: budgets, a revision cap, a citation check and a person who approves what is published.

·5 min read

Written as at 26 August 2026.

agentic-aimulti-agent-systemsarchitecture-reviewai-operationscost-engineering

For eleven weeks this series has followed one lesson through Poly, the studio I am building: Lesson 3.2, “Prompt caching”, made by an orchestrator and five workers. One lesson is a demonstration. The question Poly has to answer is what happens when the same design is asked for a full catalogue of 400 lessons, overnight, with nobody watching.

At that scale every decision from the earlier pieces holds or breaks at once. This capstone is the design review I would want before the first overnight build.

Scale turns small defects into large bills

A retry without jitter is a nuisance on one lesson and a rate-limit storm on four hundred. Rate limits apply to the whole organisation, so four hundred lessons with five workers each all draw on the same tokens-per-minute allowance. A reviewer who reads ten lessons is diligent; nobody reads four hundred before breakfast. And cost behaves differently at volume. Anthropic reported that its multi-agent research system used about 15 times the tokens of a chat interaction, so a multi-agent design has to earn its tokens back through tiering, caching and small windows.

Some illustrative arithmetic, using the example figures from my atlas rather than real Poly data. With a median of $0.60 a lesson, the night should cost about $240. If one lesson in twenty behaved like the runaway loop in Ten Ways Multi-Agent Systems Fail, at $14.20 each, the night would cost about $510. With a hard budget of $1.50 per lesson enforced in code, the worst case is $600, whatever the model does.

Every lesson meets the same four gates

Animated diagram of Poly's course factory. Three lesson builds arrive from the overnight batch: lesson 3.2 on prompt caching, lesson 5.4 on rate limits, and lesson 6.1 on its second revision. Each passes through a release gate with four checks: budget, evaluation rubric, citation check and human approval. Lesson 3.2 passes all four and goes to the publish queue. Lesson 5.4 is within budget but fails the rubric because its script runs 9 minutes 10 against an 8-minute limit, so it is sent back for revision. Lesson 6.1 passes budget and rubric but fails the citation check on its second revision, so the revision cap is reached and it is escalated to a human editor. Every outcome writes a row to the task ledger.
Figure 1. Every lesson meets the same four gates, and every outcome is written to the ledger. Example lessons and values. Animated; shown as a still image if your device is set to reduce motion.

Budget. Checked in the ledger before each delegation, not only at the end.

Rubric. Measurable criteria only: script runtime under eight minutes, a target reading grade, quiz answers that match the script. A criterion such as “make it more engaging” is how an evaluator loop runs forever.

Citations. Every factual claim traces to a source in the artifact store. Volatile facts, such as a cache lifetime or a price, come from a maintained reference file rather than the model’s memory.

Human approval. Publishing is a Commit Boundary, my term for the point where a proposal becomes an externally observable effect. At Poly that means an editor approves each module’s release with every lesson’s gate record in view, reads escalated lessons in full, and reads a sample of the rest.

Three outcomes follow: publish, revise, or escalate. The revision cap is two. This is Loop Engineering, my model for treating iteration as a control loop whose progress signals, budgets and stop conditions are enforced outside the model (Loop Engineering).

A design review comes down to eight questions

# Question A good answer at Poly Covered in
1 Is orchestrator-workers justified? Yes for a full lesson; a typo fix takes the single-agent path Start With One Agent
2 Where do plan and progress live? A durable ledger: owner, dependencies, criteria, budget, idempotency key and status per task Fan-Out, Fan-In
3 What stops a run? A per-lesson budget and a turn limit, checked in code The Arithmetic of Agent Cost
4 How does the night run? Batch for steps nobody waits on; fan-out limited to the rate limit; retries with jitter Ten Ways Multi-Agent Systems Fail
5 Who decides what “good” means? A separate evaluator, a measurable rubric, a golden set run in CI on every prompt change One Trace per Run
6 Which actions need a person? Publishing and every escalated lesson, shown as the exact lesson with its evidence The Commit Boundary
7 Which boundaries hold? The researcher reads the web but has no send or publish tool; no agent holds all three legs of the lethal trifecta Six Trust Boundaries
8 What does “working” mean? Service levels such as lessons through the gate by 7 am, cost per successful lesson and escalation rate The Agent Check-up

Question four carries the biggest saving. Anthropic’s batch processing gives about a 50 per cent discount for asynchronous work, and an overnight build is exactly the work nobody is waiting for. Question seven is Simon Willison’s lethal trifecta: an agent that combines private data, untrusted content and external communication can be steered by one poisoned page into sending the data out. Poly’s answer is structural. The researcher that reads the web cannot send or publish; the publisher sees only schema-checked fields.

Question eight needs numbers someone signs. For Poly, as examples: 95 per cent of lessons through the gate by 7 am, cost per successful lesson under $1, and no more than one lesson in twenty escalated. At that ceiling the editor starts the day with twenty lessons to read in full: a morning’s work, not a week’s.

Replay the full lesson build in the agent simulator with prompt caching switched on and off, and the cost and window gauges show why questions three and four matter.

What a review cannot promise

A good review does not make the lessons good. It makes them measurable, bounded and recoverable. It will not catch the lesson that is accurate, cited and dull, and it cannot set the editor’s standards. After the first few nights, the ledger and traces show how the factory actually ran. Agentic Flow Mining, the CAE discipline of extending process mining to how agents, tools and people actually executed, compares those paths with the design and finds the drift (Agentic Flow Mining).

An architect owns the shape, not the prompts

Across twelve pieces the pattern has been consistent. The architect owns the pattern choice, the contracts between agents, the budgets and stop conditions, the trust boundaries, the evidence and the service levels. Above all, the architect owns the decision about where a person must stay in the loop, and makes sure that decision is enforced in code.

Executives: ask for the review answers in writing before the first overnight run, with a named owner for each.

Architects: run these eight questions against every multi-agent design, and refuse a design that answers “the prompt handles it”.

Engineers: build the gates, the ledger and the budget check first, then the agents; they are what make a bad night cheap.

This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.

References

  1. The Contextual Agentic Enterprise.
  2. Anthropic (13 June 2025) "How we built our multi-agent research system". https://www.anthropic.com/engineering/multi-agent-research-system
  3. Anthropic. Batch processing (documentation). https://docs.claude.com/en/docs/build-with-claude/batch-processing
  4. Willison, S. (16 June 2025) "The lethal trifecta for AI agents: private data, untrusted content, and external communication". https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

← All articles

Related