What an A/B Test Cannot Tell You About an Agent

For: architects, SRE and evaluation leads, and anyone being asked to “just A/B test it”. The decision: what evidence would actually justify giving an agent more authority.

Somebody will suggest an A/B test. It is the right instinct — it is how a serious engineering organisation settles an argument, and refusing it looks like superstition. So it is worth being precise about where it works, where it does not, and what to do instead, because the failure mode here is not a bad experiment. It is a good experiment answering a question nobody needed answered.

Four assumptions, and an agent breaks all of them

A controlled online experiment rests on four things:

1. A stable treatment. Everyone in B gets B, and B does not change during the test.
2. Independent units. One unit’s assignment does not affect another’s outcome.
3. An observable outcome, attributable to the variant, within the window.
4. Enough events that the difference you care about would show up.

Traditional software satisfies all four comfortably. A checkout button is the same button all week, my checkout does not affect yours, the purchase happens in minutes, and conversion is a percentage point or two on a large base.

Now the agent.

The treatment is not stable. Sampling means the same input produces different trajectories, so “B” is not a thing — it is a distribution. And the provider’s model may be updated beneath you mid-test. You set out to compare two systems and ended up comparing two distributions, one of which moved.

The units are not independent. Prompt caches are shared, so an arm that runs second is cheaper and faster for reasons that have nothing to do with its merits. Rate limits are shared, so one arm’s burst is the other arm’s latency. Tool backends are shared. And if either arm writes to a store the other reads — a memory, an index, a task ledger — the interference is not subtle, it is causal. Multi-agent systems have agents that interact by design; that is what makes them multi-agent.

The outcome arrives late, or never. You can measure latency, tokens and a thumbs-up today. The thing you actually care about — was the assessment correct, did the action need reversing, did the customer come back — surfaces at appeal, in a reconciliation, or in a complaint three weeks later. So teams measure what is available and quietly redefine success as the thing they can see.

And the events you fear are too rare to test. This is the one that decides the argument, and it is arithmetic rather than opinion.

The arithmetic nobody runs first

Standard two-proportion sizing, two-sided α = 0.05, 80% power. Traffic is the 400-person division preset from the platform cost and reliability lab: 22,680 runs a month, so 11,340 per arm.

Detecting a halving of your worst failure mode takes thirty-five years at this traffic. The model will have been replaced a dozen times. Even the merely serious one — an unauthorised action attempt — needs three and a half years.

Put the other way round: inside a single quarter, this traffic can only detect the halving of a rate more common than about 1 in 723. Every failure mode that keeps an architect awake is rarer than that.

So the honest statement is not “A/B testing does not work for agents”. It is narrower and more useful:

An A/B test can tell you whether the agent got cheaper, faster or less annoying. It cannot tell you whether it got safer. Those need different evidence, and conflating them is how a system ships on the strength of a measurement that was never about the risk.

Deriving the test from the journey, not from the model

The common mistake is to start from the change — we swapped the retriever, let’s A/B it — and then look for a metric. Start from the journey instead, and the design falls out.

Take one business journey end to end and mark it up in four passes.

Pass 1 — find the decision points. Walk the journey and mark every place the system decides something: which case this is, what evidence is needed, whether to ask a person, whether to commit. Ignore the steps that merely transform data. Those are the points where a variant can change an outcome; everything else is plumbing.

Pass 2 — classify each decision by reversibility. Three buckets. Free — it can be redone at no cost, like a draft summary. Costly — redoing it burns a person’s time or a customer’s patience. Irreversible — money moved, access granted, a record changed, a message sent. This classification decides the experiment design, not the statistics.

Pass 3 — attach the metric that belongs to each decision. For a free decision, a fast proxy is fine. For a costly one, the metric is rework. For an irreversible one, the only honest metric is the rate of wrong commitments — and by the table above, you now know you cannot measure it in time. That is not a reason to skip it; it is the reason the irreversible decisions get a different mechanism, below.

Pass 4 — choose the unit of randomisation. Not the user, in most cases. The unit has to be the thing whose outcome you are measuring and which does not interfere with its neighbours:

Getting this wrong is the most common defect in agentic experiments, and it is invisible in the results: a badly chosen unit produces a confident number that is simply not about the change.

What a usable hypothesis looks like

Most stated hypotheses are not hypotheses.

Model B is better.

There is nothing there to be wrong about. Compare:

Replacing the retrieval step with the matter-scoped broker will reduce the proportion of claims runs requiring human rework from 8.0% to 6.5% or better, for task class C, randomised by run; with the escalation rate no worse than 2.8% and p95 cost per successful task no worse than $1.80; evaluated on an always-valid confidence sequence at α = 0.05; stopping at 4,700 runs per arm or at the first boundary crossing, whichever comes first.

That version names the change, the unit, the population, the effect size, the guardrails that must not move, the inference method, and the stopping rule — before anyone looks. Every one of those is a place a result goes wrong when it is left unstated.

Note the shape: one metric to improve, two that must not degrade. Most enterprise AI experiments are non-inferiority tests wearing superiority clothes. You are rarely trying to prove the new thing is better; you are trying to prove it is not worse on the things that would hurt. Say so, and the design gets simpler and more honest.

Four kinds of validity, and how each one fails here

Internal — did the difference come from the change? Threats specific to this domain: the provider silently updating a model mid-test; cache warming favouring whichever arm ran second; assignment leaking through shared memory so a “control” run reads what a “treatment” run wrote. Pin a version set — model, prompt, policy, tool and retrieval versions together — at the start of every run, and record it. If you cannot say which version set produced a result, you do not have an experiment, you have a log.

External — will it still hold next month? Lower here than anywhere else in software. Your win was measured against a model that will be deprecated. Treat every result as having a half-life and write the expiry into the decision record.

Construct — is the metric the thing you care about? A thumbs-up measures satisfaction with an answer, not correctness of an action. Deflection measures that nobody escalated, which is also what happens when people give up. This is where most programmes deceive themselves, because the available metric is always more available than the right one.

Statistical conclusion — is the number real? You will peek; everyone peeks. Fixed-horizon tests are invalid under peeking, and the error is large, not academic. Use always-valid inference so peeking is allowed by construction. And if you are watching twelve metrics, you will find a significant one at α = 0.05 roughly half the time by chance — pre-register the one that decides, and treat the rest as guardrails, not findings.

What to do instead, for the things you cannot test

Four mechanisms cover the ground an A/B test cannot reach.

Shadow running. Run B on real production traffic alongside A, and commit only A’s actions. You get a distributional comparison on the real task mix with a blast radius of zero, and you can run it for months because nothing is at stake. This is the workhorse for anything irreversible, and it is badly underused. Its limit: it tells you what B would have proposed, not what would have happened downstream once the world reacted.

Replay. Re-run historical trajectories against the new version and diff the decisions. This is the cheapest evidence available, and it works only if the evidence record is complete enough to replay — which is a concrete reason to build the evidence rail properly rather than as telemetry.

Ramp authority, not traffic. This is the CAE-native answer, and it is the one that actually resolves the arithmetic problem. Instead of moving from 5% to 50% of users, move the authority the variant holds: run B on 100% of traffic in recommend-only mode where it can be wrong for free; then let it commit reversible actions; then low-value irreversible ones under a limit; then higher. Each rung is earned with evidence appropriate to that rung — offline sets and shadow runs for the early ones, accumulated clean production runs for the later ones. Authority becomes the thing you ramp, which is also the thing you can revoke in one step when a rung turns out to be premature.

Guard the tail with mechanism, not measurement. You cannot measure your way to confidence about a one-in-ten-thousand event in any useful timeframe. So do not try. Bound it instead: a transaction limit, an invariant that refuses rather than warns, an approval bound to the exact payload. A control that makes the catastrophic action impossible needs no statistical power at all, which is the whole reason the deterministic shell exists.

Observability when there are millions of runs

Experiments need evidence, and evidence at this scale has its own design problems.

The unit of observation is the trajectory, not the request. A run is dozens of model and tool calls. Per-request dashboards will show you healthy calls inside a run that failed as a whole — the subject of One Trace per Run.

Version sets, not version labels. Tagging spans with model, prompt, policy, tool and retrieval versions separately produces a combinatorial mess and no way to group. Pin one version-set identifier per run and attach that.

Sample by consequence, not uniformly. You cannot keep every trace, and uniform sampling is the wrong economy: it keeps a representative slice of the boring and throws away the rare, which is exactly the population you need. Keep every run that committed something, was refused, was escalated, or hit an invariant. Sample the ones that did nothing consequential, hard. The rare events are rare, so retaining all of them costs almost nothing and is the only way the tail is ever visible.

A trace is not an evidence record. They have different purposes, retentions and access rules. A trace is for an engineer debugging next Tuesday and should be short-lived and sampled. An evidence record is for reconstructing a decision years later, is retained deliberately, and should hold references rather than payloads. Merging them gives you a privacy problem and an audit gap at the same time.

The components this needs, and what breaks without each

None of the above is a practice you can adopt by deciding to. Each rests on a component, and the failure when it is missing is specific enough to recognise.

The right-hand column names standards and methods rather than products, deliberately. Vendor capability in this area changes every quarter, and a paper that names tools dates within one. A paper that names the standard your tool has to implement does not.

Where it sits in an AI-native SDLC

The phases are not the familiar ones with “AI” written on them. Two are genuinely new.

Design → Build → Evaluate → Shadow → Ramp → Operate → Assure.

Evaluate is not a test phase at the end; it runs continuously against golden and adversarial sets, and it gates the build — the argument of Evaluation Is Not a Test Phase. Shadow has no equivalent in traditional delivery: a stage where the new version runs on real traffic and is permitted to affect nothing. Ramp replaces the traffic percentage with an authority rung. Assure is periodic re-testing of controls, because a model change can invalidate a result without any code changing.

The two phases teams skip are Shadow and Assure, and they are skipped for the same reason: neither produces a visible feature. Shadow is what makes an irreversible decision testable at all, and Assure is what stops a result quietly expiring.

Key takeaways

• An A/B test can show that an agent got cheaper, faster or less annoying. It cannot show that it got safer.
• At a 400-person division’s traffic, a quarter of experimentation can only detect the halving of a rate more common than about 1 in 723. Every failure mode worth fearing is rarer.
• Derive the design from the journey: decision points, reversibility, the metric each one owns, then the unit of randomisation. The unit is usually the run, not the user.
• Most enterprise AI experiments are non-inferiority tests. Name the guardrails that must not move and the stopping rule, before anyone looks.
• For what you cannot measure: shadow-run it, replay it, ramp authority rather than traffic, and bound the tail with a control that refuses rather than a metric that watches.
• Keep every consequential trace and sample the rest hard. Uniform sampling discards precisely the population that matters.
• The two SDLC phases teams skip — Shadow and Assure — are the two that make an irreversible decision testable and stop a result silently expiring. Neither ships a feature, which is why they go.

Sources and further reading

• The Contextual Agentic Enterprise, Chapters 8 and 17.
• Kohavi, R., Tang, D. and Xu, Y. (2020) Trustworthy Online Controlled Experiments. Cambridge University Press.
• Johari, R. et al. (2017) “Peeking at A/B Tests”, KDD 2017. https://doi.org/10.1145/3097983.3097992
• Deng, A. et al. (2013) “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data” (CUPED), WSDM 2013. https://doi.org/10.1145/2433396.2433413
• OpenTelemetry (2026) Semantic conventions for generative AI systems. https://opentelemetry.io/docs/specs/semconv/gen-ai/

This article is part of CAE Field Notes, a series accompanying the book, The Contextual Agentic Enterprise.