Evaluation Is Not a Test Phase

Most agentic programmes are built in this order: pick a model, write a prompt, bolt on retrieval, demonstrate it, and — somewhere near the end, usually when a risk committee asks — write some evaluations.

By then the evaluations cannot do the one thing that would have made them valuable. They cannot change the design. The assumptions are already in code, the integration is already paid for, and the evaluation set is quietly written to match what the system already does. A suite built that way will pass. It just will not tell you anything.

The order is the architecture decision. Everything else follows from it.

First: say what success means in measurable terms

Before a pattern, before a model, before retrieval, one sentence has to exist and survive contact with the person who owns the outcome: what makes an answer acceptable?

Not “high quality”. Not “accurate”. An acceptance predicate — a rule that a machine could apply to a single case and return true or false. “The credit note cites the correct purchase order, the variance is explained against the contracted rate, and every figure appears in a retrieved source document.” That is testable. “Good customer outcomes” is not.

Writing it down does two things at once. It gives you the evaluation criterion you will need in week nine, and it forces the disagreement that would otherwise surface in month five. In practice, the first draft of an acceptance predicate is where you discover that two stakeholders have been describing different systems.

If the predicate cannot be written, the work is not ready to be built. That is a finding, not a delay.

Second: expose the design assumptions while changing them is still cheap

Every agentic design rests on assumptions that feel obvious and are frequently wrong: that the evidence needed to decide is actually retrievable; that the input arrives in something like the expected shape; that the task has a single correct answer; that the user knows what they are asking for.

These assumptions get cheaper to test the earlier you test them, and the cost curve is steep. Checking whether the required evidence exists in your corpus is an afternoon with a hundred sampled cases. Discovering it does not exist after the orchestration is built is a re-architecture.

The discipline is to state each assumption as a claim with a measurement attached, and run the measurement before it is expensive to be wrong:

• Assumption: the evidence is retrievable. Measurement: recall@k on a hundred labelled real cases. Run this before choosing a pattern at all — no generation strategy recovers evidence that retrieval never returned.
• Assumption: inputs are well-formed. Measurement: sample a week of real traffic and count what fraction fails the input contract. Real enterprise input is malformed, truncated, ambiguous, in the wrong language, or asks something the system cannot answer.
• Assumption: the question is answerable. Measurement: the fraction of your sampled cases that have no defensible answer. This number is the single best predictor of whether you need an abstention path, and it is almost never zero.

That last one deserves emphasis. A system evaluated only on answerable questions learns nothing about the behaviour that will actually hurt you, which is confident output on a question that had no answer. The unanswerable slice belongs in the evaluation set from the first day.

Third: build gates, not a score

An architect changing a live agentic system needs to answer one question repeatedly: did this change make the system better? A single aggregate score cannot answer it. Aggregates hide the trade — grounding improved, abstention collapsed; latency fell, the high-consequence slice regressed.

What works is a ladder of gates, each with its own rule, run on every change:

The gates are ordered deliberately. The cheap deterministic ones run first and fail fast. The expensive statistical ones only run on changes that survived.

Two kinds of grader

The gate table above says “deterministic” and “statistical”. That is one axis. There is a second one underneath it that teams routinely collapse into the first, and the collapse is expensive.

What kind of property is being checked — an invariant that must hold on every single run, or a behaviour that holds across a distribution of cases. What kind of grader decides it — code, a program that computes pass or fail, or a model, asked to judge.

These are not the same axis, and the four cells are not equally populated.

The bottom-left cell is the one that gets skipped. Teams reach for an LLM judge the moment a property is statistical, when a large fraction of statistical properties are still perfectly code-checkable. “Does every figure in the answer appear in a retrieved document?” sounds like a judgement call. It is a string-matching problem.

The honest reason teams skip it is that code graders usually need a reference answer, and building references is work. A model judge is reference-free, so it feels cheaper. It is cheaper on the first day and more expensive on every day after.

Three rules follow.

Never use a model where code will do. A model grader is slower, costs money on every run, drifts when the provider updates the model, and returns different answers to the same question. On a property that a program can decide, every one of those is a pure loss.

A judge is a component, so evaluate it. An LLM judge is a classifier you have not tested. Measure it against a human-labelled sample, report its agreement rate, and look at where it disagrees. An unvalidated judge is not a measurement, it is a number generator.

Pin the judge and version it. Model, version, temperature, and the rubric prompt all belong in version control. A judge change is a release of its own — if the judge and the system change together, you have learnt nothing, which is the frozen-set rule applied one level up. And do not judge with the same model configuration that generated the output: correlated failure means the judge is most confident exactly where the generator is most wrong.

Two well-documented biases are worth designing around: judges prefer longer answers, and in pairwise comparison they prefer whichever candidate is presented first. Randomise the order and control for length, or you will measure verbosity.

Model grading is also the expensive gate, which is why it should not run on everything. Run the code graders on every commit and reserve the judge for the slice that needs it.

What to measure

The last row is the only one a sponsor pays for. The rows above it exist to explain the last row when it moves, and to warn you before it does.

The measurement discipline that makes the numbers mean something

Four rules separate a gate from theatre.

Freeze the set. If the evaluation set and the system change in the same release, you have learnt nothing. Version the set like code. Grow it by adding production failures, in their own release.

One change per run. A model swap, a prompt change and a retrieval change are three experiments. Bundled, they produce one number and no knowledge of which lever moved it.

Pair the comparison and state the interval. Run the incumbent and the candidate on the same cases and compare per case. A three-point move on a hundred cases is noise. Report the difference with a confidence interval and be honest about the resolution your set actually buys: a set of one hundred cannot detect a five-point improvement.

Slice before you average. Report the high-consequence slice, the unanswerable slice and the long-tail slice separately. An average that rose while the high-consequence slice fell is a regression that shipped.

In practice

Illustrative scenario. A shared-services team wants an agent to resolve supplier invoice exceptions. The acceptance predicate is written first: the proposed resolution cites the purchase order and the contracted rate, every figure appears in a retrieved document, and anything above the delegation threshold is escalated rather than posted.

Two hundred closed exceptions from last quarter become the golden set — including the thirty-one that were closed by a human phone call and have no documentary answer at all. Recall@10 on the evidence is measured before anything else is built: it comes back at 0.71, which kills the original design, because no prompt will recover the twenty-nine per cent of cases where the contract was never in the system. The corpus problem is fixed first.

The agent then ships with authority only to propose. The shadow gate runs for three weeks. Cost per resolved exception and first-pass yield are reported to the sponsor; grounding, abstention correctness and the escalation rate are reported to the architect. When a cheaper model is proposed four months later, the answer takes two days rather than an argument, because the gates already exist.

None of this is a testing activity. It is the order in which the architecture was decided.

Where the standards fit

Gates are the mechanism; published frameworks supply the checklist of what to gate against. The OWASP Top 10 for LLM Applications (2025) gives the adversarial suite its coverage targets — prompt injection, sensitive information disclosure, improper output handling, excessive agency. The NIST AI Risk Management Framework, voluntary, and its Generative AI Profile (NIST AI 600-1, July 2024) supply the governance structure the evidence feeds: Govern, Map, Measure, Manage. Neither tells you what your acceptance predicate is. That part is yours.

Share on LinkedIn (opens in a new tab)