Contextual Agentic

The AI Engineer Playbook: From Model Serving to Authorised Outcomes

Eight engineering decisions take an agent's request from model serving to a completed, authorised business outcome: define the action and its budget, serve the model, construct versioned context, evaluate three things separately, propose typed actions under delegated authority, check before commit, trace the whole workflow, and measure per completed authorised outcome. Faster tokens shorten only the first clock.

·16 min read

Written as at 10 November 2026.

ai-engineeringllm-servinginferenceevaluationobservabilitycommit-boundaryagentic-aiplatform-engineering

Faster tokens are not a faster business

Illustrative scenario: a fictional composite written to make the point concrete. It does not describe a real organisation, and its figures are not reported outcomes.

Tallow Bay Health runs eleven hospitals and a rostering problem. Around three hundred shifts a day fall vacant at short notice, most of them surfacing in the 06:00 handover. An agent now reads each vacancy, the ward roster, the award rules and the fatigue records, and proposes a staff member to fill the shift. A rostering manager has delegated it the authority to assign within a ward, with overtime capped at two hours a shift; anything above one hour goes to the unit manager. The roster system commits the change and the shift is either filled or it is not.

The platform team’s first dashboard showed tokens per second. The engine was fast. The shifts were still being filled late, because the model’s two seconds sat inside a process that waited eleven minutes for an approval, re-planned when a ward roster changed underneath it, and occasionally committed nothing at all because an evidence service was down. The playbook below is the route that team would take from the serving engine to the outcome the hospital pays for. It uses the Contextual Agentic Enterprise (CAE), Contextual Agentic’s architecture and operating model, as its frame: a probabilistic core that proposes and a Deterministic Shell that decides identity, authority, schema and commitment.

Route diagram in eight numbered stages from request to completed authorised outcome: define the action, workload, SLO and failure budget; select and serve the model with prefill, decode and batching trade-offs; construct versioned, permitted, fresh context with a cache boundary; evaluate answer, tool choice and action separately; propose a typed request under delegated authority; check identity, policy, evidence and approval at the commit boundary; trace the whole workflow; measure per completed authorised outcome. A dashed red return path from check to propose is labelled hold, deny or re-plan, a new proposal, never a silent retry. Two clocks run beneath: model latency covering stages two to five, and business-outcome latency covering all eight including approval wait and recovery. Caption: faster tokens shorten the top bar; only the bottom bar is the process the business pays for.
Figure 1. The route, and the two clocks. A proposed engineering route for a fictional workflow, not a benchmark.

Two clocks run through everything that follows. Model latency starts when the prompt is sent and ends when the proposal is parsed: time to first token, decode, tool calls. Business-outcome latency starts at the request and ends when the roster system has committed an authorised change and the record is sealed: queue, approval wait, holds and recovery included. Tuning the engine moves the first clock. Only the second is the process.

1. Define the action, the workload, the SLO and the failure budget

The decision. Before any model is chosen, write down the business action in one sentence, the workload shape, the service-level objective in business terms, and the failure budget the owners will accept.

Three panels: the business action, filling a vacant nursing shift with a permitted roster change; the workload and SLO, about three hundred vacancies a day with bursts at six in the morning, a proposal within ninety seconds and a filled shift within thirty minutes at the ninety-fifth percentile; and the failure budget, at most two per cent of proposals rejected at commit, at most one per cent held per day, and zero commits without a record. Caption: the denominator for every metric is one completed authorised roster change.
Figure 2. The denominator is the completed, authorised action.

The example. Action: fill a vacant nursing shift with a permitted roster change. Workload: about 300 vacancies a day, bursting at 06:00, prompts of 6,000 to 9,000 tokens (roster, award extract, fatigue records), answers under 300 tokens. SLO: a proposal within 90 seconds and a filled shift within 30 minutes, both at p95. Failure budget: at most 2 per cent of proposals rejected at commit, at most 1 per cent of requests held per day, and zero commits without an evidence record.

What to measure. Requests per hour by ward, prompt and output token distributions, and the two clocks, each with its denominator written beside it.

Failure and recovery. The usual failure is an SLO stated in tokens per second. Recovery is editorial: restate it as a time to a completed, authorised action and a rate of rework.

Why it matters. Every later decision (batching, caching, speculative decoding, approval routing) is a trade-off, and a trade-off needs a target. The target is the business clock.

2. Serve the model: prefill and decode are different workloads

The decision. Choose a model and an engine configuration for the actual shape of the work. A prefill-heavy, short-answer workload wants a different setup from a chat workload.

Two bars: prefill processes the whole prompt in parallel and is compute-heavy, ending at time to first token; decode produces one token per step and every step reads the weights, so it is memory-bound and measured as inter-token latency. Batching more requests raises decode throughput until memory bandwidth or KV-cache capacity binds. A decision panel for the roster agent: long prompt and short answer means prefill dominates, so cache the shared prefix; latency-sensitive and low concurrency means speculative decoding is worth testing; serve FP8 on Hopper-class hardware and evaluate the quantised model on this task first.
Figure 3. Prefill saturates the GPU; decode waits on memory.

The mechanism, with its qualifications. Prefill processes the whole prompt at once as a matrix-matrix operation that NVIDIA’s inference guide describes as effectively saturating the GPU; decode produces one token per step as a matrix-vector operation whose latency is dominated by moving weights and the KV cache from memory, which is why it is described as memory-bound. The KV cache stores each layer’s keys and values for every token so they are not recomputed; its size grows with batch, sequence length, layers and hidden size (the same guide’s worked example gives about 2 GB for a 7B model at batch 1 and 4,096 tokens in 16-bit). Grouped-query attention shrinks it by sharing key and value heads, at a quality risk the GQA paper mitigates by uptraining. PagedAttention (vLLM) manages the cache in blocks to cut fragmentation and allow sharing; the paper reports 2 to 4 times the throughput of its baselines at equal latency, a figure specific to its setup. Continuous (iteration-level) batching from Orca lets finished requests leave and new ones join at each step, with a 36.9 times throughput figure against a baseline that had no such scheduling. Batching raises decode throughput until memory bandwidth or KV capacity binds; vLLM then preempts and recomputes, which appears as tail latency. Chunked prefill schedules pending decodes first and fills the remaining token budget with prefill chunks, trading time to first token against inter-token latency with no single best setting. Disaggregated prefill and decode (DistServe; experimental in vLLM) tune the two phases separately; vLLM’s documentation states plainly that it does not improve throughput.

Quantisation reduces memory and bandwidth: weight-only methods (GPTQ, AWQ) and weight-and-activation FP8 (SmoothQuant lineage) report small quality loss on their own benchmarks, and the engines document hardware limits, for example FP8 W8A8 on Ada, Hopper and Blackwell. The quality claim is always the paper’s; the only quality claim that counts for Tallow Bay is the one its own evaluation set produces. Speculative decoding drafts several tokens cheaply and verifies them with the target model; the original papers show the output distribution is preserved, and the engines qualify that as “up to the precision limits of hardware numerics”, with gains that shrink at high request rates and depend on acceptance rate. Tensor parallelism splits layers across GPUs and needs fast interconnect; pipeline parallelism is the fallback across nodes or on hosts without NVLink.

The example. Prompts are long and answers short, so prefill dominates and the shared award text is a candidate for prefix caching. Concurrency is low and latency matters, so speculative decoding is worth a trial. The team serves an FP8 build on Hopper-class GPUs and runs the section 4 evaluation set against it before switching.

What to measure. Time to first token, inter-token latency and time per output token, queue time, KV-cache usage and preemptions, all exported by the engine (vLLM names them vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_queue_time_seconds, vllm:kv_cache_usage_perc, vllm:num_preemptions; check the live /metrics endpoint, because names change between versions).

Failure and recovery. A 06:00 burst fills the KV cache; preemptions rise; p95 first-token time passes 20 seconds. Recovery: cap concurrent sequences, cut the token budget per step, or add a replica, then watch the business clock, not the token rate.

Why it matters. Without the mechanism, every tuning knob is a superstition. With it, the team knows which knob moves which clock.

3. Construct and version the context

The decision. Define, for this agent purpose, exactly which sources it may receive, at which versions, how fresh, with which permissions, and where the cache boundary falls. In CAE this is a Context Contract.

Three sources feed the prompt: award rules at version 2026.3 with a digest, shared across wards; the ward roster at version 8,114, fresh within sixty seconds and ward-scoped; staff records with ward permission, fatigue hours at a version and personal data minimised. A dashed red cache boundary separates what may share a cached prefix across wards, the award text, from what never may, roster and staff context. Below, the Context Contract for this purpose records which sources, versions, fields, freshness and viewers, and is recorded with the proposal.
Figure 4. Provenance travels with the passage; the cache boundary follows permissions.

The example. Award rules v2026.3 are identical for every ward and are the shared prefix. The ward roster is scoped to the ward and must be fresher than 60 seconds. Staff records are the most sensitive input: only the fields the decision needs, only for this ward, and only the fatigue hours at a stated version. Each passage carries source, version, digest and retrieved-at, and the proposal records them.

The cache boundary. Prefix caching reuses the computed KV blocks of a shared prefix; vLLM’s design notes say it caches full blocks only and does not change model outputs, and that it helps on long shared prompts and multi-turn work and not when prefixes differ or when generation dominates. It also opens a timing side channel: a co-tenant can infer whether a prefix is cached from the time to first token (CVE-2025-46570 in vLLM’s security notes; the general attack is in Zheng et al. 2024). vLLM mixes a per-request cache_salt into the first block’s hash and states that it does not provide tenant isolation; the recommended answer for separate tenants is separate instances. For Tallow Bay the award text may share a cache across wards; roster and staff context never may, and wards are not tenants, so one instance with salted roster blocks is the design, reviewed with security.

What to measure. Prefix-cache hit rate per source class, context freshness at proposal time, permission denials at retrieval, and the share of proposals whose recorded versions resolve at reconstruction time.

Failure and recovery. The roster service lags; the agent proposes against v8,114 while the ward is at v8,116. Recovery is not a retry of the old proposal: the gate in section 6 holds on version drift, and the agent re-plans against the current version.

Why it matters. A permission check on the index is not optional because the model is good. The context is where enterprise data meets a probabilistic component, and the contract is what makes that meeting governable.

4. Evaluate answer quality, tool choice and action correctness separately

The decision. Keep three evaluations with three denominators, and never average them into one score.

Three columns: answer quality, whether the ranked candidate list is sensible for the vacancy, measured with a graded set and rubric; tool choice, whether the agent called the fatigue lookup before proposing overtime, derived from traces per call; action correctness, whether the typed request is valid within award and envelope, checked by a deterministic validator. Caption: three scores, three denominators, never averaged. A gate before serving a new model or prompt requires the validator to pass fully, tool-choice precision not to fall, and the grader to stay within its noise band.
Figure 5. A fluent answer can carry an invalid action.

The example. Answer quality: is the ranked candidate list sensible for this vacancy? A graded set of 400 historical vacancies with a rubric, scored by reviewers and by a grader model whose agreement with reviewers is itself measured. Tool choice: did the agent call the fatigue lookup before proposing overtime? Read from the trace, per call. Action correctness: is the typed request valid within award and envelope? A deterministic validator, which must pass on every case before any model or prompt change ships.

What to measure. Grader agreement with reviewers, tool-choice precision and recall per tool, validator pass rate, and the drift of each across releases.

Failure and recovery. A new model improves the grader score and starts proposing a rest-break pattern the award forbids. The validator catches it; the release is blocked; the grader rubric gains a case. The lesson is recorded as a correction, not a quiet fix.

Why it matters. The quality of the words and the correctness of the action are different properties. An agent that writes beautifully about an invalid change is more dangerous than one that writes badly.

5. Propose a typed request under delegated authority

The decision. The agent’s output that matters is not prose; it is a typed request whose schema refuses anything the Authority Envelope does not cover.

Left panel, the typed proposal: action assign shift, shift identifier, staff identifier, eight hours with two of overtime, basis fatigue record version 2210 and roster version 8,114, and a payload digest. Right panel, the Authority Envelope: a grant from the rostering manager to the roster agent permitting assignments within ward seven, overtime at most two hours per shift, expiring at the end of November. Caption: free text is for the person, the request is for the machine. Lower panel: the agent holds the right to propose, a named role the right to approve, the roster system the right to commit.
Figure 6. The schema is the first control the model cannot talk its way past.

The example. assign_shift for shift W7-2026-11-12-N, staff S-4471, eight hours including one of overtime, with the basis (fatigue v2210, roster v8,114) and a payload digest. The envelope: grant rostering-manager → roster-agent@2.1, may assign within ward 7, overtime at most two hours a shift, expires 30 November. An unknown ward, three hours of overtime, or a staff id with no fatigue record is refused by the schema before any policy runs.

What to measure. Schema rejections by reason, proposals per completed action (rework), and the share of proposals carrying a complete basis.

Failure and recovery. The model emits a persuasive note explaining why a three-hour overtime shift is justified this once. The schema does not read notes. The proposal is rejected and the explanation is kept as an inferred claim on the record.

Why it matters. Delegated authority is explicit: the agent may propose, a named role may approve, the owning system may commit. Inherited credentials blur the three, which is how an agent ends up able to do more than it was ever asked to.

6. Check identity, policy, evidence and approval before commit

The decision. Place the checks at the Commit Boundary, in or immediately before the system that makes the change, and make every outcome a recorded event.

A vertical chain of four checks at the commit boundary: identity, the roster agent attested; policy, award version 2026.3 and the fatigue rule; evidence, whether fatigue record version 2210 is current; approval, overtime above one hour routes to the unit manager. A side panel lists the outcomes: permit then commit, escalate then wait, deny then explain, hold then nothing when the evidence service is down, a version is stale or a grant has expired; every outcome is recorded. Lower panel: the roster system writes the assignment and returns its own reference; the record binds proposal digest, approval, versions and committed result; an unknown result is written as unknown and read back before any retry.
Figure 7. Permit, escalate, deny or hold, and the commit returns its own reference.

The example. Identity: the agent’s workload identity is attested. Policy: award v2026.3 and the fatigue rule pass. Evidence: fatigue record v2210 is the current version. Approval: one hour of overtime needs none; two hours routes to the unit manager, whose approval is bound to the payload digest and expires in four hours. On permit, the roster system writes the assignment and returns RST-77102; the record binds proposal digest, versions, approval and committed result. If the evidence service does not answer, nothing is written and the request is held: an unavailable record is a hold, never a permission.

What to measure. Outcomes by type (permit, escalate, deny, hold) and reason, approval wait, holds per day against the failure budget, and the count of commits without a record, which must be zero.

Failure and recovery. The roster system times out after the write was sent. The record says unknown; recovery reads the roster system before any retry, and a retry under the same idempotency key is the only one permitted. The evidence Field Note covers how that record is sealed and verified later.

Why it matters. This is the fifth rung of the Agentic AI Playbook seen from the engineer’s chair: deterministic code, outside the model’s argument, deciding whether a proposal becomes a change.

7. Trace the whole workflow, not the model call

The decision. One workflow identifier from the request to the recorded result, with spans for context assembly, each model phase, each tool call, validation, approval wait, commit and record, carrying identifiers and versions and never roster content.

A waterfall of spans under one workflow identifier: request, context assembly, model prefill, model decode, a fatigue-lookup tool call, proposal validation, a long approval wait, commit and record. Beneath it, model latency of about two seconds sits beside an approval wait of about eleven minutes. Captions: the trace boundary starts at the request and ends at the recorded result; spans carry identifiers and versions, never roster content; a trace shows order and duration and does not prove the approval caused the outcome.
Figure 8. Two seconds of model inside eleven minutes of process.

The example. The escalated path’s trace: 0.3 s context assembly, 0.4 s to first token from the cached prefix, 1.2 s decode, a 0.3 s fatigue lookup, validation, then 11 minutes of approval wait, a commit and a record. The OpenTelemetry GenAI conventions give names for the model spans and server metrics (gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token, gen_ai.server.request.duration); they are at “Development” status, so pin the version you adopt and expect renames. Message content attributes are opt-in in those conventions and stay off here.

What to measure. Span coverage (the share of requests whose trace reaches the recorded result), the two clocks derived from the same trace, and the gap between them by reason.

Failure and recovery. A trace that ends at the model response makes the model look like the whole process. Recovery is a trace boundary decision, not an instrumentation one: the trace ends at the record.

Why it matters. A trace shows order and duration. It does not show that the approval caused the outcome, that the omitted span never happened, or that the model’s explanation is what it computed. Use it for what it proves, and read the Evidence-First Decision Record for the rest.

8. Measure time, reliability and cost per completed, authorised outcome

The decision. Publish seven metrics together, each with its denominator, and treat a win on the first two alone as not a result.

A seven-row table of metrics with denominators and views: time to first token per model call at p50 and p95; decode throughput in tokens per second per GPU, mean under load; end-to-end workflow latency from request to recorded result at p50 and p95; completion rate, filled divided by requested shifts, daily; rework, proposals rejected at commit divided by proposals, weekly; approval wait from escalation to decision at p50 and p95; cost per completed authorised action, GPU hours plus tools plus people divided by filled shifts, monthly. Caption: the first two move when you tune the engine; only the last five tell you whether the process got better.
Figure 9. Seven numbers, seven denominators, one table.
Metric Denominator What moves it
Time to first token one model call prompt length, prefix cache, chunked prefill, batch pressure
Decode throughput tokens per second per GPU batch occupancy, KV capacity, quantisation, speculation
End-to-end workflow latency (p50, p95) one request to its recorded result everything above plus queue, approval wait, holds, recovery
Completion rate filled shifts ÷ requested proposal validity, approval availability, roster drift
Rework proposals rejected at commit ÷ proposals context freshness, schema fit, policy alignment
Approval wait (p50, p95) escalations routing, approver load, expiry settings
Cost per completed, authorised action (GPU-hours + tools + people) ÷ filled shifts all of the above

The example. Tallow Bay’s engine upgrade doubled decode throughput and changed completion rate by nothing, because the shifts were waiting on approvals. Routing one-hour overtime to automatic permit under the envelope cut approval wait by half and moved the completion rate, which the token dashboard could never have shown.

On cost. Tokens per dollar depends on batch occupancy: a GPU billed by the hour (provisioned capacity is billed that way by at least one major provider) produces tokens in proportion to how full its batches are, so a half-empty engine is an expensive one however fast each token comes out. Per-token pricing moves the risk to the provider and the denominator problem stays: the cost that matters is per completed, authorised action, and it includes the approver’s eleven minutes.

Failure and recovery. A monthly report quotes p50 workflow latency and omits p95 and rework. Recovery: the table above is the report, with all seven rows or none.

Why it matters. Denominators decide what a number means. “Latency fell 40 per cent” is a statement about tokens unless the denominator says otherwise.

Follow one request through the route

Interactive companion. Choose a path (within envelope, needs approval, hold and re-plan), then play or step through eight stages while two clocks run: model latency and business-outcome latency. A text equivalent is provided below the controls. Values are illustrative.

The companion has three paths. On the first, one hour of overtime stays within the envelope and the business clock stops at a few seconds. On the second, two hours of overtime escalates: the model clock stops at 1.9 seconds and the business clock runs for eleven minutes until the unit manager decides. On the third, the evidence service is unavailable at commit time: nothing is written, the request holds, and when the service returns the ward roster has moved on, so a fresh proposal is made rather than the stale one retried. Play, pause, step and restart are buttons; the arrow keys step; the sequence plays without motion when your device asks for reduced motion.

What this route does not establish

The route makes the engine’s trade-offs legible and puts deterministic checks where the change happens. It does not make a model’s ranking right, it does not prove that an approval caused a good outcome, and it does not turn a trace into a reconstruction. The serving figures quoted from papers are the authors’ own on their setups; Tallow Bay’s numbers are invented to show the shape of the measurement, not its value. Engine behaviour, metric names and standards status change between versions; the sources below are as at 10 November 2026.

Sources and further reading

The book develops the patterns and the evidence at length

The book treats the enterprise agent design patterns, the governance and evidence constructs, and observability and replay in the chapters listed above, including the full Context Contract and evidence bundle that this playbook names only in grouped form. Any adoption of CAE should be tailored to the organisation in consultation with Contextual Agentic or another advisory service. More about the book.

This article is part of the AI Engineer Playbook, a CAE series accompanying the book, The Contextual Agentic Enterprise.

← All articles

Related