The AI Engineer Playbook: From Model Serving to Authorised Outcomes
Eight engineering decisions take an agent's request from model serving to a completed, authorised business outcome: define the action and its budget, serve the model, construct versioned context, evaluate three things separately, propose typed actions under delegated authority, check before commit, trace the whole workflow, and measure per completed authorised outcome. Faster tokens shorten only the first clock.
Written as at 10 November 2026.
ai-engineeringllm-servinginferenceevaluationobservabilitycommit-boundaryagentic-aiplatform-engineering
Faster tokens are not a faster business
Illustrative scenario: a fictional composite written to make the point concrete. It does not describe a real organisation, and its figures are not reported outcomes.
Tallow Bay Health runs eleven hospitals and a rostering problem. Around three hundred shifts a day fall vacant at short notice, most of them surfacing in the 06:00 handover. An agent now reads each vacancy, the ward roster, the award rules and the fatigue records, and proposes a staff member to fill the shift. A rostering manager has delegated it the authority to assign within a ward, with overtime capped at two hours a shift; anything above one hour goes to the unit manager. The roster system commits the change and the shift is either filled or it is not.
The platform team’s first dashboard showed tokens per second. The engine was fast. The shifts were still being filled late, because the model’s two seconds sat inside a process that waited eleven minutes for an approval, re-planned when a ward roster changed underneath it, and occasionally committed nothing at all because an evidence service was down. The playbook below is the route that team would take from the serving engine to the outcome the hospital pays for. It uses the Contextual Agentic Enterprise (CAE), Contextual Agentic’s architecture and operating model, as its frame: a probabilistic core that proposes and a Deterministic Shell that decides identity, authority, schema and commitment.
Two clocks run through everything that follows. Model latency starts when the prompt is sent and ends when the proposal is parsed: time to first token, decode, tool calls. Business-outcome latency starts at the request and ends when the roster system has committed an authorised change and the record is sealed: queue, approval wait, holds and recovery included. Tuning the engine moves the first clock. Only the second is the process.
1. Define the action, the workload, the SLO and the failure budget
The decision. Before any model is chosen, write down the business action in one sentence, the workload shape, the service-level objective in business terms, and the failure budget the owners will accept.
The example. Action: fill a vacant nursing shift with a permitted roster change. Workload: about 300 vacancies a day, bursting at 06:00, prompts of 6,000 to 9,000 tokens (roster, award extract, fatigue records), answers under 300 tokens. SLO: a proposal within 90 seconds and a filled shift within 30 minutes, both at p95. Failure budget: at most 2 per cent of proposals rejected at commit, at most 1 per cent of requests held per day, and zero commits without an evidence record.
What to measure. Requests per hour by ward, prompt and output token distributions, and the two clocks, each with its denominator written beside it.
Failure and recovery. The usual failure is an SLO stated in tokens per second. Recovery is editorial: restate it as a time to a completed, authorised action and a rate of rework.
Why it matters. Every later decision (batching, caching, speculative decoding, approval routing) is a trade-off, and a trade-off needs a target. The target is the business clock.
2. Serve the model: prefill and decode are different workloads
The decision. Choose a model and an engine configuration for the actual shape of the work. A prefill-heavy, short-answer workload wants a different setup from a chat workload.
The mechanism, with its qualifications. Prefill processes the whole prompt at once as a matrix-matrix operation that NVIDIA’s inference guide describes as effectively saturating the GPU; decode produces one token per step as a matrix-vector operation whose latency is dominated by moving weights and the KV cache from memory, which is why it is described as memory-bound. The KV cache stores each layer’s keys and values for every token so they are not recomputed; its size grows with batch, sequence length, layers and hidden size (the same guide’s worked example gives about 2 GB for a 7B model at batch 1 and 4,096 tokens in 16-bit). Grouped-query attention shrinks it by sharing key and value heads, at a quality risk the GQA paper mitigates by uptraining. PagedAttention (vLLM) manages the cache in blocks to cut fragmentation and allow sharing; the paper reports 2 to 4 times the throughput of its baselines at equal latency, a figure specific to its setup. Continuous (iteration-level) batching from Orca lets finished requests leave and new ones join at each step, with a 36.9 times throughput figure against a baseline that had no such scheduling. Batching raises decode throughput until memory bandwidth or KV capacity binds; vLLM then preempts and recomputes, which appears as tail latency. Chunked prefill schedules pending decodes first and fills the remaining token budget with prefill chunks, trading time to first token against inter-token latency with no single best setting. Disaggregated prefill and decode (DistServe; experimental in vLLM) tune the two phases separately; vLLM’s documentation states plainly that it does not improve throughput.
Quantisation reduces memory and bandwidth: weight-only methods (GPTQ, AWQ) and weight-and-activation FP8 (SmoothQuant lineage) report small quality loss on their own benchmarks, and the engines document hardware limits, for example FP8 W8A8 on Ada, Hopper and Blackwell. The quality claim is always the paper’s; the only quality claim that counts for Tallow Bay is the one its own evaluation set produces. Speculative decoding drafts several tokens cheaply and verifies them with the target model; the original papers show the output distribution is preserved, and the engines qualify that as “up to the precision limits of hardware numerics”, with gains that shrink at high request rates and depend on acceptance rate. Tensor parallelism splits layers across GPUs and needs fast interconnect; pipeline parallelism is the fallback across nodes or on hosts without NVLink.
The example. Prompts are long and answers short, so prefill dominates and the shared award text is a candidate for prefix caching. Concurrency is low and latency matters, so speculative decoding is worth a trial. The team serves an FP8 build on Hopper-class GPUs and runs the section 4 evaluation set against it before switching.
What to measure. Time to first token, inter-token latency and time per output token, queue time, KV-cache usage and preemptions, all exported by the engine (vLLM names them vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_queue_time_seconds, vllm:kv_cache_usage_perc, vllm:num_preemptions; check the live /metrics endpoint, because names change between versions).
Failure and recovery. A 06:00 burst fills the KV cache; preemptions rise; p95 first-token time passes 20 seconds. Recovery: cap concurrent sequences, cut the token budget per step, or add a replica, then watch the business clock, not the token rate.
Why it matters. Without the mechanism, every tuning knob is a superstition. With it, the team knows which knob moves which clock.
3. Construct and version the context
The decision. Define, for this agent purpose, exactly which sources it may receive, at which versions, how fresh, with which permissions, and where the cache boundary falls. In CAE this is a Context Contract.
The example. Award rules v2026.3 are identical for every ward and are the shared prefix. The ward roster is scoped to the ward and must be fresher than 60 seconds. Staff records are the most sensitive input: only the fields the decision needs, only for this ward, and only the fatigue hours at a stated version. Each passage carries source, version, digest and retrieved-at, and the proposal records them.
The cache boundary. Prefix caching reuses the computed KV blocks of a shared prefix; vLLM’s design notes say it caches full blocks only and does not change model outputs, and that it helps on long shared prompts and multi-turn work and not when prefixes differ or when generation dominates. It also opens a timing side channel: a co-tenant can infer whether a prefix is cached from the time to first token (CVE-2025-46570 in vLLM’s security notes; the general attack is in Zheng et al. 2024). vLLM mixes a per-request cache_salt into the first block’s hash and states that it does not provide tenant isolation; the recommended answer for separate tenants is separate instances. For Tallow Bay the award text may share a cache across wards; roster and staff context never may, and wards are not tenants, so one instance with salted roster blocks is the design, reviewed with security.
What to measure. Prefix-cache hit rate per source class, context freshness at proposal time, permission denials at retrieval, and the share of proposals whose recorded versions resolve at reconstruction time.
Failure and recovery. The roster service lags; the agent proposes against v8,114 while the ward is at v8,116. Recovery is not a retry of the old proposal: the gate in section 6 holds on version drift, and the agent re-plans against the current version.
Why it matters. A permission check on the index is not optional because the model is good. The context is where enterprise data meets a probabilistic component, and the contract is what makes that meeting governable.
4. Evaluate answer quality, tool choice and action correctness separately
The decision. Keep three evaluations with three denominators, and never average them into one score.
The example. Answer quality: is the ranked candidate list sensible for this vacancy? A graded set of 400 historical vacancies with a rubric, scored by reviewers and by a grader model whose agreement with reviewers is itself measured. Tool choice: did the agent call the fatigue lookup before proposing overtime? Read from the trace, per call. Action correctness: is the typed request valid within award and envelope? A deterministic validator, which must pass on every case before any model or prompt change ships.
What to measure. Grader agreement with reviewers, tool-choice precision and recall per tool, validator pass rate, and the drift of each across releases.
Failure and recovery. A new model improves the grader score and starts proposing a rest-break pattern the award forbids. The validator catches it; the release is blocked; the grader rubric gains a case. The lesson is recorded as a correction, not a quiet fix.
Why it matters. The quality of the words and the correctness of the action are different properties. An agent that writes beautifully about an invalid change is more dangerous than one that writes badly.
5. Propose a typed request under delegated authority
The decision. The agent’s output that matters is not prose; it is a typed request whose schema refuses anything the Authority Envelope does not cover.
The example. assign_shift for shift W7-2026-11-12-N, staff S-4471, eight hours including one of overtime, with the basis (fatigue v2210, roster v8,114) and a payload digest. The envelope: grant rostering-manager → roster-agent@2.1, may assign within ward 7, overtime at most two hours a shift, expires 30 November. An unknown ward, three hours of overtime, or a staff id with no fatigue record is refused by the schema before any policy runs.
What to measure. Schema rejections by reason, proposals per completed action (rework), and the share of proposals carrying a complete basis.
Failure and recovery. The model emits a persuasive note explaining why a three-hour overtime shift is justified this once. The schema does not read notes. The proposal is rejected and the explanation is kept as an inferred claim on the record.
Why it matters. Delegated authority is explicit: the agent may propose, a named role may approve, the owning system may commit. Inherited credentials blur the three, which is how an agent ends up able to do more than it was ever asked to.
6. Check identity, policy, evidence and approval before commit
The decision. Place the checks at the Commit Boundary, in or immediately before the system that makes the change, and make every outcome a recorded event.
The example. Identity: the agent’s workload identity is attested. Policy: award v2026.3 and the fatigue rule pass. Evidence: fatigue record v2210 is the current version. Approval: one hour of overtime needs none; two hours routes to the unit manager, whose approval is bound to the payload digest and expires in four hours. On permit, the roster system writes the assignment and returns RST-77102; the record binds proposal digest, versions, approval and committed result. If the evidence service does not answer, nothing is written and the request is held: an unavailable record is a hold, never a permission.
What to measure. Outcomes by type (permit, escalate, deny, hold) and reason, approval wait, holds per day against the failure budget, and the count of commits without a record, which must be zero.
Failure and recovery. The roster system times out after the write was sent. The record says unknown; recovery reads the roster system before any retry, and a retry under the same idempotency key is the only one permitted. The evidence Field Note covers how that record is sealed and verified later.
Why it matters. This is the fifth rung of the Agentic AI Playbook seen from the engineer’s chair: deterministic code, outside the model’s argument, deciding whether a proposal becomes a change.
7. Trace the whole workflow, not the model call
The decision. One workflow identifier from the request to the recorded result, with spans for context assembly, each model phase, each tool call, validation, approval wait, commit and record, carrying identifiers and versions and never roster content.
The example. The escalated path’s trace: 0.3 s context assembly, 0.4 s to first token from the cached prefix, 1.2 s decode, a 0.3 s fatigue lookup, validation, then 11 minutes of approval wait, a commit and a record. The OpenTelemetry GenAI conventions give names for the model spans and server metrics (gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token, gen_ai.server.request.duration); they are at “Development” status, so pin the version you adopt and expect renames. Message content attributes are opt-in in those conventions and stay off here.
What to measure. Span coverage (the share of requests whose trace reaches the recorded result), the two clocks derived from the same trace, and the gap between them by reason.
Failure and recovery. A trace that ends at the model response makes the model look like the whole process. Recovery is a trace boundary decision, not an instrumentation one: the trace ends at the record.
Why it matters. A trace shows order and duration. It does not show that the approval caused the outcome, that the omitted span never happened, or that the model’s explanation is what it computed. Use it for what it proves, and read the Evidence-First Decision Record for the rest.
8. Measure time, reliability and cost per completed, authorised outcome
The decision. Publish seven metrics together, each with its denominator, and treat a win on the first two alone as not a result.
| Metric | Denominator | What moves it |
|---|---|---|
| Time to first token | one model call | prompt length, prefix cache, chunked prefill, batch pressure |
| Decode throughput | tokens per second per GPU | batch occupancy, KV capacity, quantisation, speculation |
| End-to-end workflow latency (p50, p95) | one request to its recorded result | everything above plus queue, approval wait, holds, recovery |
| Completion rate | filled shifts ÷ requested | proposal validity, approval availability, roster drift |
| Rework | proposals rejected at commit ÷ proposals | context freshness, schema fit, policy alignment |
| Approval wait (p50, p95) | escalations | routing, approver load, expiry settings |
| Cost per completed, authorised action | (GPU-hours + tools + people) ÷ filled shifts | all of the above |
The example. Tallow Bay’s engine upgrade doubled decode throughput and changed completion rate by nothing, because the shifts were waiting on approvals. Routing one-hour overtime to automatic permit under the envelope cut approval wait by half and moved the completion rate, which the token dashboard could never have shown.
On cost. Tokens per dollar depends on batch occupancy: a GPU billed by the hour (provisioned capacity is billed that way by at least one major provider) produces tokens in proportion to how full its batches are, so a half-empty engine is an expensive one however fast each token comes out. Per-token pricing moves the risk to the provider and the denominator problem stays: the cost that matters is per completed, authorised action, and it includes the approver’s eleven minutes.
Failure and recovery. A monthly report quotes p50 workflow latency and omits p95 and rework. Recovery: the table above is the report, with all seven rows or none.
Why it matters. Denominators decide what a number means. “Latency fell 40 per cent” is a statement about tokens unless the denominator says otherwise.
Follow one request through the route
Interactive companion. Choose a path (within envelope, needs approval, hold and re-plan), then play or step through eight stages while two clocks run: model latency and business-outcome latency. A text equivalent is provided below the controls. Values are illustrative.
The companion has three paths. On the first, one hour of overtime stays within the envelope and the business clock stops at a few seconds. On the second, two hours of overtime escalates: the model clock stops at 1.9 seconds and the business clock runs for eleven minutes until the unit manager decides. On the third, the evidence service is unavailable at commit time: nothing is written, the request holds, and when the service returns the ward roster has moved on, so a fresh proposal is made rather than the stale one retried. Play, pause, step and restart are buttons; the arrow keys step; the sequence plays without motion when your device asks for reduced motion.
What this route does not establish
The route makes the engine’s trade-offs legible and puts deterministic checks where the change happens. It does not make a model’s ranking right, it does not prove that an approval caused a good outcome, and it does not turn a trace into a reconstruction. The serving figures quoted from papers are the authors’ own on their setups; Tallow Bay’s numbers are invented to show the shape of the measurement, not its value. Engine behaviour, metric names and standards status change between versions; the sources below are as at 10 November 2026.
Sources and further reading
- The Contextual Agentic Enterprise, Chapters 9, 16 and 17 (chapter titles as listed on the book page).
- Verma, S. and Vaidya, N., “Mastering LLM Techniques: Inference Optimization”, NVIDIA Technical Blog, 17 November 2023, updated 27 December 2025. Prefill and decode characterisation; KV cache sizing example.
- Kwon, W. et al. (2023) “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023. Throughput figures are the paper’s own.
- Yu, G.-I. et al. (2022) “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI 2022. Iteration-level scheduling.
- Zhong, Y. et al. (2024) “DistServe”, arXiv:2401.09670. Disaggregated prefill and decode; goodput.
- vLLM documentation (opened 10 November 2026): Optimization and Tuning, Automatic Prefix Caching, Prefix caching design, Security, Speculative Decoding, FP8 W8A8, Parallelism and Scaling, Metrics. One engine’s documented behaviour.
- Zheng, L. et al. (2023) “SGLang”, arXiv:2312.07104 (RadixAttention).
- Ainslie, J. et al. (2023) “GQA”, arXiv:2305.13245.
- Frantar, E. et al. (2022) “GPTQ”; Lin, J. et al. (2023) “AWQ”; Xiao, G. et al. (2022) “SmoothQuant”. Quality and speed claims are each paper’s own.
- Leviathan, Y. et al. (2022) “Fast Inference from Transformers via Speculative Decoding”; Chen, C. et al. (2023) “Accelerating Large Language Model Decoding with Speculative Sampling”; Li, Y. et al. (2024) “EAGLE”.
- Shoeybi, M. et al. (2019) “Megatron-LM”, arXiv:1909.08053. A training paper, cited for the definition of tensor parallelism only.
- NVIDIA TensorRT-LLM documentation, “Speculative Decoding” and “GPT Attention” (opened 10 November 2026). One engine’s documented behaviour.
- OpenTelemetry, “Semantic conventions for GenAI metrics” and “GenAI agent and framework spans” (Status: Development, opened 10 November 2026).
- Zheng, Z. et al. (2024) “The Early Bird Catches the Leak”, arXiv:2409.20002. Timing side channels in LLM serving.
- AWS, “Provisioned Throughput”, Amazon Bedrock User Guide (hourly billing of purchased capacity; no prices cited).
The book develops the patterns and the evidence at length
The book treats the enterprise agent design patterns, the governance and evidence constructs, and observability and replay in the chapters listed above, including the full Context Contract and evidence bundle that this playbook names only in grouped form. Any adoption of CAE should be tailored to the organisation in consultation with Contextual Agentic or another advisory service. More about the book.
This article is part of the AI Engineer Playbook, a CAE series accompanying the book, The Contextual Agentic Enterprise.
Related
The Agentic AI Playbook: Five Rungs from Model to Governed Enterprise
LLM, RAG, agent, agentic system: every popular diagram stops at four rungs. The enterprise needs a fifth, where authority is checked before anything changes.
The AI-Native SDLC: What Moves When Agents Write the Code
When agents produce most of the code, the bottleneck moves from typing to context and verification. Five things shift: the artifact, the review, the tests, the roles and the bottleneck, and the evidence for each is thinner than the slogans suggest.
The Commit Boundary: Where AI Recommendations Become Enterprise Consequences
Controls placed at the start of an agent conversation check the wrong thing. The decisive check evaluates the exact action about to become real, at the moment it becomes real.
Evidence-First Decision Records: Reconstructing What the Agent Knew
A model's explanation is not an audit record, and logs are not a reconstruction. An Evidence-First Decision Record keeps claims, evidence and versions retrievable long after everything has changed.
Who Can Prove What an Agent Decided?
Traces, audit logs and tamper-evident ledgers each prove something different. For a consequential agent-assisted decision, the evidence that must survive is the binding between proposal, versioned evidence, delegated authority, approval and the change that was actually made, and someone outside the writing team must be able to check it.
Why Enterprise AI Needs a Deterministic Shell
Don't try to make the model deterministic or simply trust it. Let it propose, and put identity, authority, schema, state and commitment in a shell that decides at the Commit Boundary.
Context Is Not a Prompt: Introducing the Context Contract
Most context failures are not retrieval misses. They are correct material given to the wrong purpose. A Context Contract declares, versions and tests what an agent may know.