Contextual Agentic

What Survives a Model Swap: Keeping the Shell, Replaying the Decisions

Models behind an agent are retired on published schedules, and a newer one does not behave like the old one. This note sorts an estate into what carries over, what is re-tested, what is rebuilt and what is rented, and shows how to replay past decisions through a new core before cutover. Includes a fictional example, an original animated diagram and a companion simulator with invented numbers.

·17 min read

Written as at 2 November 2026.

enterprise-architectureagentic-aimodel-lifecycleevaluationevidenceai-governance

A model swap used to sound like a project. It now behaves more like weather. Hosted models are retired on published schedules, replacements arrive with different habits, and a cheaper or more capable option appears every few months. The question for an enterprise is not which model leads this quarter. It is what the estate still holds on the day the model behind an agent changes, and how fast it can find out what else changed.

This note was prompted by a public LinkedIn post by @rakeshgohel01 on that very question, which framed it as owning what ‘good’ means after a swap. The framing below, the example, the three diagrams and the companion simulator are this site’s own, built on the Contextual Agentic Enterprise (CAE), Contextual Agentic’s framework. No wording, chart or layout from the post is reused.

A model swap is routine, and an estate learns what it owns when one arrives

Consider what a swap actually triggers. A provider announces that a model will be retired, or finance asks why a workload costs what it does, or a regulator or customer asks where inference happens, or a better model ships. Whatever the cause, someone has to answer four plain questions. Which parts of the estate never depended on the old model? Which depend on it only loosely? Which are shaped to its habits? And how will anyone know, before customers do, that the new one behaves differently?

Organisations that cannot answer them tend to discover the answers in production. A tuned prompt starts producing malformed output. A reviewer notices that approvals have crept up. A fine-tune that nobody documented quietly stops being available. None of this is exotic. It is the ordinary cost of renting the most volatile component of the system and building the rest around its quirks.

The remedy is not to avoid swaps. It is to arrange the estate so that the model is the part that changes and everything else is a fixed frame that the new model has to fit.

Providers publish the schedule and the interface, and say little about behaviour

The published terms differ, and they are worth reading as a set, because together they show that a swap is scheduled rather than exceptional. As at 2 November 2026, Anthropic’s deprecation page commits to at least 60 days’ notice before a publicly released model is retired, and says that requests to a retired model fail. It also says deprecated models are likely to be less reliable than active ones, so the date that matters in practice may be earlier than the retirement date. OpenAI’s page gives generally available models at least six months’ notice and specialised variants at least three, and says preview models can be retired on notice as short as about two weeks. It also allows shorter notice where safety or compliance requires it.

Microsoft’s retirement page for Azure OpenAI models in Foundry describes an 18-month lifecycle for generally available models and at least 60 days’ notice to subscription owners. It adds two details that matter to an architect. A retired deployment returns an HTTP 410 error. And standard deployments can be upgraded automatically to a newer version, depending on a setting that can also be set to never upgrade, in which case the deployment simply stops working at retirement. Provisioned deployments are not upgraded for you. In other words, on some platforms a swap can happen to an estate whether or not anyone chose it.

Notice periods are not the same thing as behavioural continuity, and none of these pages promises the latter. They describe dates and interfaces. A provider’s own migration guide shows what that means in practice. Anthropic’s guide for moving to its newer Sonnet model lists request settings that are now rejected with an error, including sampling parameters, a prefilled assistant turn and forced tool choice, and tells readers to re-evaluate model-specific prompt instructions and re-run their effort settings. OpenAI’s guide for its newest models similarly recommends comparing old and new on your own representative workloads at the same reasoning effort, and auditing the instruction files a model reads. Both are one provider’s advice about its own models. Neither claims a new model will behave like the old one.

Two research papers give the underlying reason for the caution, with limits. Chen, Zaharia and Zou compared the March and June 2023 versions of the GPT-3.5 and GPT-4 services on seven kinds of task and found substantial changes, in both directions. GPT-4’s accuracy on one prime-number task fell from 84 per cent to 51 per cent, while GPT-3.5 improved on the same task. They conclude that a service with the same name can change meaningfully in a short time and needs continuous monitoring. It is a study of two snapshots of two services, not a rule. Sclar and colleagues studied formatting alone and found that small, meaning-preserving changes to a few-shot prompt moved accuracy by as much as 76 points for one open model. The sensitivity persisted with larger models, more examples and instruction tuning, and which formats worked well was only weakly correlated from one model to another. That was a study of few-shot prompting on open models, so it does not transfer to every production agent. But it supports one practical point: a prompt that was tuned to one model is not a portable artefact.

The Deterministic Shell stays, and the Probabilistic Core is what gets swapped

CAE splits an agentic system into a Probabilistic Core and a Deterministic Shell. The core is the model: it interprets, reasons and proposes. The shell is everything that does not depend on that reasoning being right: identity, the Context Contract that states what the core may see and propose, the authority envelope, the checks at the Commit Boundary, and the evidence records. A well-built shell lets a wrong or changed model output fail safely. Earlier notes cover the shell in detail, in Why Enterprise AI Needs a Deterministic Shell and The Contextualised Enterprise.

A swap is the sharpest test of that split, because the core is exactly the part being exchanged. If the shell is real, it should not notice. If it quietly depends on the old core, the swap will reveal it. A useful check is to search the shell, the policies, the schemas and the evidence records for the old model’s name. Every hit is a place where the frame was shaped by the thing it was meant to contain.

The first figure draws the idea. Records, the Context Contract, the policy gate and the system of record stay in place. Only the core changes, and four steps follow: notice, replay, compare and decide. The picture plays once, in under five seconds, and is already complete if a device asks for reduced motion. The same steps can be walked one at a time in the companion lab, Model swap replay.

Diagram of a model swap. The decision records, the Context Contract, the policy gate at the commit boundary and the system of record stay in place. Only the probabilistic core changes, from Core A, which is retiring, to Core B, which is new, re-tuned and re-tested. Four numbered steps: notice that the core is retiring, replay decision records through the new core, compare at the commit boundary, and decide to cut over, run a canary or hold. A conceptual design, not a validated architecture.
Figure 1. What a model swap changes and what it does not. A proposed conceptual design, not a validated architecture.

Text description of Figure 1. Decision records feed the probabilistic core, which proposes actions. The Context Contract is attached to the core. The core’s proposals pass to a policy gate at the commit boundary, which decides from facts the core cannot assert for itself, and then to the system of record. In the swap, Core A is retired and Core B takes its place. Everything else is unchanged. The four steps are notice, replay, compare and decide.

Four fates: what survives, what is re-tested, what is rebuilt and what is rented

Sorting an estate by what happens to each part in a swap gives four groups, drawn in the second figure.

The first group stays as it is, because nothing in it names a model. That is the Context Contract, written as a versioned specification of what the core may see and propose, which Context Is Not a Prompt covers in its own right. It is the policy and authority checks at the commit boundary, which decide from identity, delegation and current state, and which The Model Is an Engine, Not the Database keeps outside the model, with entitlements checked first. And it is the decision records, written in the manner of Evidence-First Decision Records: what was proposed, decided and committed, and on what basis. These three do the most to determine whether a swap is calm or frightening. They are also the cheapest to keep model-neutral, because they are ordinary engineering.

The second group stays, then needs a re-test. The evaluation set carries over, because it is a set of cases with known right outcomes. The procedures and tools carry over, because they are written as files and typed calls rather than prose tuned to one model. But a different model reads the same procedure differently, so each needs to be run again. So do any checks that use a model as the grader, since the grader is itself a model that has changed. Evaluation Is Not a Test Phase treats evaluation as ongoing rather than a phase. A swap is the moment that argument becomes concrete.

The third group is rebuilt. Prompt text that was tuned to the old core’s habits has to be rewritten and re-tuned, not translated. A fine-tune or adapter is tied to its base model. OpenAI’s pages say a fine-tuned model follows the shutdown date of its base model, and Microsoft’s describes separate retirement dates for training and for deployment of fine-tuned models. Nothing on either page suggests a fine-tune transfers to a new base. Whoever depends on one has a retraining decision hiding inside the swap.

The fourth group is rented or held. A hosted model follows the provider’s timetable. An open-weight model follows yours, and also brings serving, patching and re-evaluation as work you own. Owning the Weights Is Not Owning the System covers what changes and what does not when the weights are yours, so this note does not repeat it.

Four bands. Stays as it is: the Context Contract, the policy gate at the commit boundary and the decision records, because nothing in them names a model. Stays, then re-tested: the evaluation set, procedures and tools, and model-based graders. Rebuilt: prompt text tuned to the old core, and fine-tunes and adapters. Rented or held: the model itself, on the provider's schedule or your own. A proposed conceptual sorting, not a validated architecture.
Figure 2. What survives a model swap, in four groups. CAE's way of sorting an estate.

One more part of the estate sits above the model and is easy to forget. Platforms increasingly run the agent loop for you. Microsoft’s Copilot harness for Foundry prompt agents, announced on 28 October 2026 and in public preview, is a managed runtime in which Microsoft runs the loop and sandbox while the developer supplies the model, instructions, skills and tools. Its documentation notes that the harness is chosen when an agent is created and cannot be switched afterwards, and that bringing a model through an AI gateway is not currently supported. None of that is a criticism. It is a reminder that the loop is another rented layer, and the same four questions apply to it: what would carry over if it changed, and what is shaped to its habits.

A fictional returns agent changes its core

Illustrative scenario: a fictional composite written to make the point concrete. It does not describe a real organisation, and its figures are not reported outcomes. The numbers come from the invented profiles in the companion lab.

A parts distributor runs an agent that handles return requests. For each request it proposes one of three actions: approve the refund, deny it, or escalate it to a named approver. It reads the order, the customer’s history and the returns policy, and it proposes a refund amount. A gateway checks the proposal before anything reaches the finance system. The policy says refunds above a delegated limit must be escalated, and certain accounts are blocked. The Context Contract is a versioned file. The returns procedure is written as tuned prompt text. A thirty-case evaluation set was assembled by hand two years ago.

The provider announces the retirement of the model behind the agent. A newer model is stronger on paper at every kind of request. The team has the past eighteen months of decision records, which hold what was proposed, what the gateway decided and what was committed.

They replay the records through the old and new models, with the prompts left as they were. The new model’s structured output fails on roughly one request in nine, because it reads the old prompt’s formatting habits differently. Those requests are held for a person, so wrongful refunds do not rise, but the queue grows. The gateway overrides every proposal to approve a refund that policy would have escalated or denied, so the policy-covered errors never reach the finance system. What it cannot see is a wrong refund amount inside the limit. That class of error is the one that remains, and the replay is the only thing that measures it.

They then re-tune the prompts for the new model and replay again. The format failures fall to about the old level and the correct-outcome rate ends above the old model’s. The thirty-case evaluation set, run beside the replay, said only that it could not tell. It gave the team no basis for either a cutover or a hold.

Nothing in this story required cleverness. It required that the shell and the records existed before the notice arrived.

A swap can be run in five steps, with a replay at the centre

The practice below is a plain sequence, not a named method.

  1. Inventory. List what the estate holds under the four groups, using the ledger. Mark anything that names the old model, and mark what is missing: no gate, no decision records, no evaluation set. The missing items are the first work.
  2. Assemble the replay corpus. Use real decision records, sampled to match the live mix of cases, including the rare ones. If records were not kept, an evaluation set written from imagination over-represents easy cases, which is where a swap looks safest.
  3. Replay through both cores. Run the same cases through the current core and the new one, behind the same gate. Run the new core twice: once with prompts unchanged, to see the cost of inheriting the old style, and once re-tuned, to see what the work buys.
  4. Compare at the commit boundary. Do not compare text. Compare the outcomes that would have become real: the share of correct outcomes, wrongful commits, cases held for a person, and proposals the gate overrode. State the tolerance in advance, and report the margin the sample allows.
  5. Decide, then keep watching. Cut over, run a canary on a small share of traffic, or hold and re-tune. Keep replaying a sample of live decisions after cutover. Chen and colleagues’ conclusion about monitoring applies to a model that has not been swapped as well.

The third figure draws the replay and its comparison.

Flow diagram. Decision records become a replay corpus of the same cases. The cases run through the current core and the new core, both behind the same policy gate at the commit boundary. Outcomes are sorted into correct outcomes, wrongful commits and cases held for a person. They are compared against a stated tolerance with the margin the sample allows. The result is either cut over or run a canary, or hold and re-tune. A proposed conceptual design, not a validated architecture.
Figure 3. Replaying decisions before a swap. A proposed conceptual design, not a validated architecture.

Text description of Figure 3. Decision records give a replay corpus. The same cases go to the current core and to the new core, each behind the same policy gate. The outcomes that result are correct, wrongful or held for a person. They are compared with the tolerance and the margin the sample allows. The decision is either to cut over or run a canary, or to hold and re-tune.

A thin evaluation set cannot tell a small change from none

The thirty-case set in the example is not a strawman. Many teams hold one. The trouble is arithmetic. If each core gets about 94 per cent of cases right, the difference between them measured on 30 cases carries a 95 per cent margin of roughly plus or minus 12 points. On 400 cases it is roughly plus or minus 3. The figures come from the usual normal approximation for the difference of two proportions, treating the runs as independent, so they are a guide rather than a guarantee. A margin of twelve points cannot distinguish a tolerable change of four points from a damaging one of fifteen.

That does not make small sets useless. They are good at catching a total failure and at showing what a case looks like. They are poor at licensing a decision that the change is within tolerance. The honest verdict from a thin set is often ‘inconclusive’, and the lab reports exactly that. Two things shrink the margin: more cases, and cases that match the live mix. The second matters more than it sounds. A set weighted toward routine requests will show the new model doing well on the cases where nothing is at stake and say nothing about the rare cases where something is.

Six ways a swap goes wrong

The shell quietly depended on the old core. A schema, a parser or a policy was written around the old model’s output habits. The swap breaks it, and the break looks like a model problem.

Prompts are carried over as if portable. The evidence on formatting sensitivity, limited as it is, says that format effects differ by model. A prompt tuned to one core is a dependency, not a specification.

The only evaluation is the one the old core was tuned against. A set built from the old core’s best cases flatters the old core and understates the change.

Errors the gate cannot see are assumed to be handled. A policy gate absorbs errors that policy can express. It does not catch a wrong amount, a wrong recipient or a wrong summary inside policy. Those need their own checks and their own measurement.

A silent swap. On platforms that upgrade deployments automatically, a version can change without a decision. Pin versions where the platform allows it, and replay samples of live decisions so that a change is noticed.

Records are kept in the wrong place. Decision records may hold personal or commercial information. The replay corpus needs the same access control as the originals. A public transparency log is public, so assess what it would disclose before writing to one.

What executives, architects and engineers each do

Executives decide which model changes need a documented replay before cutover, who may accept the tolerance, and what the organisation will do when a provider’s notice is shorter than its own migration time. They also decide whether decision records are kept long enough to be useful. A regulator or auditor who asks how the organisation knew a new model was safe will want the answer in records, not recollection. NIST’s AI RMF and its generative AI profile are voluntary. They point to post-deployment monitoring and change management, and the passages read for this note do not address a base-model swap by name. A replay record is one way to show that monitoring and change management were done.

Architects keep the shell model-neutral and test the claim by searching for the model’s name. They put version identifiers in configuration, not in policy. They record, for each procedure and tool, whether it is portable or tuned. And they keep the Context Contract as a versioned file that states what the core may see and propose without naming a model.

Engineers write the replay harness so that the same cases run through any core behind the same gate. They sort outcomes at the commit boundary, record the seed and the configuration for each run so a result can be reproduced, and report margins along with rates. They keep prompts, procedures and tools in version control with the core they were tuned for noted beside them. And they treat provider notices, version pinning and automatic-upgrade settings as part of the runbook.

Part of the estate Question to ask before a swap If the answer is poor
Context Contract Does it name any model? Is it a versioned file? Extract it from prompts before the swap
Policy gate at the commit boundary Does it decide from facts the core cannot assert? Put one in place first; nothing else compensates
Decision records Are they kept, accessible and representative? Start keeping them; an evaluation set is a poor substitute
Evaluation set How many cases, and from where? Widen it, or report a verdict of inconclusive
Procedures and tools Portable files, or tuned prompt text? Plan a rewrite and re-tune
Fine-tunes and adapters Which base model are they tied to? Plan a retrain or a retirement
The model Whose schedule is it on? Pin versions and watch published notices

What this does not establish

This note does not say which model to pick or when to swap. It does not show that any provider’s model changes in any particular way; the research cited is limited to the tasks and dates stated. The simulator is a teaching toy: its cores, accuracies, traffic mix and tolerance are invented, and it shows the shape of a swap rather than a measurement. The gate in it knows the policy answer for three of the four kinds of case and nothing about the fourth, and real estates have all four in proportions of their own.

It also does not cover everything a swap touches. Contract terms, data residency and the provider’s data handling may change with a different provider, and a regulated function may need a fresh approval whatever the replay says. The replay only tells you about the past distribution of cases. Cases that did not occur in the records are not tested by it. The CAE framework is Contextual Agentic’s own, and nothing here has been validated across organisations.

Key takeaways

  • A model swap is a scheduled event for hosted models, and on some platforms it can occur through automatic upgrades, so the estate should be arranged to expect it.
  • Published terms cover dates and interfaces, not behaviour, and the research available shows behaviour and prompt sensitivity can differ between versions and models, within stated limits.
  • Parts of the estate that never named a model, namely the Context Contract, the policy gate and the decision records, carry over, and they are the cheapest to keep that way.
  • Prompts tuned to one core and fine-tunes tied to one base model are rebuilt, not translated.
  • Replaying past decisions through both cores behind the same gate, and comparing outcomes at the commit boundary with a stated tolerance and margin, finds what a thin evaluation set cannot.
  • A policy gate absorbs errors that policy can express; wrong amounts inside policy need their own measurement.

Sources and further reading

The book develops this ground at length

The book The Contextual Agentic Enterprise treats the enterprise model portfolio in its own chapter, alongside the deterministic and probabilistic parts of an agentic system. More about the book

This article is part of CAE Field Notes, a series accompanying the book, The Contextual Agentic Enterprise.

← All articles

Related