Contextual Agentic

The Agent Check-up: A Quarterly Health Review for Agents That Can Act

Agents drift. Their models change, their tools multiply, their grants linger and their context goes stale. A quarterly check-up with the same eight vital signs catches it before an incident does.

·7 min read

Written as at 17 August 2026.

agentic-aiai-operationsai-assuranceai-governanceplaybook

Nobody would put a new member of staff into a role with access to payments and customer accounts and then never review them again. Yet that is roughly how many organisations treat AI agents. There is a careful launch: design review, security sign-off, a pilot, a go-live. Then the agent joins the estate and is looked at again only when something goes wrong.

The trouble is that agents change even when nobody touches them. The model behind them is upgraded by the provider. Someone connects a new tool because it was one line of configuration. A permission granted for a one-off migration is never removed. The policy document the agent retrieves from is replaced, and the index still holds the old one. Each change is small, and none of them triggers a review. Six months later you are running a different agent from the one you approved.

The fix is unglamorous and effective: a regular check-up, with the same questions every time.

The check-up cycle

A five-step cycle: intake (who owns it, what it can touch), history (incidents, reversals, changes since last review), tests (evaluations, red-team, tool inventory diff), plan (narrow, retrain, retire or promote) and follow-up (date set, owner named, recorded), repeated every quarter and after any incident. Below, a sample chart for a service-desk agent with eight vital signs marked healthy, watch or treat.
Figure 1. The agent check-up: five steps and eight vital signs. The values in the chart are examples.

The shape will be familiar to anyone who has been to a GP: intake, history, tests, a treatment plan, and a follow-up date.

1. Intake. Confirm the basics before anything else. Who is the named human owner, and are they still in the role? What is the agent for, in one sentence? Which systems can it read, and which can it change? If the owner has moved on, stop here: an agent without an owner is the first thing to fix.

2. History. What has happened since the last review? Incidents and near misses; actions that were reversed or corrected; escalations and how they were resolved; and every change to the model, the prompt, the tools or the grants. A change log for the agent is the single most useful document in the check-up, and many teams discover at their first review that they don’t have one.

3. Tests. Run the evaluation suite again, even if nothing has “changed”, because the model underneath may have. Compare the agent’s current tool inventory with the approved one. Where the agent handles anything sensitive, run a short red-team exercise against the latest known prompt-injection patterns. And sample real runs, not just test cases: agentic flow mining, which discovers the paths the agent actually took and checks them against the ones it should take, is the best way to do this at scale (see Agentic Flow Mining).

4. Plan. Every check-up ends with one of four decisions for each action type the agent performs: narrow its scope or grants, retrain it (a new prompt, new context, new evaluation cases), retire it, or promote it one step up the autonomy staircase. “No change” is allowed, but it has to be written down as a decision.

5. Follow-up. Set the date of the next review, name who owns each action, and record the outcome. Then run the cycle again every quarter, and immediately after any serious incident.

The eight vital signs

The chart is where the check-up earns its keep. The same eight measures, every quarter, for every agent that can change something, with each one marked healthy, watch or treat.

Identity and scope. Does the agent run under its own workload identity, with no shared keys and no borrowed human credentials? Does what it can reach still match what it is for?

Authority grants. Is every grant still used, still sponsored by a named person, and still time-limited? Unused grants are the easiest risk in the estate to remove. A grant that hasn’t been exercised in ninety days should be removed unless someone argues for it.

Context freshness. When were the documents, policies and reference data the agent relies on last refreshed? A policy pack that is four months old is a policy the agent may be applying incorrectly today.

Tool inventory drift. Which tools, MCP servers or APIs has it gained or lost since the last review, and who approved each change? New tools are new capabilities; they deserve the same review as the original design.

Evaluation scores. How does the agent score on its regression suite now, compared with last quarter? A small drop is a watch; a drop you can’t explain is a treat.

Escalation rate. What share of its actions does it hand to a person, and is that rising or falling? A sudden fall can be as worrying as a rise: it may mean the agent has found a way round its limits.

Evidence completeness. Does every action have a complete evidence record: what was proposed, which grant covered it, which policy allowed it, and why? Actions with a missing reason are the ones you won’t be able to explain later.

Undo path tested. When did you last rehearse reversing, or compensating for, one of its actions in a realistic setting? An undo path that has never been tested is a hope, not a control.

Animated: an agent monitor with a heartbeat line; eight vital signs are scanned in turn and stamped healthy, watch or treat, followed by a treatment plan.
Figure 2. A check-up in progress: eight vital signs, one plan. Example values. Animated; shown as a still image if your device is set to reduce motion.

Making it routine

Three things turn the check-up from a good intention into an operating habit.

Automate the chart, not the judgement. Most of the eight measures can be produced automatically from the agent’s evidence records, grant store, tool registry and evaluation runs. The review meeting should spend its time on the watches and treats, not on gathering numbers.

Make the owner present it. The named owner walks through the chart and proposes the plan. If nobody is willing to own an agent at its check-up, that tells you what to do with it.

Link it to the management system. If you are aligning with ISO/IEC 42001 or the NIST AI Risk Management Framework, the check-up is a natural home for their monitoring and review expectations. It gives auditors a consistent record, per agent, per quarter.

What to do this quarter

Executives: ask for a list of every agent that can change something in a system of record, with its owner and the date of its last review. The length of the list and the number of blank dates will tell you where to start.

Platform and operations teams: build the eight-sign chart as a report generated from data you already hold. Start with identity, grants and tool inventory; they are the easiest to automate and the most often wrong.

Agent owners: hold the first check-up for your agent before someone asks you to. Write the change log from memory if you have to, and start keeping it properly from today.

Agents don’t need to be treated as patients because they are fragile. They need it because they keep changing, and a regular, boring, consistent review is the cheapest way to make sure the agent you are running is still the one you approved.

References

  1. The Contextual Agentic Enterprise, Chapters 12, 17 and 22.
  2. NIST (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. https://doi.org/10.6028/NIST.AI.100-1
  3. ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system. https://www.iso.org/standard/81230.html

← All articles

Related