What Happens on the GPU When Your Agent Thinks

Most architects can design an agent without thinking about GPUs, until the latency is wrong or the bill doubles. Then the explanation is almost always physical: where the tokens sit in memory, and which part of the chip is the bottleneck.

Take one turn from Poly, the studio I am building. The researcher agent sends a 42,000-token window and gets back 600 tokens. Here is that turn, top to bottom.

A caveat before the numbers. Frontier model providers do not publish their model internals, so the GPU arithmetic below uses an illustrative open model: a 70-billion-parameter dense transformer with 80 layers, grouped-query attention with 8 key-value heads of dimension 128, weights in 16-bit precision, served on one node of eight 80 GB GPUs. The numbers are for that model only. The mechanics are the same for the models you call.

The turn starts on ordinary CPUs

The agent runtime is a modest container. It holds the message list as JSON, and at roughly four characters per token a 42,000-token window is only about 170 KB of text. Memory on the agent host is cheap. It assembles the request (system prompt, tool definitions, history, the latest tool result) and sends it.

At the provider’s front door, the request passes TLS, authentication and rate limits, which are usually set per organisation in requests and tokens per minute. A 429 response comes from here, before any GPU is involved. An overloaded response means the capacity pool behind it is full.

Next, a tokeniser turns text into token IDs, and a scheduler places the sequence into a running batch alongside other users’ sequences. The scheduler also looks up the prompt prefix in blocks. If the leading blocks match a prefix already processed, byte for byte from the first token, their stored state is reused.

Prefill is compute-bound and sets time to first token

Every uncached prompt token now goes through every layer in parallel, as large matrix multiplications. The work is roughly 2 × parameters floating-point operations per token: about 140 GFLOP each for the illustrative model. For 42,000 uncached tokens that is about 5.9 PFLOP, plus an attention term that grows with the square of the context length and adds roughly another 2.3 PFLOP here.

This phase is limited by compute. On the illustrative node at realistic utilisation it takes on the order of two seconds, and nothing streams back until it finishes. Prefill length is the main driver of time to first token.

A cached prefix skips this work for the tokens it covers. If the first 30,000 tokens (system prompt, tools and a fixed style guide) are cached, about 70 per cent of the weight computation disappears. That is why providers can bill cache reads at a fraction of the base input price, as Anthropic’s prompt caching documentation describes, and why time to first token drops with them.

The KV cache lives in HBM, and HBM is the scarce resource

Prefill leaves something behind. For every token and every layer, the model stores a key vector and a value vector so that decode never recomputes them. The size per token is 2 × layers × KV heads × head dimension × bytes: 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 320 KB. The researcher’s 42,000-token window therefore holds about 13.8 GB of KV cache in the GPUs’ high-bandwidth memory (HBM).

The weights already take 140 GB of the node’s 640 GB. Allowing some headroom, roughly 30 windows of this size fit at once. Every long window crowds out other sequences, which is part of why long context costs more and slows under load. Kwon and colleagues’ PagedAttention manages this memory in fixed-size blocks, like pages in an operating system, which cuts waste from fragmentation and lets sequences share blocks for a common prefix.

Decode is bound by memory bandwidth

Output tokens are produced one at a time. Each step reads all the weights and the sequence’s KV cache from HBM, so decode is limited by memory bandwidth, not compute. Reading 140 GB of weights across eight GPUs sets a floor of several milliseconds per step before the KV cache is counted. Typical figures for hosted models run from about 8 to 25 milliseconds per output token, so the researcher’s 600 tokens take roughly 5 to 15 seconds.

Batching is what makes this affordable. One read of the weights serves every sequence in the batch, so parallel agents share the most expensive part of each step. Fan-out cuts wall-clock time without raising the price per token.

Then the turn goes back up. Tokens stream to the runtime, which parses the tool call, validates it, runs it through the tool gateway and appends the result. The next turn re-sends the whole window. This is why Loop Engineering, my explanatory model for treating an agent’s iteration as a control loop with budgets and stop conditions enforced outside the model, starts with the size of what each loop re-reads. Loop Engineering covers the model.

Four design rules follow from the physics

The second rule is the whole argument of The Orchestrator Should Never Read the Raw Material, seen from the hardware. The fourth is the one you can watch go wrong in the worker-failure scenario of the agent simulator.

What this does not solve: you do not control the provider’s scheduler, batch sizes, hardware or precision. Mixture-of-experts models read only some of their weights per token, lower-precision weights halve the bytes, and techniques such as speculative decoding change the per-token constants. Treat the numbers as an explanation of direction, not as a service level. For that, measure time to first token and time per output token on your own traffic.

Who does what

Executives: when an agent is slow or expensive, ask for the input size, cache hit rate and output length per call before approving a bigger model.

Architects: design the prompt layout for caching, the handovers for small windows, and the fan-out for your rate limits.

Engineers: record cache read and write tokens, time to first token and output tokens on every model-call span, and add a CI check that fails when the cached prefix changes.

This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.

References

1. The Contextual Agentic Enterprise.
2. Kwon, W. et al. (2023) "Efficient Memory Management for Large Language Model Serving with PagedAttention". SOSP. https://arxiv.org/abs/2309.06180
3. Anthropic documentation, "Prompt caching". https://docs.claude.com/en/docs/build-with-claude/prompt-caching
4. Brooker, M. "Timeouts, retries, and backoff with jitter". Amazon Builders' Library. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/