Inference at scale

Running a model for one person on one GPU is easy. Running it for thousands of people at once, quickly and without wasting expensive hardware, is the problem every AI provider and every self-hosting team has to solve. The choices made here decide how fast your agent feels, how much each request costs, and how many users one machine can carry.

This article covers the mechanics of inference, which is the word for running a trained model to get predictions, as opposed to training it.

Two phases: prefill and decode

Every request to an LLM goes through two distinct phases.

Prefill is when the model reads your prompt. All the prompt tokens are already known, so the GPU can process them in parallel, in one big pass. This is the kind of work GPUs are best at (Article 9): lots of arithmetic at once. Prefill ends when the model produces the first output token.

Decode is when the model writes the answer. As Article 1 explained, generation is a loop: one token per pass, each depending on the one before. So decode is sequential. For each new token, the GPU has to read the model's weights from memory again, plus the KV cache (Article 8), and do relatively little arithmetic with them.

That difference matters. Prefill is limited by how fast the GPU can calculate. Decode is mostly limited by how fast it can move data out of memory. They are bottlenecked by different parts of the hardware, which is why serving systems treat them differently.

The diagram shows the two phases of one request, and where the two most quoted speed numbers come from.

flowchart TD
    P["Prompt: 6,000 tokens"]:::io --> F["Prefill: all tokens at once"]:::model
    F --> T["First token arrives"]:::warn
    T --> D["Decode: one token per step"]:::model
    D -->|"repeat"| D
    D --> R["Full answer streamed"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 10.1: Prefill reads the whole prompt in parallel and decides the time to first token. Decode then produces the rest one token at a time.

Latency and throughput

Two words get used loosely, so let's pin them down.

• Latency is how long one user waits. For LLMs it splits into two numbers. Time to first token (TTFT) is the wait before anything appears, which is mostly queueing plus prefill. Time per output token is the gap between tokens after that, set by decode speed.
• Throughput is how much total work the system does, usually measured in tokens per second across all users.

These pull against each other. The main lever that trades one for the other is batching.

Batching: sharing one pass between many users

During decode, the GPU spends most of each step loading the weights from memory. If it is going to load all those weights anyway, it might as well use them for many requests at once. Processing several requests together in one pass is called batching.

Batching a second request onto a decode step costs much less than running it separately, because the expensive part, reading the weights, is shared. So throughput goes up a lot. But each step does get somewhat slower as the batch grows, so each individual user sees their tokens arrive a bit more slowly.

Worked example: the trade-off in numbers

The numbers below are illustrative, chosen to show the shape of the trade-off. Real figures depend on the model, the GPU and the serving software.

Suppose one decode step takes:

• 20 ms with a batch of 1
• 25 ms with a batch of 16
• 40 ms with a batch of 64

Each step produces one token for every request in the batch. So:

Going from a batch of 1 to 64, each user's speed halves (50 to 25 tokens per second), but the machine does 32 times as much work. If you pay for the GPU by the hour, the cost per token falls by the same factor. This is why providers batch aggressively, and why a dedicated, unbatched model is fast but expensive.

Static versus continuous batching

The simple way to batch is static batching: collect a group of requests, run them together until every one has finished, then start the next group. The problem is that answers have very different lengths.

Imagine four requests that need 50, 100, 200 and 400 output tokens. In a static batch, all four slots stay reserved for 400 steps, because the batch is not done until the longest request is. The work actually needed is 50 + 100 + 200 + 400 = 750 token steps, out of 4 × 400 = 1,600 slot steps reserved. That is 750 ÷ 1,600 ≈ 47% useful work. More than half the batch capacity sits idle.

Continuous batching fixes this by deciding the batch again at every step. When a request finishes, its slot is immediately given to the next request in the queue. The batch is always full of live work.

The chart compares the two approaches using the four requests above, with new requests E to H waiting in the queue.

Figure 10.2: Teal is useful work, dashed coral is idle capacity. Continuous batching hands each freed slot to the next waiting request, so the GPU stays busy.

I am simplifying here. Real schedulers also have to decide how to mix new requests' prefill work with existing requests' decode steps, since a big prefill can briefly slow everyone else down. Some systems split long prefills into chunks, and some even run prefill and decode on separate GPUs. The core idea stays the same: keep the batch full.

Memory is the real limit: PagedAttention

Every active request needs its own KV cache, the stored intermediate results that save the model from re-reading the whole conversation for each new token (Article 8). The KV cache grows with every token, and for long prompts it can be large. On a busy server, KV cache memory, not compute, usually decides how many requests fit in a batch.

Older serving code reserved one continuous block of GPU memory per request, sized for the longest answer it might produce. Most requests never use that much, so a lot of memory was reserved but empty.

PagedAttention, introduced with the open source serving engine vLLM in 2023, borrows an idea from operating systems. It splits the KV cache into small fixed-size blocks, called pages, and hands them out as a request actually grows. A table keeps track of which pages belong to which request, so the pages do not need to sit next to each other in memory. Less memory is wasted, more requests fit in a batch, and throughput goes up. Requests that share the same prompt prefix can even share the same pages, which is one way prompt caching (Article 13) is implemented.

The diagram shows the main parts of a vLLM-style serving setup.

flowchart TD
    C["Agent harness requests"]:::io --> A["API server"]:::infra
    A --> S["Scheduler: continuous batching"]:::infra
    S --> G["GPU: model weights"]:::infra
    S --> K["Paged KV cache"]:::model
    G --> O["Streamed tokens"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8

Figure 10.3: A serving engine. The scheduler rebuilds the batch every step and hands out KV cache pages as requests grow.

vLLM exposes an OpenAI-compatible HTTP API, so applications talk to a self-hosted model the same way they would talk to a hosted one. Other serving engines exist, such as SGLang and NVIDIA's TensorRT-LLM, and as of October 2026 they compete closely on performance, and which is fastest depends on the model and the workload.

Mixture-of-experts: big models that run like smaller ones

One more technique changes the economics of inference. In a normal, dense model, every parameter is used for every token. In a mixture-of-experts (MoE) model, some layers are split into several parallel sub-networks called experts, plus a small router that picks which few experts handle each token. The rest sit unused for that token.

This gives two different parameter counts:

• Total parameters: everything in the model. This decides how much memory you need, because every expert must be loaded, since any token might need any of them.
• Active parameters: what is actually used for one token. This decides how much compute each token costs.

Worked example: Mixtral 8x7B

Mistral AI published these figures for its Mixtral 8x7B model: 8 experts per MoE layer, 2 chosen per token, about 46.7 billion total parameters and about 12.9 billion active per token.

Memory. At 16-bit precision each parameter takes 2 bytes. So the weights need 46.7 billion × 2 bytes ≈ 93 GB. You must hold all of it.

Compute. A common rule of thumb is about 2 arithmetic operations per active parameter per token. So:

• Mixtral: 2 × 12.9 billion ≈ 26 billion operations per token.
• A dense 46.7-billion-parameter model: 2 × 46.7 billion ≈ 93 billion operations per token.

So Mixtral needs the memory of a 47B model but does roughly the per-token arithmetic of a 13B model, about 3.6 times less than a dense model of the same size. That is why many large models are MoE: you get the knowledge capacity of a big model at closer to the running cost of a small one, provided you have the memory.

Where this shows up in our bug-fix agent

Working on BUG-1423, the agent calls the model many times: to plan, to search for find_user_by_email, to read auth/login.py, to write the fix, to read the pytest output, and to write the pull request description. Each call sends the whole conversation so far.

• Prefill dominates. Each call's prompt is thousands of tokens of instructions, code and test output, while each answer may be only a few dozen tokens, like a tool call or the one-line fix email = email.strip().lower(). Most of the compute is reading, not writing. That makes prompt caching (Article 13) especially valuable.
• TTFT is what the developer feels. If a developer is watching the agent work, a long wait before each step feels slow even when the total throughput is fine.
• Batch jobs want throughput. If the same agent triages 500 old tickets overnight, nobody is watching. Those requests can go to a large batch, or a provider's discounted batch API, trading latency for cost.
• The provider's choices leak through. Whether the model behind the agent is dense or MoE, and how full its batches run, shows up to you as price per token and speed. Article 17 shows how an LLM gateway can route urgent and batch traffic differently.

Common misconceptions

"A faster GPU means my answers come faster." Not necessarily. On a shared service, your speed depends on how full the batch is and how long the queue is. A busy provider on fast hardware can feel slower than a quiet one on older hardware.

"Tokens per second is one number." There are two: what one user sees, and what the whole system produces. A vendor quoting thousands of tokens per second usually means total throughput across many users, not what your request will get.

"An MoE model with 13 billion active parameters needs as much memory as a 13B model." It needs memory for all its parameters, active or not. Active parameters set the compute cost, total parameters set the memory bill.

Key takeaways

• Inference has two phases: prefill reads the prompt in parallel and sets time to first token, decode writes the answer one token at a time and is limited by memory speed.
• Batching shares each pass between many requests, raising total throughput and lowering cost per token, at the price of slightly slower speed for each user.
• Continuous batching refills each slot the moment a request finishes, avoiding the idle capacity of static batches.
• KV cache memory usually limits batch size. PagedAttention, from vLLM, allocates it in small pages to waste less.
• Mixture-of-experts models need memory for all their parameters but only compute with the few experts each token uses.

Glossary

• Inference: running a trained model to get outputs, as opposed to training it.
• Prefill: the phase where the model processes the whole prompt in parallel.
• Decode: the phase where the model generates output tokens one at a time.
• Latency: how long a single request waits.
• Time to first token (TTFT): the delay before the first output token appears.
• Throughput: the total amount of work a system completes, usually tokens per second across all users.
• Batching: processing several requests together in one pass through the model.
• Static batching: running a fixed group of requests until all of them finish.
• Continuous batching: rebuilding the batch at every step so finished requests are replaced immediately.
• PagedAttention: storing the KV cache in small fixed-size pages allocated on demand.
• vLLM: an open source engine for serving LLMs with high throughput.
• Mixture-of-experts (MoE): a model design where a router sends each token to a few expert sub-networks.
• Total parameters / active parameters: all the parameters in an MoE model, versus those used for one token.

Next up

So far we have assumed someone else's model on someone else's servers. In Article 11: Proprietary vs open-weight models, we look at the choice between calling a closed model through an API and running one whose weights you can download, what the licences actually allow, and why "open weight" and "open source" are not the same thing.