GPUs explained

Almost every conversation about AI eventually becomes a conversation about GPUs: who has them, what they cost, why there are never enough. If you run infrastructure, you will be asked to size, rent or justify them. The good news is that the core ideas are simple. An LLM spends nearly all its time doing one kind of arithmetic, GPUs are built to do that arithmetic massively in parallel, and the real bottleneck is usually not the arithmetic at all but how fast numbers can be moved from memory.

CPUs and GPUs: a few experts versus a huge crowd

A CPU has a small number of powerful cores, typically from a handful in a laptop to around a hundred in a big server. Each core is fast and clever. It can jump between tasks, follow complicated branching logic and run your operating system, your database and your web server.

A GPU has thousands of much simpler cores. Individually they are weaker, but they all work at once on the same kind of operation. GPUs were designed to compute the colour of millions of pixels at the same time, where each pixel's calculation is simple and independent of the others.

An analogy: a CPU is a team of eight senior engineers. A GPU is a stadium of ten thousand people with calculators. Ask for a system design and you want the engineers. Ask for ten million additions and the stadium wins easily.

The diagram contrasts the two designs.

Figure 9.1: A CPU has a few powerful cores. A GPU has thousands of simple ones that all do the same kind of work at once. Real GPUs have far more cores than shown.

What the model is actually computing

Articles 4 and 5 showed that a model is layers of weights, and that every layer combines its input numbers using those weights. That combining step is matrix multiplication, and it makes up the large majority of the arithmetic in an LLM. If you make matrix multiplication fast, you make the model fast.

Here is matrix multiplication at its smallest. A token is represented as a list of numbers (its vector, from Article 3), and a layer has a grid of weights. Take a token vector with three numbers and a layer that produces two outputs:

token vector:  [1, 2, 3]

weights:       output 1   output 2
               0.5        -1
               1           0
               0           2

Each output is the token's numbers multiplied by one column of weights, then added up:

output 1 = 1 × 0.5 + 2 × 1 + 3 × 0  = 0.5 + 2 + 0 = 2.5
output 2 = 1 × -1  + 2 × 0 + 3 × 2  = -1 + 0 + 6  = 5

result: [2.5, 5]

That took 6 multiplications and 4 additions. Now scale it up. A layer in a mid-sized model might take 4,096 numbers in and produce 4,096 out. That is 4,096 × 4,096, about 16.8 million multiply-and-add operations, for one token, in one of many matrices, in one of dozens of layers.

The key property is that every one of those multiplications is independent. Output 1 does not need to wait for output 2. That is exactly the kind of work a stadium of simple cores is good at.

Tensor cores and number formats

Modern data centre GPUs go further than thousands of general cores. They contain tensor cores: circuits dedicated to multiplying small blocks of numbers, for example a small grid by another small grid, in a single step. For LLM work they do most of the heavy lifting.

Tensor cores are fastest with lower-precision numbers. A standard "full precision" number takes 32 bits (4 bytes). LLMs usually run with 16-bit numbers (2 bytes), and increasingly with 8-bit or even 4-bit formats. Fewer bits mean less memory and faster arithmetic, at some cost in accuracy. Article 12 covers this trade-off under the name quantization.

VRAM: does the model fit?

A GPU has its own memory, called VRAM, separate from the server's normal RAM. For good performance the model's weights must sit in VRAM, because fetching them from anywhere else is far too slow.

Sizing is straightforward arithmetic:

memory for weights = number of parameters × bytes per parameter

• An 8 billion parameter model at 16 bits: 8 × 2 = 16 GB
• A 70 billion parameter model at 16 bits: 70 × 2 = 140 GB
• The same 70B model at 4 bits: 70 × 0.5 = 35 GB

Weights are not the only thing in VRAM. The KV cache from Article 8 grows with every token of context for every conversation being served. The diagram shows how memory might be divided on one GPU serving an 8B model. The split is illustrative.

Figure 9.2: VRAM holds the weights and the KV cache. Every number the tensor cores use must travel through the memory connection, marked in coral.

If the weights do not fit on one GPU, the model is split across several GPUs that work together, connected by very fast links. Article 10 covers how.

As of October 2026, published figures for some common GPUs look like this:

AMD and other vendors make competitive chips too. The table is only there to give a sense of scale.

Memory bandwidth: the real bottleneck

Memory bandwidth is how many bytes per second can move between VRAM and the compute units. It turns out to be the number that most limits how fast a model generates text.

Here is why. When generating one token (Article 8), the model must use every weight once. For a dense model serving one user, that means reading every weight from VRAM, for every single token. The arithmetic per weight is tiny: roughly two operations, a multiply and an add. So the compute units spend most of their time waiting for numbers to arrive.

A worked example: the speed limit of one token

Take an 8 billion parameter model at 16 bits, running on a GPU with 3,350 GB/s of bandwidth and around 1,000 trillion operations per second of 16-bit compute. Those are roughly the published figures for an H100 SXM.

Memory limit. Each token requires reading all the weights:

weights = 8 billion × 2 bytes = 16 GB per token
3,350 GB/s ÷ 16 GB = about 209 tokens per second, at most

Compute limit. Each token needs about 2 operations per parameter:

operations = 2 × 8 billion = 16 billion per token
1,000,000 billion per second ÷ 16 billion = about 62,500 tokens per second

The memory ceiling is around 300 times lower than the compute ceiling. For a single user, the GPU is mostly idle, waiting on memory. That is what people mean when they say generation is memory-bandwidth bound. These are simplified upper limits that ignore the KV cache and other overheads, so real speeds are lower, but the gap is real.

This also points to the fix. If 64 users are being served at once, the GPU can read each weight once and use it for all 64 tokens. The same memory traffic now does 64 times the useful work. That trick, batching, is the heart of Article 10.

The diagram traces one generation step and marks the slow part.

flowchart TD
    A["Weights stored in VRAM"]:::infra --> B["Read every weight"]:::warn
    B --> C["Tensor cores multiply"]:::infra
    C --> D["Next token"]:::model
    D -->|"repeat per token"| B
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 9.3: Generating each token means reading the whole model from memory. That read, in coral, is usually the limiting step.

Reading the prompt is different. All the prompt's tokens are known up front, so the GPU processes them together in one pass, reusing each weight for every prompt token. This phase, called prefill, is limited by compute rather than memory, and it is fast per token. The token-by-token phase is called decode. That is why a model can read a 5,000 token file in well under a second but takes several seconds to write a 300 token answer.

Where this shows up in our bug-fix agent

The agent harness itself runs on an ordinary CPU server. It reads files, runs pytest and calls the GitHub API, none of which needs a GPU. Every time it calls the model, though, the request lands on GPUs at the model provider or in your own cluster.

Consider one run on BUG-1423:

• Prefill. Each call sends the system prompt, the ticket, auth/login.py, tests/test_login.py and previous tool results. A few thousand tokens, processed in one fast compute-bound pass.
• Decode. Each call then generates a tool request or a short explanation, perhaps 100 to 300 tokens, one at a time, at a speed limited by memory bandwidth.

Suppose the run takes 15 model calls averaging 300 generated tokens each, and the model generates 60 tokens per second. These numbers are illustrative. That is 4,500 tokens ÷ 60 = 75 seconds of pure generation, before any time spent running tests. Most of the agent's thinking time is spent waiting on memory bandwidth.

Common misconceptions

"More compute power means faster answers." For generating text one token at a time, memory bandwidth usually matters more than compute. A GPU with more bandwidth but fewer operations per second can generate faster for a single user.

"If it fits in VRAM, it will run well." Fitting the weights is the first requirement, not the last. You also need room for the KV cache, which grows with context length and the number of users. A model that only just fits will handle very little traffic.

"GPUs are just faster CPUs." They are a different design, built for doing the same simple operation on huge amounts of data at once. They are poor at the branching, sequential work CPUs handle well, which is why the agent harness still runs on a CPU.

Key takeaways

• LLMs spend most of their time on matrix multiplication, which is made of many independent multiply-and-add operations.
• GPUs run thousands of simple cores in parallel, and tensor cores accelerate matrix multiplication further, especially at low precision.
• Weights must fit in VRAM: parameters × bytes per parameter, plus room for the KV cache.
• Generating one token requires reading every weight, so decode speed is limited by memory bandwidth, not compute.
• Prefill (reading the prompt) is compute-bound and fast. Decode (writing the answer) is bandwidth-bound and slower.

Glossary

• GPU (graphics processing unit): a chip with thousands of simple cores designed for parallel arithmetic.
• Core: an individual processing unit inside a CPU or GPU.
• Matrix multiplication: combining a list of numbers with a grid of weights through many multiply-and-add operations.
• Tensor core: a GPU circuit specialised for multiplying small blocks of numbers in one step.
• Precision: how many bits are used to store each number, such as 32, 16, 8 or 4.
• VRAM: the GPU's own high-speed memory, where the model's weights and KV cache live.
• Memory bandwidth: how many bytes per second can move between VRAM and the compute units.
• Memory-bandwidth bound: limited by how fast data can be moved, not by how fast it can be processed.
• Prefill: processing all the prompt tokens in one pass before generation starts.
• Decode: generating output tokens one at a time.
• Batching: serving several requests at once so each weight read is reused for many tokens.

Next up

One user per GPU leaves almost all of its power unused. In Article 10: Inference at scale, we look at how providers serve thousands of users at once: batching, the trade-off between latency and throughput, mixture-of-experts models, and serving software such as vLLM.