When an AI coding agent fixes the wrong function or ignores a comment that holds the answer, the cause is often what was, or wasn't, in its context. The mechanism that decides how a model uses its context is called attention, and it is the core of the transformer, the design behind every modern LLM. Understanding it explains why context is so powerful, and why long contexts get expensive.

The problem attention solves

Look at the line email = email.strip(). from our bug fix. To predict lower next, the model needs information from far away in the text: the comment # emails are stored in lowercase a few lines up, and the ticket that mentions uppercase letters even earlier.

In Article 4 we saw that a layer of neurons processes a vector. If each token's vector went through the layers on its own, the token . at the end of strip(). would know nothing about the comment or the ticket. Older designs passed information along the sequence one step at a time, so details from far back tended to fade.

Attention fixes this directly. At every layer, each token can look at every earlier token in the text and pull in whatever information is relevant to it. Distance doesn't matter. A comment 30 lines up is as reachable as the previous word.

The transformer in one picture

The transformer was introduced in 2017 in a paper by researchers at Google titled "Attention Is All You Need". Almost every LLM since then follows its basic layout:

  1. Token IDs are turned into vectors with the embedding table (Article 3).
  2. Information about each token's position is added, so the model knows word order.
  3. The vectors pass through a stack of identical transformer blocks. Each block has two parts: an attention step, where tokens share information, and a feed-forward step, where each token's vector goes through layers of neurons like the ones in Article 4.
  4. After the last block, the model produces scores for every possible next token (Article 1).

The diagram shows the stack of transformer blocks.

flowchart TD
    A["Token IDs"]:::io --> B["Embeddings plus position"]:::model
    B --> C["Attention"]:::model
    C --> D["Feed-forward layers"]:::model
    D -->|"next block, dozens of times"| C
    D --> E["Next-token scores"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 5.1: A transformer repeats the same block, attention then feed-forward, dozens of times before scoring the next token.

A useful way to think about the split: attention moves information between tokens, and the feed-forward layers process information within each token. Large models repeat this pair dozens of times, and each repetition builds a richer understanding of the text.

How attention works: queries, keys and values

At each attention step, every token produces three vectors from its current vector, using weights learned in training:

  • a query: what this token is looking for
  • a key: what this token offers, a label others can match against
  • a value: the information this token will hand over if someone attends to it

Think of a library. Your query is the question you walk in with. Each book's key is the title on its spine. The value is what's inside the book. You compare your question with every spine, decide how relevant each book is, and read mostly from the most relevant ones.

For each token, attention does three things:

  1. Score. Compare its query with every earlier token's key using a dot product (Article 3). A higher score means more relevant.
  2. Normalise. Turn the scores into weights that are all positive and add up to 1, using a function called softmax.
  3. Mix. Take a weighted average of the earlier tokens' values. That mix becomes the token's new information.

In generation models, a token can only attend to tokens before it, never after. This is called causal masking. It makes sense: when predicting the next token, the future doesn't exist yet.

A worked example: one attention step by hand

Let's compute attention for the position right after email = email.strip()., where the model is about to predict the next token. To keep the arithmetic small, I'm using two-number vectors and only three earlier tokens. Real models use vectors with thousands of numbers and attend over every earlier token. I'm also leaving out a scaling step that real models apply to the scores. The numbers are illustrative.

The current position's query: q = [1, 1].

The keys of three earlier tokens:

  • lowercase (from the comment): key = [2, 1]
  • email: key = [1, 0]
  • password: key = [0, 0]

Step 1: score each token with a dot product.

  • lowercase: (1 × 2) + (1 × 1) = 3
  • email: (1 × 1) + (1 × 0) = 1
  • password: (1 × 0) + (1 × 0) = 0

Step 2: normalise with softmax. Softmax raises the number e (about 2.718) to the power of each score, then divides each result by the total. Here are the values:

  • e³ ≈ 20.09, e¹ ≈ 2.72, e⁰ = 1
  • Total: 20.09 + 2.72 + 1 = 23.81
  • lowercase: 20.09 ÷ 23.81 ≈ 0.84
  • email: 2.72 ÷ 23.81 ≈ 0.11
  • password: 1 ÷ 23.81 ≈ 0.04

The weights add up to about 1 (0.99 here because of rounding). Notice how softmax exaggerates differences. A score of 3 against 1 became a weight of 0.84 against 0.11.

Step 3: mix the values. Say the values are lowercase = [1, 0], email = [0, 1] and password = [0, 0]. The new information for our position is:

  • 0.84 × [1, 0] + 0.11 × [0, 1] + 0.04 × [0, 0]
  • = [0.84, 0] + [0, 0.11] + [0, 0]
  • = [0.84, 0.11]

The result is mostly the information carried by lowercase. In plain terms, at this position the model has "looked back" at the comment and brought its meaning forward, which is exactly what it needs to predict lower.

The diagram shows the same attention weights as lines from the current position to earlier tokens.

lowercase email password .strip(). ? 0.84 0.11 0.04

Figure 5.2: Illustrative attention weights. The position after ".strip()." draws most of its information from "lowercase" in the comment.

Many heads, many layers

One attention calculation can only look for one kind of relevance at a time. So each attention step actually runs many of them side by side, called attention heads, each with its own query, key and value weights. One head might learn to link variable names to where they were defined, another to match brackets, another to connect a ticket's wording to code. I'm simplifying, since heads rarely have such neat jobs, but they do specialise.

And because the transformer repeats its block dozens of times, attention is applied again and again. Early layers tend to capture simple things like syntax and nearby words. Later layers combine those into higher-level meaning. The output of the last layer is what produces the next-token scores.

Why context matters

Here is the practical consequence. The model can only attend to what is in its context. The context is everything in the current input: the system prompt, the conversation so far, any files or documents included, and the tokens generated so far. Its maximum size is the context window, measured in tokens.

Anything outside the context is invisible, apart from what was learned into the weights during training. If the comment # emails are stored in lowercase is not in the context, the model cannot attend to it. It might still guess the fix from general experience, but it is guessing.

Context also has a cost. Every token attends to every earlier token, so the number of comparisons grows with the square of the length:

  • 1,000 tokens: about 1,000 × 1,000 = 1 million comparisons per head per layer
  • 100,000 tokens: about 100,000 × 100,000 = 10 billion

Causal masking roughly halves these numbers, and engineering tricks make attention much faster in practice, but the growth pattern remains. A 100 times longer input means about 10,000 times more attention work. That is why long inputs are slower and why providers often charge more for very long prompts. As of October 2026, many leading models accept context windows of several hundred thousand tokens, and some accept a million or more. Article 13 covers how to use that space well.

Where this shows up in our bug-fix agent

For the bug-fix agent, the context window is the model's entire working view of the world. The harness decides what goes into it.

The diagram shows how the agent harness decides what the model can attend to.

flowchart TD
    T["Ticket BUG-1423"]:::io --> H["Harness picks files"]:::infra
    H -->|"login.py, test file"| W["Context window"]:::model
    H -.->|"rest of the repo"| X["Never seen by model"]:::warn
    W --> M["Attention over context"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 5.3: The model can only attend to what the harness puts in the context. Files left out are invisible to it.

A good harness includes the ticket, auth/login.py with its comment about lowercase storage, and the relevant test file. Attention then links the word "uppercase" in the ticket, the comment in the code and the find_user_by_email call, and the fix follows naturally.

A poor harness either leaves out auth/login.py, so the model can't see the bug, or stuffs in the entire repository, which is slow, expensive, and buries the important lines among thousands of irrelevant ones. Choosing context well is a large part of what makes one coding agent better than another. Articles 13, 14 and 16 return to this from different angles.

Common misconceptions

"The model reads the context once, from start to finish, like a person." Every token attends to all earlier tokens at once, at every layer. There is no single reading pass. In a sense, the model re-examines the whole context for every token it processes.

"A bigger context window means the model uses all of it equally well." Fitting a long document in the window does not guarantee the model pays good attention to every part. Quality on very long inputs varies between models, so test with your own data rather than trusting the maximum size.

"Attention means the model is focusing like a human does." The word is a loose metaphor. Attention is a weighted average computed from dot products and softmax. It is useful to think of it as focus, but it is arithmetic.

Key takeaways

  • The transformer is the design behind modern LLMs: embeddings and position information, then dozens of blocks of attention and feed-forward layers, then next-token scores.
  • Attention lets each token pull in information from any earlier token, using queries, keys and values, scored with dot products and normalised with softmax.
  • Many attention heads run in parallel and the blocks repeat, so the model builds up meaning layer by layer.
  • The model can only attend to what is in its context window. Anything else is invisible, apart from what it learned in training.
  • Attention work grows with the square of the context length, which is why long inputs are slower and more expensive.

Glossary

  • Transformer: the neural network design used by modern LLMs, built from repeated blocks of attention and feed-forward layers.
  • Transformer block: one attention step followed by one feed-forward step. Models stack dozens of them.
  • Attention: the mechanism that lets each token gather information from other tokens, weighted by relevance.
  • Query, key, value: three vectors each token produces for attention: what it looks for, what it offers as a label, and the information it passes on.
  • Softmax: a function that turns a list of scores into positive weights that add up to 1, exaggerating the differences between them.
  • Attention head: one of several attention calculations that run side by side, each learning to look for different relationships.
  • Causal masking: the rule that each token can only attend to tokens before it.
  • Feed-forward layers: the layers of neurons in each block that process each token's vector on its own.
  • Context: everything in the model's current input, including prompts, files and the tokens generated so far.

Next up

We have now seen the full machine: tokens, embeddings, neurons and attention. But all those billions of weights started as random numbers. In Article 6: Training, we look at how a model learns, walk through gradient descent with a small numeric example, and see why training a large model costs millions.