Training

Every capability we have described so far, from predicting the next token to linking email with .lower(), comes from one process: training. Nobody writes these skills into the model. They are the side effect of adjusting billions of numbers, a tiny bit at a time, until the model's predictions stop being wrong. Understanding training explains why models know what they know, why they have a knowledge cutoff, and why only a handful of companies can afford to build the largest ones.

Starting from random numbers

In Article 4 we saw that a model is a huge collection of weights: numbers that decide how inputs are combined into outputs. Before training, those weights are set to small random values. A freshly created model is not stupid in an interesting way. It is pure noise. Ask it to continue email = email.strip(). and every token in its vocabulary gets roughly the same probability.

Training is the process of changing those weights so that the model's predictions match real text. It happens in stages. This article covers the first and largest stage, pretraining. Article 7 covers the later stages that turn a pretrained model into an assistant.

Pretraining: predict the next token, trillions of times

Pretraining uses the same job we met in Article 1. Take a piece of real text, hide the next token, ask the model to predict it, and check the answer.

The beauty of this setup is that the answers come for free. Any text is a training example. A sentence from a manual, a Python file from a public repository, a forum post: for every position in that text, the "correct answer" is simply the token that actually comes next. No human has to label anything.

The pipeline before training is mostly about data.

The diagram shows how raw text becomes a pretrained base model.

flowchart TD
    A["Web pages, books, code"]:::io --> B["Filter and deduplicate"]:::infra
    B --> C["Tokenize"]:::model
    C --> D["Pretraining on GPUs"]:::infra
    D --> E["Base model"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 6.1: The pretraining pipeline. Most of the engineering effort goes into the data steps before the GPUs ever start.

Filtering matters a lot. Training sets are cleaned to remove spam, duplicates, low quality pages and some kinds of personal data. The quality and mix of the data, for example how much code versus prose, shapes what the model ends up good at.

This also explains the knowledge cutoff. The model can only learn from text collected before training started. Anything that happened later simply is not in its weights.

Measuring wrongness: the loss

To improve the model, we need a number that says how wrong it was. That number is called the loss. Lower is better.

For next-token prediction, the standard loss looks at one thing: the probability the model gave to the token that actually came next. If the model was confident and right, the loss is small. If it gave the correct token a tiny probability, the loss is large.

The formula is the negative logarithm of that probability. You do not need to calculate logarithms by hand. What matters is how the numbers behave:

If the training text says .strip().lower() and the model gave lower a probability of 0.9, it barely gets penalised. If it gave lower 0.01 because it was betting on upper, it gets a big penalty. Averaged over billions of positions, this single number tells us how good the model is at predicting text.

Gradient descent: which way to nudge each weight

Now the key question. We have a loss, and we have billions of weights. Which weights should change, and by how much?

The answer is gradient descent. For each weight, work out how the loss would change if that weight went up a tiny bit. That rate of change is the weight's gradient. If increasing a weight makes the loss go up, decrease it. If increasing it makes the loss go down, increase it. Then repeat.

A useful picture is standing on a hillside in fog. You cannot see the valley, but you can feel which way the ground slopes under your feet. Take a small step downhill, feel again, take another step. The learning rate is the size of your step.

Real models compute every gradient at once using an algorithm called backpropagation, which is calculus applied efficiently through all the layers. I am skipping how it works internally. The important part is what it produces: one gradient per weight, saying which way to nudge it.

A worked example: training a one-weight model

Let's train the smallest possible model by hand. It has one weight, w, and it predicts an output by multiplying the input by w:

prediction = w × input

We have one training example: input 2, correct answer 6. The perfect weight is obviously 3, but the model does not know that. We start it at w = 1.

For the loss we use the squared error, which is easier to work with by hand than the next-token loss. The idea is identical: a number that is zero when the answer is right and grows as it gets more wrong.

loss = (prediction - correct)²

Step 1: measure. Prediction = 1 × 2 = 2. Error = 2 - 6 = -4. Loss = (-4)² = 16.

Step 2: find the slope. Nudge the weight up by 0.01 and see what happens. With w = 1.01, prediction = 2.02, error = -3.98, loss = 15.8404. The loss dropped by 0.1596 for a 0.01 increase, so the slope is about -0.1596 ÷ 0.01 = -16. Negative slope means increasing w reduces the loss. (Calculus gives the exact slope with a shortcut: 2 × error × input = 2 × -4 × 2 = -16. Same answer.)

Step 3: step downhill. With a learning rate of 0.05, the update is:

new w = w - learning_rate × slope
      = 1 - 0.05 × (-16)
      = 1 + 0.8 = 1.8

Step 4: repeat.

Each step moves w closer to 3 and the loss shrinks. Notice that the steps get smaller as we approach the bottom, because the slope flattens out.

The chart plots the loss for every possible value of the weight, with our four steps sliding down towards the minimum at 3.

Figure 6.2: Gradient descent on one weight. Each step follows the slope downhill, and the steps shrink as the curve flattens near the bottom.

A real LLM does exactly this, with three differences. It has billions of weights instead of one, so the "hillside" has billions of directions. It uses the next-token loss instead of squared error. And it never processes one example at a time. It processes a batch of many sequences, averages their gradients, and takes one step. One pass through a batch and one update is a training step. A full pretraining run is hundreds of thousands of steps or more.

The diagram shows the loop that repeats for every training step.

flowchart TD
    A["Batch of real text"]:::io --> B["Model predicts next tokens"]:::model
    B --> C["Compare with actual tokens"]:::model
    C --> D["Loss: how wrong"]:::warn
    D --> E["Gradients for every weight"]:::model
    E --> F["Nudge weights slightly"]:::model
    F -->|"next batch"| A
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 6.3: One training step. The whole of pretraining is this loop, repeated hundreds of thousands of times across thousands of GPUs.

Why training costs millions

The cost comes from simple multiplication. A widely used rule of thumb says training takes about 6 calculations per parameter per training token. This is an approximation, but a good one for standard models.

Let's work through an illustrative example: a 70 billion parameter model trained on 15 trillion tokens.

6 × 70,000,000,000 × 15,000,000,000,000
= 6,300,000,000,000,000,000,000,000 calculations
= 6.3 × 10²⁴

Suppose each GPU actually achieves 400 trillion calculations per second in practice. That figure is illustrative. Real numbers depend on the hardware and how well the run is tuned.

6.3 × 10²⁴ ÷ 400,000,000,000,000 = 15.75 billion GPU-seconds
15.75 billion ÷ 3,600 ≈ 4.4 million GPU-hours

At an illustrative price of $2 per GPU-hour, that single run is roughly $8.75 million in compute alone. Spread over 10,000 GPUs it would take about 18 days.

For a real reference point, Meta's published model card for Llama 3.1 reports about 30.8 million H100 GPU-hours for its largest, 405 billion parameter model.

And the final run is only part of the bill:

• Failed and experimental runs. Teams train many smaller models to test data mixes and settings before committing to the big one. Runs crash and have to restart from saved checkpoints.
• The cluster itself. Thousands of GPUs, fast networking between them, power and cooling.
• Data and people. Collecting, cleaning and licensing data, plus the research and engineering staff.

That is why pretraining frontier models is limited to a small number of well-funded labs, while most companies build on top of existing models instead. Articles 14 and 15 cover how.

Where this shows up in our bug-fix agent

When our agent writes email = email.strip().lower(), it is using knowledge it picked up during pretraining. Somewhere in its training data were thousands of codebases that normalise email addresses before looking them up. Every time the model predicted upper or strip where the real code said lower, the loss went up and gradient descent nudged its weights. After enough of those nudges, the pattern "emails get lowercased before lookup" is encoded across many weights.

Two practical consequences for the agent:

• It only knows libraries that existed before its cutoff. If our login code uses a framework released last month, the model may guess at its API. The harness has to give it the real documentation or code in the prompt, which Article 13 covers.
• Its weights do not change while it works. The agent does not "learn" from fixing BUG-1423. Training and using a model are separate phases. If the same bug appears tomorrow, the model starts from the same weights. Anything it should remember has to be stored outside the model and fed back in.

Common misconceptions

"The model learns from my conversations as I use it." Using a model, called inference, does not change its weights. Some providers may use conversation data to train future versions, depending on their terms and your settings, but that is a separate, later training run.

"Training means storing the training data." The model does not keep a copy of its training text. It keeps weights shaped by that text. That said, models can memorise passages that appeared many times, which is one reason data deduplication matters and why data leakage is a real concern (Article 19).

"More training always makes it better." Training too long on the same data can lead to overfitting, where the model gets very good at its training text but worse at new text. Teams watch the loss on held-out data the model never trains on to catch this.

Key takeaways

• A model starts as random weights. Pretraining adjusts them so the model predicts real text well.
• The loss is a single number for how wrong the predictions were. For LLMs it is based on the probability given to the correct next token.
• Gradient descent nudges every weight in the direction that lowers the loss, one small step at a time, repeated across huge numbers of batches.
• Training compute grows with parameters times tokens, roughly 6 calculations per parameter per token, which is why large runs cost millions.
• Training and use are separate. A deployed model's weights do not change as it works.

Glossary

• Pretraining: the first and largest training stage, where a model learns to predict the next token on huge amounts of text.
• Loss: a number measuring how wrong the model's predictions are. Training tries to make it smaller.
• Gradient: for one weight, how much the loss changes if that weight increases slightly. It tells you which way to nudge the weight.
• Gradient descent: repeatedly nudging weights against their gradients to reduce the loss.
• Learning rate: how big each nudge is.
• Backpropagation: the algorithm that efficiently computes the gradient for every weight in a network.
• Batch: a group of training examples processed together before one weight update.
• Training step: one batch processed and one weight update applied.
• Knowledge cutoff: the point after which no training data was collected, so the model knows nothing later.
• Inference: using a trained model to produce outputs, without changing its weights.
• Overfitting: when a model fits its training data so closely that it gets worse on new data.

Next up

Pretraining gives us a base model that is brilliant at continuing text but useless at following instructions. In Article 7: From raw model to assistant, we look at the extra training that teaches a model to answer questions, follow a system prompt and behave like the helpful assistant inside our bug-fix agent.