A model fresh out of pretraining is strangely unhelpful. Paste in our bug ticket and it might continue with a second bug ticket, because that is what often follows one in a public issue tracker. The assistants and agents you actually use went through further training that taught them to treat text as instructions and to respond in a useful way. Knowing how that works explains why assistants follow some instructions and ignore others, why they refuse certain requests, and what the system prompt in your app really does.

The problem with a base model

Article 6 ended with a base model: a model that has learned to predict the next token across trillions of tokens of text. It knows a great deal, but it has no idea that it is supposed to help anyone.

Give a base model this:

Fix the login bug where uppercase emails are rejected.

A plausible continuation, from the model's point of view, might be:

Fix the signup bug where long passwords are rejected.
Fix the reset bug where the link expires too early.

It has seen many lists of tasks like this, so it continues the list. That is correct next-token prediction, and completely useless to us. What we want is a model whose most likely continuation of an instruction is a helpful response to it.

The good news is that we do not need to start over. The base model already knows Python, knows what login code looks like and knows what a good explanation reads like. We just need to shift which continuations it considers most likely. That takes far less data and compute than pretraining.

The diagram shows the usual stages from base model to assistant.

flowchart TD
    A["Base model"]:::model -->|"+ example conversations"| B["Instruction tuning"]:::model
    B -->|"+ ranked answers"| C["Preference tuning"]:::model
    C --> D["Assistant model"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 7.1: Post-training. Each stage starts from the previous model and adjusts its weights with a different kind of data.

This is a simplification. Real recipes vary between labs, often repeat stages, and mix in other techniques. But almost every assistant model goes through some version of these two steps, collectively called post-training.

Stage 1: instruction tuning

Instruction tuning, also called supervised fine-tuning or SFT, trains the model on example conversations: an instruction, followed by the response we would like.

The training method is the same next-token prediction from Article 6. The only difference is the data. Instead of random web pages, the model sees thousands to millions of examples like:

User: Why might login fail for Maria@Example.com but work for maria@example.com?
Assistant: The lookup is probably case-sensitive. If emails are stored in
lowercase, normalise the input with email.strip().lower() before querying.

Usually the loss is only counted on the assistant's part. The model is not being taught to predict user questions, only to predict good answers given a question.

After enough examples, the most likely continuation of a question becomes an answer. The model has learned the format of being an assistant: answer the question asked, stop when done, follow formatting requests. The examples are written or reviewed by people, generated by other models, or a mix of both.

Stage 2: preference tuning

Instruction tuning teaches the format, but writing perfect example answers for every situation is slow and expensive. It is often easier for a person to compare two answers than to write a perfect one. That is the idea behind preference tuning.

The process looks like this:

  1. Give the model a prompt and generate two or more different responses.
  2. A person, or another model, picks which response is better.
  3. Adjust the model so that preferred responses become more likely and rejected ones less likely.

The best known version is RLHF, reinforcement learning from human feedback. It first trains a separate reward model that learns to score responses the way the human raters would. The assistant model then generates responses, the reward model scores them, and the assistant is updated to produce higher-scoring answers.

There are several popular alternatives that reach a similar goal:

  • DPO (direct preference optimisation) skips the separate reward model and updates the assistant directly from pairs of preferred and rejected answers. It is simpler to run.
  • RLAIF, reinforcement learning from AI feedback, uses a model instead of people to judge responses. Anthropic's Constitutional AI is one example: the judging model is guided by a written list of principles.
  • Rewards from checkable results. For tasks like maths and coding, the reward can come from an automatic check, for example "did the tests pass?", instead of a judge's opinion. This is especially relevant for coding agents.

The diagram shows one round of preference tuning, using our bug ticket.

flowchart TD
    P["Prompt: fix BUG-1423"]:::io --> M["Model writes two answers"]:::model
    M --> J["Judge picks the better one"]:::io
    J --> U["Update weights"]:::model
    U -->|"preferred answer now more likely"| M
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 7.2: Preference tuning. The judge can be a person, another model following written principles, or an automatic check such as a test suite.

Preference tuning is where much of an assistant's "personality" comes from: how cautious it is, how long its answers are, when it asks a clarifying question, and what it refuses to do. It is also where things can go wrong. If raters tend to prefer confident, long answers, the model learns to sound confident and write at length, whether or not it is right.

A worked example: one preference pair

Let's trace a single training example through preference tuning. The prompt is our ticket:

Fix BUG-1423: Login fails when the email has uppercase letters.

The model produces two candidate answers:

  • Answer A: "Add email = email.strip().lower() as the first line of login() in auth/login.py, and add a test that logs in with Maria@Example.com."
  • Answer B: "The authentication module is poorly designed. I recommend rewriting it using a new framework."

A reviewer prefers A. It is small, targeted, and testable. B might even contain good ideas, but it ignores the actual ticket.

Suppose that before this training step, the model's probabilities for starting each answer were 0.30 for A and 0.20 for B. These numbers are illustrative. After the update, A might move up to 0.33 and B down to 0.17. One example makes a tiny change. Across tens or hundreds of thousands of comparisons, the model shifts steadily towards small, focused fixes. That is why a well tuned coding model suggests a one-line fix rather than a rewrite.

Chat templates: how a conversation becomes tokens

The model still only reads a single sequence of tokens. So how does it know which parts are instructions from the app, which parts are the user, and which parts it wrote itself?

The answer is a chat template: a fixed format that marks each turn of the conversation with special tokens. These tokens never appear in normal text, and the model learned their meaning during post-training.

The diagram shows a simplified chat template for the start of our agent's conversation. The exact markers are made up for illustration. Each model family uses its own.

‹system› You are a coding agent. ‹user› Fix BUG-1423. ‹assistant› I'll search for login… All one sequence of tokens. The model generates the highlighted part.

Figure 7.3: A chat template. Role markers are special tokens the model learned during post-training. The conversation is still one long sequence.

When an API receives a list of messages from your code, the provider applies the template before the model sees anything. If you run an open-weight model yourself (Article 12), getting the template exactly right matters. A wrong template is one of the most common reasons a self-hosted model gives strange answers.

What a system prompt really is

The system prompt is the first message in the conversation, written by the developer of the app rather than the end user. It sets the rules: the model's role, the tools it has, the style of answer, what to avoid.

Mechanically, it is just text at the start of the sequence, marked with the system role. It works because post-training taught the model to give system instructions high priority. Many labs explicitly train their models to follow the system prompt over conflicting user messages.

That priority is learned behaviour, not a hard rule. A system prompt is not a security boundary. A determined user, or malicious text hidden in a file the agent reads, can sometimes talk the model out of its instructions. Article 19 covers this, under the name prompt injection.

Where this shows up in our bug-fix agent

Our agent depends on post-training at every step:

  • Following the system prompt. The harness starts every run with a system prompt along the lines of: "You are a coding agent working in this repository. Make the smallest change that fixes the ticket. Always add a test. Use the provided tools." Post-training is why the model takes this seriously.
  • Using tools in the right format. The model needs to output tool requests in an exact format the harness can parse. That skill comes from post-training on examples of tool use (Article 16).
  • Preferring small fixes. The one-line email.strip().lower() fix, instead of a rewrite, reflects preference tuning that favoured focused changes.
  • Knowing when to stop. A base model would keep generating forever. An assistant model learned to produce an end-of-turn token once it has said what it needs to.

Coding models in particular are often trained with rewards from running real tests, so "the tests pass" is literally part of what they were optimised for.

Common misconceptions

"Post-training teaches the model new knowledge." Mostly it changes behaviour, not knowledge. The facts and coding skills come from pretraining. Post-training decides how that knowledge is presented and when it is used. This is also why fine-tuning is usually a poor way to add new facts (Article 15).

"The system prompt is enforced by the software." It is just text the model was trained to prioritise. It shapes behaviour strongly, but it can be overridden. Anything that must never happen needs a real control outside the model, such as permissions in the harness.

"RLHF makes the model truthful." RLHF makes the model produce answers that raters prefer. Raters prefer correct answers, but they also tend to prefer confident, well-formatted ones, and they cannot always tell which answer is right. Truthfulness improves, but it is not guaranteed.

Key takeaways

  • A base model continues text. Post-training turns it into an assistant that responds to instructions.
  • Instruction tuning trains on example conversations, using the same next-token loss as pretraining.
  • Preference tuning (RLHF, DPO, AI feedback, or automatic checks like tests) shifts the model towards answers a judge prefers.
  • Chat templates mark each turn with special tokens, so a conversation becomes one token sequence the model understands.
  • A system prompt is high-priority text, not a security control.

Glossary

  • Post-training: all training after pretraining that shapes how the model behaves, including instruction and preference tuning.
  • Instruction tuning (supervised fine-tuning, SFT): training on example instructions paired with good responses.
  • Preference tuning: training that makes preferred responses more likely and rejected ones less likely.
  • RLHF (reinforcement learning from human feedback): preference tuning where a reward model trained on human comparisons scores the model's outputs.
  • Reward model: a model trained to score responses the way human or AI judges would.
  • DPO (direct preference optimisation): a preference tuning method that learns directly from pairs of preferred and rejected answers, without a separate reward model.
  • RLAIF: preference tuning where another model provides the feedback instead of people.
  • Constitutional AI: an approach from Anthropic where AI feedback is guided by a written set of principles.
  • Chat template: the format, with special role tokens, that turns a list of messages into a single token sequence.
  • System prompt: developer-written instructions placed at the start of a conversation.

Next up

We now have an assistant that wants to help. But every time it produces a token, it has to choose one from a list of probabilities. In Article 8: Generation and sampling, we look at exactly how that choice is made, what temperature does with real numbers, and why your answer appears on screen word by word.