"Can we train it on our data?" is one of the first questions every team asks about LLMs. Sometimes the answer is yes and fine-tuning is exactly right. More often, the real problem is better solved with a good prompt or with retrieval, which are cheaper, faster to change, and easier to debug.

What fine-tuning actually does

Article 6 described pretraining: showing a model trillions of tokens and nudging its parameters after every wrong prediction. Article 7 described how that raw model becomes an assistant through further training on examples of good behaviour.

Fine-tuning is the same process again, on a much smaller scale, with your own examples. You collect pairs of "here is an input" and "here is the output we want", run training on them, and the model's parameters shift so that it produces outputs more like yours.

The key word is behaviour. Fine-tuning is good at changing how a model responds: its format, tone, style, which conventions it follows, how it handles a narrow task. It is much less good at reliably adding facts, especially facts that change.

Prompting, RAG or fine-tuning?

There are three main levers for adapting a model, and they solve different problems.

  • Prompting (Article 13) changes the instructions. It is instant, free to iterate, and should always be tried first.
  • RAG (Article 14) changes what the model reads. It is the right tool when the model is missing knowledge, especially knowledge that changes often or must be cited.
  • Fine-tuning changes the model itself. It is the right tool when the model has the knowledge it needs but keeps doing the task the wrong way, even with a good prompt.

The diagram shows a simple decision path for choosing between them.

flowchart TD
    A["Model gets it wrong"]:::io --> B["Missing knowledge?"]:::warn
    B -->|"yes"| C["Use RAG"]:::infra
    B -->|"no"| D["Improve the prompt"]:::infra
    D --> E["Still wrong?"]:::warn
    E -->|"yes"| G["Consider fine-tuning"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 15.1: Try the prompt first, use RAG for missing knowledge, and fine-tune only when behaviour stays wrong.

Good reasons to fine-tune:

  • Consistent format. The output must always follow a strict structure, and long prompt instructions are not enough.
  • A narrow, repeated task. Classifying support tickets into your 40 categories, thousands of times a day.
  • Shorter prompts. If every request needs two pages of examples to behave, baking that behaviour in removes those tokens from every call.
  • A smaller, cheaper model. Training a small model on outputs from a large one, called distillation, can give you most of the quality at a fraction of the cost for one specific task.

Poor reasons to fine-tune:

  • "So it knows our documentation." Documentation changes. Use RAG, so updates are a re-index, not a retrain.
  • "So it stops making things up." Fine-tuning on facts does not reliably stop hallucination and can make the model confidently wrong in new ways.
  • "Because it feels more serious than prompting." Many teams fine-tune and then discover a better prompt matched the result.

Full fine-tuning vs LoRA

The straightforward approach is full fine-tuning: update every parameter in the model. For a model with 7 billion parameters, every one of those 7 billion numbers can change.

That is expensive mainly because of memory. During training, the GPU must hold not only the weights but also a gradient for each weight (how it should change) and extra bookkeeping numbers used by the optimiser. A common rule of thumb for standard mixed-precision training is around 16 bytes per parameter. I am simplifying, because exact numbers depend on the setup, but the scale is right.

For 7 billion parameters: 7,000,000,000 × 16 bytes = 112,000,000,000 bytes, about 112 GB. That needs several data-centre GPUs (Article 9) just to start.

LoRA (low-rank adaptation) takes a cleverer route. It freezes all the original weights and trains a small add-on instead. The insight is that the change a fine-tune needs to make is usually simple enough to describe with far fewer numbers than the weights themselves.

Worked example: counting LoRA parameters

Article 4 showed that a model's weights are organised into large grids of numbers called matrices. Take one weight matrix of size 4,096 × 4,096, a typical size inside a 7-billion-parameter model.

Full fine-tuning of this one matrix updates every number in it:

  • 4,096 × 4,096 = 16,777,216 parameters

LoRA leaves that matrix frozen and adds two thin matrices next to it. One, called A, is 4,096 × 8. The other, B, is 8 × 4,096. The 8 is called the rank, and it is a setting you choose. Multiplying A by B produces a full 4,096 × 4,096 adjustment, but you only store and train the two thin pieces:

  • A: 4,096 × 8 = 32,768
  • B: 8 × 4,096 = 32,768
  • Total: 32,768 + 32,768 = 65,536 parameters

That is 65,536 ÷ 16,777,216 ≈ 0.0039, or about 0.4% of the original matrix.

Now scale it up. Suppose the model has 32 layers and we attach LoRA to 4 matrices in each layer:

  • 32 × 4 × 65,536 = 8,388,608 trainable parameters, about 8.4 million.
  • Compared with 7 billion: 8,388,608 ÷ 7,000,000,000 ≈ 0.0012, roughly 0.12%.

This is a simplification. Real models have matrices of several different shapes, and you can attach LoRA to more or fewer of them. But the order of magnitude holds: you train around a tenth of a percent of the parameters.

The memory picture changes completely. The frozen 7 billion weights still need to sit in GPU memory, at 2 bytes each that is 7,000,000,000 × 2 = 14 GB, but gradients and optimiser state are only needed for the 8.4 million LoRA parameters. A variant called QLoRA stores the frozen weights in 4-bit form (Article 12 explains quantization), which cuts the 14 GB to roughly 7,000,000,000 × 0.5 bytes = 3.5 GB. That fits on a single consumer GPU.

The diagram shows LoRA's shape: one large frozen matrix and two thin trainable ones beside it.

W, frozen 4,096 × 4,096 16.8M numbers + A B trained 65,536 total sizes not to scale

Figure 15.2: LoRA trains two thin matrices, A and B, and leaves the large weight matrix untouched.

A practical bonus: the trained LoRA weights, called an adapter, are small files, often tens of megabytes. You can keep one base model and swap adapters for different tasks.

What it costs

Costs come in three parts.

Compute. For LoRA on a mid-sized open-weight model with a few thousand examples, training can take hours on a single GPU. As an illustration only, if a rented GPU cost $2 per hour and training took 6 hours, compute would be 2 × 6 = $12. Real GPU rental prices vary widely by provider and GPU type and change often. Many model providers also offer fine-tuning as a paid API, usually priced per training token, so check their current price list before planning.

Data. This is usually the real cost. You need hundreds to thousands of high-quality examples, and someone has to write, collect or check them. Bad examples teach bad behaviour very efficiently.

Ownership. A fine-tuned model is a thing you now maintain. When a better base model is released, your fine-tune does not transfer. You re-run training, re-check quality, and redeploy.

Where this shows up in our bug-fix agent

Should the bug-fix agent be fine-tuned on the company's codebase? Mostly, no.

The code it needs to read changes every day. If auth/login.py was learned during a fine-tune last month, the model's memory of it is already out of date. The agent should read the current file through tools and retrieval (Articles 14 and 16), every time.

Fine-tuning can help with behaviour, though. Suppose the team has years of merged pull requests and wants every agent PR to follow house conventions: always add a test in tests/, use the team's commit message format, title PRs like "Normalise email case on login (BUG-1423)". A LoRA adapter trained on past ticket-to-PR examples could make the agent follow those habits more consistently than prompt instructions alone.

Even then, the sensible order is the decision path in Figure 15.1: first put the conventions in the system prompt with two good examples. Fine-tune only if that keeps failing.

Common misconceptions

"Fine-tuning is how you teach a model your company's knowledge." It is a poor way to add facts. Knowledge that changes or must be accurate belongs in retrieval, where you can update it instantly and see exactly what the model was given.

"You need a huge dataset." For behaviour changes with LoRA, a few hundred to a few thousand good examples can be enough. Quality matters far more than quantity.

"Fine-tuning makes the model smarter." It makes it more specialised. A model fine-tuned hard on one task can get worse at others, an effect called catastrophic forgetting. LoRA reduces this risk because the original weights stay frozen.

"Once fine-tuned, you're done." Every new base model release, every change in your conventions, and every drift in your data means retraining and re-evaluating.

Key takeaways

  • Fine-tuning changes how a model behaves. RAG changes what it reads. Prompting changes what it is told. Try them in the order prompt, RAG, fine-tune.
  • Fine-tune for format, narrow repeated tasks, shorter prompts or distillation, not for facts that change.
  • Full fine-tuning needs memory for weights, gradients and optimiser state, roughly 112 GB for a 7-billion-parameter model.
  • LoRA trains two thin matrices per weight matrix, often around 0.1% of the parameters, so it fits on far smaller hardware.
  • The biggest costs are usually good training data and ongoing maintenance, not the GPU hours.

Glossary

  • Fine-tuning: training an existing model further on your own examples to change its behaviour.
  • Full fine-tuning: fine-tuning that updates every parameter in the model.
  • Gradient: for each weight, a number saying which way and how much it should change to reduce error.
  • Optimiser state: extra numbers the training algorithm keeps per weight to make updates work well.
  • LoRA (low-rank adaptation): fine-tuning by freezing the original weights and training two small matrices alongside each chosen weight matrix.
  • Rank: in LoRA, the inner size of the two small matrices, such as 8. Higher rank means more trainable parameters.
  • QLoRA: LoRA with the frozen base weights stored in 4-bit form to save memory.
  • Adapter: the small set of trained LoRA weights, stored separately from the base model.
  • Distillation: training a smaller model to imitate a larger model's outputs on a task.
  • Catastrophic forgetting: a model losing earlier abilities after being trained heavily on something new.

Next up

So far the model has only produced text. In Article 16: Tool use, agents and MCP, we finally open up the agent loop: how a model asks to run a tool, how the harness executes it, how the agent decides what to do next, and how the Model Context Protocol lets any tool plug into any agent. We will trace the whole BUG-1423 run from ticket to pull request.