Running models locally

Running a model on your own machine used to require a research lab. Now a decent laptop can run a useful model offline, which matters when your code cannot leave the building, when you want to experiment without a bill, or when you simply want to understand what is going on. The hard part is not installing the software. It is knowing whether a given model will fit and how fast it will run, and that comes down to some simple arithmetic.

What has to fit in memory

To generate text, three things need to sit in fast memory, ideally the GPU's memory (VRAM, Article 9):

1. The weights. Every parameter of the model.
2. The KV cache. The stored attention data for every token in the current context (Article 8). It grows as the conversation grows.
3. Overhead. The runtime itself, temporary working buffers, and so on.

On Apple Silicon Macs, the CPU and GPU share one pool of unified memory, so the same arithmetic applies to that pool instead of to VRAM. If a model does not fit in fast memory, some runtimes can spill layers to ordinary system RAM and run them on the CPU, but it gets much slower.

Weights: parameters times bytes

Each parameter is a number, and how many bytes it takes depends on its precision:

Models are usually released in 16-bit. So the weights need roughly:

• An 8-billion-parameter model: 8 billion × 2 bytes = 16 GB.
• A 70-billion-parameter model: 70 billion × 2 bytes = 140 GB.

That 70B model will not fit on any single consumer graphics card. This is where quantization comes in, but first the other two pieces.

KV cache: per token, times tokens

The KV cache stores two vectors (a key and a value) for every token, in every layer. The size per token is:

2 × layers × KV heads × head size × bytes per number

Worked example: an 8B model's memory budget

Let's use the published configuration of Meta's Llama 3 8B: 32 layers, 8 key-value heads, and a head size of 128 numbers, with the cache stored in 16-bit (2 bytes).

KV cache per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, which is exactly 128 KB.

For a context of 8,192 tokens: 8,192 × 128 KB = 1,048,576 KB = 1 GB.

For 32,768 tokens: four times that, 4 GB.

Overhead: a rough rule of thumb is to allow another 10 to 20% on top. This varies a lot by runtime, so treat it as a planning margin, not a law.

Total for the 8B model in 16-bit with a 32K context: 16 GB weights + 4 GB cache + about 2 GB overhead ≈ 22 GB. That is a tight fit on a 24 GB card. Shrink the weights and it becomes comfortable.

The diagram shows how that memory might be split on a 12 GB GPU once the weights are shrunk to 4-bit, with a 16K-token context.

Figure 12.1: An 8B model quantized to 4-bit with a 16K context fits comfortably in 12 GB, with room to grow the context.

Quantization: fewer bits per number

Quantization stores each weight with fewer bits than it was trained with. Instead of a precise 16-bit number, each weight is rounded to one of a small set of levels. With 4 bits there are only 16 possible levels.

Worked example: quantizing four weights to 4-bit

Here is a simplified version of the idea. Take four weights from one small block of a model:

0.82, −0.37, 0.05, −1.10

Step 1: pick a scale. Using signed 4-bit levels from −7 to +7, divide the largest absolute value by 7: 1.10 ÷ 7 ≈ 0.157. Each step between levels is worth 0.157.

Step 2: round each weight to a level. Divide by the scale and round to the nearest whole number:

Step 3: store the four small levels (5, −2, 0, −7) at half a byte each, plus the one scale number for the whole block.

Every weight is now slightly off, but only slightly. Because a model's behaviour comes from billions of weights acting together, small rounding errors mostly average out. Real methods are more careful than this: they choose block sizes and scales cleverly, and some keep the most sensitive weights at higher precision. The fuller picture is that quantization is a trade-off you can measure, not a free lunch.

Because each block also stores a scale, real 4-bit files come out a little above 0.5 bytes per parameter. An 8B model at "4-bit" is typically around 4.5 to 5 GB rather than exactly 4.

The chart compares weight sizes for two common model sizes at each precision.

Figure 12.2: Weights only, before KV cache and overhead. Halving the bits halves the memory. Bars share one scale.

How much quality do you lose? As general guidance, 8-bit is usually very close to the original, 4-bit typically loses a little and is the popular sweet spot for local use, and going below 4 bits tends to hurt noticeably, especially for smaller models. It varies by model and task, so for anything important, test the quantized model on your own examples (Article 18).

How fast will it run?

Article 10 explained that decode is limited by memory speed: for every new token, the hardware must read all the weights. That gives a quick upper bound:

tokens per second ≤ memory bandwidth ÷ model size in memory

Using illustrative bandwidth figures: a GPU with 1,000 GB/s reading a 4.5 GB model can manage at most 1,000 ÷ 4.5 ≈ 220 tokens per second. A laptop with 100 GB/s manages at most 100 ÷ 4.5 ≈ 22 tokens per second. Real speeds land below these ceilings, but the formula explains why quantization speeds things up as well as saving memory: fewer bytes to read per token.

The tools: llama.cpp and Ollama

llama.cpp is an open source inference engine written in C and C++. It runs models on CPUs, Apple Silicon and most GPUs, and it popularised the GGUF file format, which packs a quantized model into a single file. You will see quantization variants named things like Q4_K_M or Q8_0. The number is roughly the bits per weight, and the letters describe the method.

Ollama is the friendly layer most people start with. It downloads models with one command, manages them, and serves them on a local HTTP API, including an OpenAI-compatible endpoint. It started as a wrapper around llama.cpp and, as of October 2026, also ships its own engine for some models.

ollama pull llama3.1:8b        # download a 4-bit quantized model
ollama run llama3.1:8b "Why might email login be case sensitive?"

Other options include LM Studio, a desktop app with a graphical interface.

Local tools versus production serving

The diagram contrasts a developer's local setup with a team's production setup.

flowchart TD
    D["Developer laptop"]:::io --> O["Ollama or llama.cpp"]:::infra
    O --> Q["4-bit GGUF model"]:::model
    T["Team's agents"]:::io --> V["vLLM on GPU server"]:::infra
    V --> F["16-bit or 8-bit model"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8

Figure 12.3: Local tools are built for one user on modest hardware. Production engines are built for many concurrent users.

Ollama and llama.cpp are excellent for one person, offline work, prototypes and small internal tools. Production serving for many users is a different job: continuous batching (Article 10), multiple GPUs, authentication, monitoring and autoscaling. That is what engines like vLLM are built for. A useful rule: if more than a handful of people or agents will hit it at once, move to a production engine.

Where this shows up in our bug-fix agent

Suppose a developer wants to try the agent on BUG-1423 without sending any code outside their laptop.

• Pick a model that fits. A laptop with 16 GB of unified memory can hold an 8B coding model at 4-bit (about 5 GB) with plenty of room for the KV cache.
• Point the harness at it. Because Ollama offers an OpenAI-compatible API on localhost:11434/v1, the agent harness only needs a different base URL. Nothing leaves the machine.
• Watch the context size. The agent's prompt holds instructions, tool definitions, auth/login.py, search results and pytest output, which can easily reach 20,000 to 30,000 tokens. At 128 KB per token that is 2.5 to 3.75 GB of KV cache for this kind of model. Local runners often default to a context window smaller than the model supports, and when the prompt is longer, older parts get cut. The agent then "forgets" the ticket halfway through. Raise the context setting to match.
• Expect the quality trade-off. A small local model will very likely produce email = email.strip().lower() for a bug this simple. For harder bugs, the gap to a large hosted model shows up quickly.

Common misconceptions

"If the weights fit, the model fits." The KV cache and overhead come on top, and the cache grows with context. An agent with a long prompt can run out of memory on a model that loaded fine.

"Quantized models are much worse." At 8-bit the difference is usually hard to notice, and 4-bit is often a reasonable trade. Very low bit widths are where quality drops sharply. Test on your own tasks.

"Ollama is a production server." It is a great local and small-team tool. For many concurrent users, a production engine with continuous batching will serve far more requests on the same hardware.

"More RAM in my PC means I can run big models fast." System RAM is much slower than VRAM or unified memory. A model spilled into ordinary RAM will run, but often painfully slowly.

Key takeaways

• Memory needed = weights (parameters × bytes per parameter) + KV cache (grows with context) + overhead.
• A 16-bit model needs 2 bytes per parameter. 8-bit halves that, 4-bit halves it again.
• Quantization rounds weights to fewer levels with a shared scale per block. Small errors mostly average out, but always test.
• Decode speed is capped by memory bandwidth divided by model size, so smaller files also run faster.
• Ollama and llama.cpp are ideal for local and small-scale use. Many concurrent users need a production engine such as vLLM.

Glossary

• VRAM: the memory on a GPU, where weights and KV cache ideally live.
• Unified memory: a single memory pool shared by CPU and GPU, as on Apple Silicon.
• Precision: how many bits are used to store each number.
• Quantization: storing weights with fewer bits by rounding them to a small set of levels.
• Scale: the number used to convert quantized levels back to approximate original values.
• GGUF: a single-file format for (usually quantized) models, used by llama.cpp and Ollama.
• llama.cpp: an open source C/C++ engine for running LLMs on a wide range of hardware.
• Ollama: a tool that downloads, manages and serves local models through a simple API.
• Memory bandwidth: how many bytes per second the hardware can read from memory.

Next up

Whether local or hosted, every model has a limit on how much text it can consider at once. In Article 13: Context windows and prompt engineering, we work out what actually fits in our agent's context, which prompt techniques make a measurable difference, and how prompt caching cuts the cost of re-sending the same context again and again.