Every LLM bill, every context limit and every "message too long" error is counted in tokens, not words or characters. If you run AI in production, tokens are your unit of cost and your unit of capacity. They also explain some odd behaviour, like a model that writes fluent code but struggles to count the letters in a word.

What a token is

In Article 1 we said a model predicts the next token, and described a token as a chunk of text. Here is the precise version.

A tokenizer is a separate, fixed program that runs before the model. It splits text into pieces from a fixed list, called the vocabulary, and replaces each piece with its position in that list. That position is the token ID. The model never sees text. It sees a list of numbers.

For example, a tokenizer might turn the word login into a single token with ID 4521, and the text log in into two tokens. The numbers here are made up for illustration. Every model family has its own tokenizer, so the same text produces different IDs and often different splits depending on the model.

When the model answers, the process runs in reverse. It produces token IDs, and a detokenizer looks each one up in the vocabulary and glues the text back together.

The diagram shows where tokenization sits in a request to the model.

flowchart TD
    A["Ticket text"]:::io --> B["Tokenizer"]:::model
    B --> C["Token IDs"]:::model
    C --> D["LLM"]:::model
    D --> E["Output token IDs"]:::model
    E --> F["Detokenizer"]:::model
    F --> G["Reply text"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 2.1: Text becomes token IDs on the way in and token IDs become text on the way out. The model only ever works with the numbers.

Why not letters, or whole words?

There are two obvious ways to split text, and both are bad.

One token per letter gives a tiny vocabulary, but sequences get very long. A 300-line Python file might be 10,000 characters. As Article 5 explains, the work a model does grows quickly with sequence length, so this is expensive. Each step would also carry very little meaning.

One token per word gives short sequences, but the vocabulary explodes. Every name, typo, variable name and URL would need its own entry, and anything not in the list could not be represented at all. find_user_by_email would be a single, rare word that the model barely ever saw.

Real tokenizers sit in between. Common words get one token each. Rare words are built from smaller pieces. Nothing is ever unrepresentable, because in the worst case the tokenizer can fall back to individual characters or even raw bytes.

Byte pair encoding: building the vocabulary

The most widely used method for building that in-between vocabulary is byte pair encoding, or BPE. Many tokenizers in use today are variants of it. The idea is simple: start with single characters and repeatedly glue together the pair that appears most often.

This happens once, before the model is trained, on a large sample of text. The list of merges it produces is then frozen and used forever.

The size of the vocabulary is a design choice. GPT-2's tokenizer, from 2019, had 50,257 tokens. As of October 2026, many recent models use vocabularies of roughly 100,000 to just over 250,000 tokens. A bigger vocabulary means more text fits in fewer tokens, at the cost of a bigger lookup table inside the model.

A worked example: BPE by hand

Let's run BPE on a tiny "training sample" taken from an auth codebase. It contains the word login three times and logout twice. I'm simplifying here: real BPE works on bytes and handles spaces and punctuation too, but the merging logic is the same.

Step 0: split into characters.

  • l o g i n (appears 3 times)
  • l o g o u t (appears 2 times)

Step 1: count adjacent pairs. Count each pair, weighted by how often its word appears.

  • l o: 3 + 2 = 5
  • o g: 3 + 2 = 5
  • g i: 3
  • i n: 3
  • g o: 2, o u: 2, u t: 2

l o and o g tie at 5. We take the first one and merge l and o into a new token lo. Our vocabulary now has one more entry.

Step 2: count again. The words are now lo g i n and lo g o u t. The pair lo g appears 3 + 2 = 5 times, more than any other. Merge it into log.

Step 3. Now the words are log i n (3) and log o u t (2). The pairs log i and i n both appear 3 times. Take log i to make logi.

Step 4. logi n appears 3 times. Merge it into login.

The diagram traces the four merges.

flowchart TD
    A["l o g i n"]:::io -->|"merge l + o"| B["lo g i n"]:::model
    B -->|"merge lo + g"| C["log i n"]:::model
    C -->|"merge log + i"| D["logi n"]:::model
    D -->|"merge logi + n"| E["login"]:::warn
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 2.2: Byte pair encoding merges the most frequent pair at each step. After four merges, the frequent word "login" is a single token.

After four merges, login is one token. logout is split as log + o + u + t, four tokens, because it was less common in our sample. That is the core behaviour of BPE: frequent text becomes few tokens, rare text becomes many tokens. A real tokenizer runs tens of thousands of merges over billions of words, so common English words, common code keywords and common word endings all end up as single tokens.

Why code and non-English text cost more

That last rule has a direct consequence for your bill. Text that was common in the tokenizer's training sample is cheap. Text that was rare is expensive.

Non-English text. If the training sample was mostly English, English words get their own tokens while words in other languages are built from smaller pieces. A sentence in Greek, Arabic or Hindi often needs noticeably more tokens than the same sentence in English, sometimes several times more, depending on the language and the tokenizer. More tokens means a higher cost, slower responses and less room in the context. If you serve users in several languages, measure this with your model's tokenizer rather than guessing.

Code. Common keywords like def, return and import are usually single tokens. But code is full of things that are rare in ordinary text: long variable names, unusual punctuation, indentation, hashes and IDs. find_user_by_email might be four or five tokens. A minified JavaScript file or a lockfile full of hashes can be very token-hungry for how little meaning it carries.

Numbers and identifiers. Long numbers, UUIDs and base64 strings break into many small tokens. A single API key or commit hash can cost more tokens than a full sentence.

For rough planning in English, a common rule of thumb is that one token is about four characters, or about three quarters of a word. For code and other languages, measure instead.

Where this shows up in our bug-fix agent

Tokenization touches the bug-fix agent in three places.

The bug itself. Here is how the two versions of the email might tokenize. The splits are illustrative, since they vary by tokenizer.

Typed by the user Stored in the database Mar ia @ Example .com maria @ example .com 5 tokens 4 tokens

Figure 2.3: Illustrative tokenization. Capital letters change which tokens are used (coral), so the model sees "Maria" and "maria" as different IDs.

To the model, Example and example are different token IDs with no built-in connection. It has learned during training that they usually mean the same thing, which is why it can reason about the bug at all. Article 3 explains how that learned connection is stored.

The fix. In Article 1 we showed the fix line generated token by token: email, =, email, .strip, ()., lower, (). Notice the leading spaces. Most tokenizers fold the space into the following word, so email with a space and email without one are different tokens. Again, the exact split depends on the tokenizer.

The cost. Every step of the agent sends tokens in and gets tokens back. Suppose one step sends the ticket, auth/login.py and some test output, totalling 3,000 input tokens, and the model replies with 500 output tokens. Using illustrative prices of $3 per million input tokens and $15 per million output tokens:

  • Input: 3,000 ÷ 1,000,000 × $3 = $0.009
  • Output: 500 ÷ 1,000,000 × $15 = $0.0075
  • One step: $0.009 + $0.0075 = $0.0165

If the agent takes 20 steps to search, read, fix, test and open the pull request, that is 20 × $0.0165 = $0.33 for the ticket. Real prices vary widely between models and change often, so check current pricing before you plan a budget. Notice that output tokens usually cost more per token than input tokens, and that re-sending the same file at every step adds up. Article 13 covers prompt caching, which reduces exactly that cost.

Common misconceptions

"A token is a word." Sometimes it is, often it is not. Common words are usually one token, but rare words, names, code and other languages are split into several. Plan with tokens, not word counts.

"The model can see the letters in a word." It sees token IDs. If strawberry is one or two tokens, the model has no direct view of its individual letters, which is why simple letter-counting questions can trip it up. It can only answer from what it learned about those tokens during training.

"Every model counts tokens the same way." Each model family has its own tokenizer and vocabulary. The same prompt can be a different number of tokens on two models, so a cost estimate for one model does not carry over to another.

Key takeaways

  • Models work on token IDs, not text. A tokenizer converts text to IDs before the model runs, and a detokenizer converts the output back.
  • Byte pair encoding builds the vocabulary by repeatedly merging the most frequent pair of pieces, so frequent text becomes few tokens and rare text becomes many.
  • Code, identifiers and non-English text usually need more tokens than plain English, which makes them slower and more expensive to process.
  • Tokens are the unit of cost and of capacity. Input and output tokens are usually priced separately, with output costing more.
  • Uppercase and lowercase versions of a word are different tokens. The model links them only through what it learned in training.

Glossary

  • Tokenizer: the program that splits text into tokens from a fixed vocabulary and converts them to token IDs.
  • Token ID: the number that identifies a token's position in the vocabulary. This is what the model actually receives.
  • Detokenizer: the reverse step that turns the model's output token IDs back into text.
  • Byte pair encoding (BPE): a method for building a vocabulary by repeatedly merging the most frequent adjacent pair of characters or pieces.
  • Merge: one step of BPE, where a frequent pair of pieces becomes a new single token.
  • Vocabulary size: how many distinct tokens a tokenizer knows.
  • Input tokens and output tokens: the tokens you send to a model and the tokens it generates. Providers usually price them separately.
  • Context window: the maximum number of tokens a model can take into account at once. Article 13 covers it in depth.

Next up

The model receives a list of token IDs, but a number like 4521 says nothing about meaning. In Article 3: Embeddings, we look at how each token is turned into a list of numbers that captures meaning, why "login" and "sign in" end up close together, and how the same idea lets our agent find the right file for a bug ticket.