A token ID is just a label. Knowing that login is token 4521 tells the model nothing about what logging in means. Embeddings are how models attach meaning to those labels, and they are also the engine behind semantic search, document retrieval and "find me something similar" features. If you build anything with AI search in it, you are building on embeddings.

From an ID to a list of numbers

An embedding is a list of numbers that represents a piece of text. A list of numbers like this is called a vector, and the number of entries in it is its dimension.

Inside every LLM, the very first step after tokenization is a lookup in an embedding table. The table has one row per token in the vocabulary, and each row is a vector. Token 4521 goes in, row 4521 comes out. With a vocabulary of 100,000 tokens and vectors of 4,096 numbers each, that table alone holds 100,000 × 4,096 = 409,600,000 numbers.

Those numbers are not written by hand. They start random and are adjusted during training (Article 6), along with everything else in the model. Over time, tokens that are used in similar ways end up with similar vectors, because that helps the model predict the next token.

The diagram shows where the embedding lookup sits inside the model.

flowchart TD
    A["Token IDs"]:::io --> B["Embedding table lookup"]:::model
    B --> C["One vector per token"]:::model
    C --> D["Transformer layers"]:::model
    D --> E["Next-token scores"]:::model
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 3.1: Each token ID is swapped for its row in the embedding table. From then on, the model works with vectors, not IDs.

Meaning as geometry

Here is the useful mental picture. Treat each vector as the coordinates of a point. A vector with two numbers is a point on a flat map. A vector with three numbers is a point in a room. A vector with 4,096 numbers is a point in a space with 4,096 directions. Nobody can picture that, but the arithmetic works exactly the same way as in two dimensions.

In that space, things with similar meaning end up close together. login, sign in and authenticate cluster in one region. colour, theme and button cluster somewhere else. Distance becomes a measure of how related two things are.

To make this concrete, imagine a toy space with just two directions: "how much is this about authentication" and "how much is this about visual design". I'm simplifying a lot here. Real embedding directions are not labelled, and a single meaning is spread across many of them. But the toy version shows the idea.

The plot shows a two-dimensional toy embedding space with code and words from our running example.

about visual design → about authentication → login sign in auth/login.py ticket colour theme.css button

Figure 3.2: A toy embedding space. Related things sit close together, so the bug ticket lands next to the login code, far from the styling files.

A famous early example comes from word2vec, a word embedding method published by researchers at Google in 2013. In that kind of space, the vector for king, minus man, plus woman, lands close to the vector for queen. The relationship "male to female" turned out to be a rough direction in the space. It only works approximately and not for every pair, but it shows that the geometry captures real structure.

Embeddings for whole texts

So far we have talked about one vector per token, inside an LLM. There is a second, very practical kind: text embeddings produced by a dedicated embedding model. You give it a sentence, a paragraph or a whole function, and it returns a single vector for the entire thing.

These are what power semantic search. You embed every document once and store the vectors. When a question comes in, you embed the question and look for the stored vectors closest to it. Because closeness tracks meaning, a search for "users can't log in with capital letters" can find a function called login that never contains the words "capital letters" at all. Keyword search would miss it.

Text embeddings typically have from a few hundred to a few thousand dimensions. The exact size depends on the embedding model you choose.

Measuring "close": dot product and cosine similarity

To compare two vectors, the standard tool is cosine similarity. It measures whether two vectors point in the same direction. A score of 1 means the same direction, 0 means unrelated, and negative scores mean opposite directions. It is built from the dot product: multiply the two vectors entry by entry, then add up the results.

A worked example: which file matches the ticket?

Let's use tiny three-dimensional vectors so the arithmetic fits on a page. Real vectors are much longer, but the steps are identical. The numbers are illustrative.

  • Ticket "Login fails when the email has uppercase letters": T = [0.9, 0.1, 0.3]
  • File auth/login.py: L = [0.8, 0.2, 0.4]
  • File ui/theme.css: C = [0.1, 0.9, 0.2]

Step 1: dot products. Multiply entry by entry and add.

  • T · L = (0.9 × 0.8) + (0.1 × 0.2) + (0.3 × 0.4) = 0.72 + 0.02 + 0.12 = 0.86
  • T · C = (0.9 × 0.1) + (0.1 × 0.9) + (0.3 × 0.2) = 0.09 + 0.09 + 0.06 = 0.24

Already the login file scores much higher. But a dot product also grows when vectors are simply longer, so we correct for length.

Step 2: lengths. The length of a vector is the square root of the sum of its squared entries. Square roots are the one step here beyond plain arithmetic, so I'll give the results.

  • Length of T = √(0.81 + 0.01 + 0.09) = √0.91 ≈ 0.954
  • Length of L = √(0.64 + 0.04 + 0.16) = √0.84 ≈ 0.917
  • Length of C = √(0.01 + 0.81 + 0.04) = √0.86 ≈ 0.927

Step 3: cosine similarity = dot product ÷ (length × length).

  • Ticket vs auth/login.py: 0.86 ÷ (0.954 × 0.917) = 0.86 ÷ 0.875 ≈ 0.98
  • Ticket vs ui/theme.css: 0.24 ÷ (0.954 × 0.927) = 0.24 ÷ 0.884 ≈ 0.27

The login file is a near-perfect match, the stylesheet is not. In practice, many embedding models return vectors already scaled to length 1, and then cosine similarity is just the dot product, which makes the comparison very cheap.

Similarity search at scale

A real codebase might be split into tens of thousands of chunks, each with its own vector. Comparing a question against every one of them is fine for a small project. For millions of vectors, vector databases use approximate nearest neighbour indexes that find very close matches without checking everything. Article 14 covers chunking, vector databases and re-ranking in depth.

Where this shows up in our bug-fix agent

Embeddings appear twice in the bug-fix agent.

Inside the model. When the LLM reads the ticket, Maria@Example.com and the code of login(), every token is first turned into a vector from the embedding table. In Article 2 we saw that Example and example are different token IDs. Their embedding vectors, though, sit close together, because the model learned during training that they are usually interchangeable. That learned closeness is part of why the model can connect an uppercase email in a ticket to a lowercase email in a database comment.

Finding the file. Before the model can fix anything, the agent harness has to find the right code. One common approach is to embed every function in the repository ahead of time, then embed the ticket and look up the nearest matches.

The diagram shows how the agent harness uses embeddings to find candidate files for BUG-1423.

flowchart TD
    A["Ticket BUG-1423"]:::io --> B["Embedding model"]:::model
    B --> C["Ticket vector"]:::model
    C --> D["Vector index of code"]:::infra
    D --> E["Top match: auth/login.py"]:::warn
    E --> F["File added to prompt"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 3.3: The ticket is embedded and compared with pre-computed vectors for the codebase. The closest file goes into the model's prompt.

Many coding agents also use plain text search, such as searching for find_user_by_email, and combine both approaches. Embeddings shine when the ticket's wording and the code's wording are different.

Common misconceptions

"Each number in an embedding means something specific." In our toy example the two directions had names. In real embeddings, individual numbers rarely have a clean meaning. Meaning is spread across many dimensions at once, and only the overall position and direction are reliable.

"Embeddings from different models can be compared." Each embedding model builds its own space. A vector from one model and a vector from another are not comparable, even if they have the same length. If you change embedding models, you have to re-embed everything you stored.

"Similar vectors mean the texts say the same thing." Similar vectors mean the texts are about related things. "Login is case sensitive" and "login must not be case sensitive" may sit very close together while saying opposite things. Similarity search finds candidates, and something else, often the LLM itself, has to read them.

Key takeaways

  • An embedding is a vector, a list of numbers, that represents a token or a piece of text.
  • Inside an LLM, an embedding table turns each token ID into a learned vector before any other processing happens.
  • Similar meanings end up close together, so distance in the vector space works as a measure of relatedness.
  • Cosine similarity, built from the dot product, is the standard way to measure that closeness.
  • Dedicated embedding models turn whole texts into single vectors, which is the basis of semantic search and retrieval.

Glossary

  • Embedding: a vector that represents a token or a piece of text, learned so that similar meanings get similar vectors.
  • Vector: an ordered list of numbers, which can be thought of as the coordinates of a point.
  • Dimension: the number of entries in a vector.
  • Embedding table: the lookup table inside an LLM with one learned vector per token in the vocabulary.
  • Embedding model: a model that turns a whole piece of text into a single vector, used for search and retrieval.
  • Dot product: multiply two vectors entry by entry and add up the results.
  • Cosine similarity: the dot product divided by both vectors' lengths. It measures whether two vectors point the same way, from 1 (same direction) down to -1 (opposite).
  • Vector database: a database that stores vectors and finds the nearest ones to a query quickly.

Next up

We now have vectors going into the model, and we keep mentioning "billions of parameters". In Article 4: Weights and neurons, we open the box and calculate a single neuron by hand, then scale it up to see what "70B parameters" really means, including how much memory it takes.