Every LLM product exposes a few settings with mysterious names: temperature, top-p, max tokens. They control the last step of every prediction, where the model turns its scores into one actual token. Get them right and a coding agent writes the same careful fix every time. Get them wrong and it invents a function that does not exist. This article explains what happens at that final step, then looks at two engineering tricks that make generation feel fast: streaming and the KV cache.

From scores to probabilities: softmax

In Article 1 we said the model outputs a probability for every token in its vocabulary. That was a slight simplification. The last layer of the model actually outputs a raw score for each token, called a logit. Logits can be any number: positive, negative, large or small. A higher logit means "more likely", but logits are not probabilities and do not add up to 1.

A function called softmax converts logits into probabilities. It does two things:

  1. Raises the number e (about 2.718) to the power of each logit. This makes every value positive and stretches the gaps, so higher scores pull further ahead.
  2. Divides each result by the total, so everything adds up to 1.

You do not need to calculate powers of e by hand. Any calculator or spreadsheet has an exp button. What matters is the shape of the result, which the worked example below makes concrete.

Choosing a token: greedy versus sampling

Once we have probabilities, we have to pick one token. There are two basic approaches.

Greedy decoding always picks the most likely token. It is simple and predictable. For the same input it usually gives the same output. The downside is that it can get stuck in repetitive loops and tends to produce flat, generic text, because the single most likely next word is often the dullest one.

Sampling picks a token at random, weighted by the probabilities. If lower has a probability of 0.88, it gets picked 88% of the time, and other tokens get their small share. This produces more varied, natural text, at the cost of sometimes choosing a poor token.

Here is how a weighted random pick works in practice. Line the probabilities up end to end between 0 and 1, then draw a random number:

lower     0.00 to 0.88
casefold  0.88 to 0.96
upper     0.96 to 1.00

random number = 0.91  →  falls in casefold's range  →  pick "casefold"

Most draws land in lower's large range, but now and then a draw lands elsewhere.

Temperature: turning the randomness up or down

Temperature controls how adventurous sampling is. It works by dividing every logit by the temperature value before softmax.

  • A temperature below 1 makes the gaps between logits bigger, so the top token dominates even more.
  • A temperature above 1 shrinks the gaps, so less likely tokens get a bigger share.
  • A temperature of 0 is treated as greedy decoding: always take the top token.

Top-k and top-p: cutting off the long tail

A vocabulary has tens of thousands of tokens or more, and even very unlikely ones get a tiny probability. Sampling from all of them means the occasional absurd token sneaks through. Two common filters remove the tail before the random pick:

  • Top-k keeps only the k most likely tokens, for example the top 40, and discards the rest.
  • Top-p, also called nucleus sampling, keeps the smallest set of top tokens whose probabilities add up to at least p, for example 0.9. When the model is confident, that might be just one or two tokens. When it is unsure, it might be hundreds.

After filtering, the remaining probabilities are rescaled so they add up to 1 again.

The diagram shows the order in which these steps run for every single token.

flowchart TD
    A["Logits for every token"]:::model --> B["Divide by temperature"]:::model
    B --> C["Softmax to probabilities"]:::model
    C --> D["Top-p or top-k filter"]:::model
    D --> E["Weighted random pick"]:::model
    E --> F["Token sent to user"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 8.1: The decoding pipeline. These steps run once per generated token, so a 300-token answer goes through it 300 times.

A worked example: temperature with real numbers

To keep the arithmetic short, imagine the model only has three candidates after email = email.strip().. Real models score their whole vocabulary, but the maths is identical. The logits are illustrative:

Token Logit
lower 4.0
casefold 1.5
upper 1.0

Temperature 1 (no change). Raise e to each logit:

exp(4.0) = 54.60    exp(1.5) = 4.48    exp(1.0) = 2.72
total = 54.60 + 4.48 + 2.72 = 61.80

lower    = 54.60 ÷ 61.80 = 0.883
casefold =  4.48 ÷ 61.80 = 0.073
upper    =  2.72 ÷ 61.80 = 0.044

Temperature 0.5. Divide each logit by 0.5 first, giving 8.0, 3.0 and 2.0:

exp(8.0) = 2981.0    exp(3.0) = 20.1    exp(2.0) = 7.4
total = 3008.5

lower    = 0.991
casefold = 0.007
upper    = 0.002

Temperature 2. Divide each logit by 2, giving 2.0, 0.75 and 0.5:

exp(2.0) = 7.39    exp(0.75) = 2.12    exp(0.5) = 1.65
total = 11.16

lower    = 0.662
casefold = 0.190
upper    = 0.148

The chart puts the three results side by side.

Chance of picking "lower" T = 0.5 99% T = 1 88% T = 2 66% Lower temperature, safer choice.

Figure 8.2: The same logits at three temperatures. Lowering the temperature concentrates probability on the top token. Raising it spreads probability out.

At temperature 2, one time in three the model would write something other than lower. For a poem that variety might be welcome. For a bug fix it is a liability.

Top-p on top. Take a slightly bigger illustrative distribution: lower 0.81, casefold 0.07, upper 0.05, replace 0.04, title 0.03. With top-p set to 0.9, add from the top until the total reaches 0.9: 0.81, then 0.88, then 0.93. That is three tokens, so replace and title are discarded. Rescale the survivors by dividing by 0.93: lower becomes 0.871, casefold 0.075, upper 0.054. The absurd options are now impossible, while some variety remains.

Stopping

Generation ends when one of three things happens: the model produces its special end-of-turn token (it learned when to stop during post-training, Article 7), the output hits the max tokens limit you set, or the output matches a stop sequence you specified. If an answer ends abruptly mid-sentence, the max tokens limit is the usual cause.

Streaming

Because tokens come out one at a time, there is no reason to make the user wait for the whole answer. Streaming sends each token to the client as soon as it is generated, typically over a long-lived HTTP connection. That is why chat interfaces show text appearing word by word.

Streaming introduces two speed measures you will see everywhere:

  • Time to first token (TTFT): how long until the first token arrives. This includes reading the whole prompt.
  • Tokens per second: how fast the rest of the answer flows once it starts.

The KV cache

Article 1 said the model reads all the tokens so far, every time it predicts the next one. Done naively, that is enormously wasteful. To predict token 1,001, the model would reprocess tokens 1 to 1,000, all of which it already processed when predicting token 1,000.

The fix is the KV cache. In Article 5 we saw that attention computes a "key" and a "value" for every token. Those do not change once computed, so the model stores them in GPU memory. For each new token it computes keys and values for that one token only, and reads the rest from the cache.

The diagram compares one generation step with and without the cache.

Without KV cache email = email .strip(). lower Every earlier token recomputed With KV cache email = email .strip(). lower Old tokens read from memory

Figure 8.3: The KV cache. Coral tokens are recomputed work. Teal tokens are read from GPU memory. Only the new purple token is computed in full.

Some counting shows why this matters. Say the prompt is 1,000 tokens and the model generates 100 more.

  • Without a cache, step 1 processes 1,000 tokens, step 2 processes 1,001, and so on up to 1,099. The total is 100 × 1,000 plus 0 + 1 + ... + 99, which is 100,000 + 4,950 = 104,950 token-processings.
  • With a cache, the prompt is processed once (1,000) and each new token once (100), for 1,100.

That is roughly 95 times less work. I am simplifying: with a cache, each new token still has to look at all the cached keys and values through attention, so longer contexts still cost more. But the expensive part of recomputing them is gone.

The catch is memory. The cache grows with every token of context, for every conversation being served at once. Managing that memory is a central problem in serving models, covered in Article 10.

Where this shows up in our bug-fix agent

  • Low temperature for code. The harness calls the model with a temperature at or near 0. We want email.strip().lower() every time, not a creative alternative. The agent's ability to try different approaches comes from the loop of attempts and test results, not from randomness inside a single answer.
  • Max tokens on every call. Each step of the agent has a max tokens limit so a confused model cannot generate forever and run up the bill.
  • Streaming for the human. When a developer watches the agent work, its reasoning streams onto the screen. The harness itself often waits for a complete tool request before acting on it.
  • The KV cache across a long session. By the time the agent has read auth/login.py, the test file and some test output, the context might be thousands of tokens. The KV cache is what keeps each new step from re-reading all of it from scratch. Providers also cache repeated prompt prefixes between calls, which Article 13 covers as prompt caching.

Common misconceptions

"Temperature 0 means the output is always identical." It removes deliberate randomness, but tiny differences in how GPUs add numbers in parallel, and in how requests are batched together on the server, can still flip a close decision between two tokens. Outputs are usually the same, not guaranteed to be.

"Higher temperature makes the model more creative or smarter." It makes the model more random. Sometimes that reads as creativity. It never adds knowledge, and too high a temperature quickly produces nonsense.

"Streaming makes the model faster." The total time to generate the full answer is the same. Streaming improves how fast it feels, because the user starts reading after the first token instead of after the last.

Key takeaways

  • The model outputs logits. Softmax turns them into probabilities that add up to 1.
  • Greedy decoding always takes the top token. Sampling picks randomly, weighted by probability.
  • Temperature divides the logits before softmax: below 1 is safer and more predictable, above 1 is more varied.
  • Top-k and top-p remove unlikely tokens before the random pick.
  • Streaming sends tokens as they are made, and the KV cache stores earlier work so each new token is cheap to compute.

Glossary

  • Logit: the raw score the model gives each possible next token before conversion to probabilities.
  • Softmax: the function that turns logits into probabilities that add up to 1.
  • Decoding: the process of choosing actual tokens from the model's probabilities.
  • Greedy decoding: always choosing the most likely token.
  • Sampling: choosing a token at random, weighted by the probabilities.
  • Temperature: a setting that divides the logits before softmax, controlling how random sampling is.
  • Top-k: keeping only the k most likely tokens before sampling.
  • Top-p (nucleus sampling): keeping the smallest set of top tokens whose probabilities add up to at least p.
  • Max tokens: a limit on how many tokens a single response may contain.
  • Stop sequence: a string that ends generation as soon as the model produces it.
  • Streaming: sending tokens to the client as they are generated.
  • Time to first token (TTFT): the delay before the first token of a response arrives.
  • KV cache: stored attention keys and values for earlier tokens, so they are not recomputed for each new token.

Next up

That wraps up Part 1: how an LLM works. All of it runs on one kind of hardware. In Article 9: GPUs explained, we look at why graphics chips ended up running AI, what VRAM and memory bandwidth are, and why generating one token at a time is limited by memory speed more than by raw calculating power.