Context windows and prompt engineering

A model knows two kinds of things: what it learned in training, and what is in front of it right now. The second kind, the context, is the only part you control at request time. What you put there, in what order, and how often you pay to send it again, decides most of the quality and cost of an LLM application.

The context window

The context window is the maximum number of tokens a model can take into account in one request. It covers everything the model sees: system prompt, documents, conversation history, tool results, and, for most models, the tokens it generates in reply. Anything outside the window does not exist as far as the model is concerned.

One fact surprises people: the model has no memory between requests. A chat that seems to remember what you said ten messages ago works because the application re-sends the whole conversation every time. The same is true for agents. Every step of our bug-fix agent sends the full history so far.

Context windows have grown enormously. As of October 2026, many leading models accept somewhere between 200,000 and over 1,000,000 tokens per request. Check the exact limit for the model you use. At roughly three quarters of a word per token, 200,000 tokens is a long novel.

Bigger is not free

Even when everything fits, filling the window has costs:

• Money. You pay for every input token on every request. Sending 100,000 tokens fifty times is five million tokens.
• Time. All of it goes through prefill (Article 10) before the first output token appears.
• Attention quality. Models do not use all positions equally well. A 2023 study, "Lost in the Middle" by Liu and colleagues, found that models were better at using information near the start or end of a long context than in the middle. Model makers have worked on this since, and newer models cope better, but relevant, focused context still beats a dump of everything.

What fits: a token budget

Let's budget one step of our agent: the call where it writes the fix for BUG-1423. The token counts are illustrative estimates for a typical agent, not measurements.

Worked example: the agent's context at the "write the fix" step

Adding it up: 1,200 + 2,500 + 120 + 6,000 + 1,500 + 350 + 900 + 1,800 + 4,000 = 18,370 tokens. That fits easily in a 200,000-token window.

But agents grow. If a harder bug takes 40 steps and each step adds about 3,000 tokens of tool output and reasoning, the history alone reaches 40 × 3,000 = 120,000 tokens. Now the budget is tight, every call is slow and expensive, and the important early details sit in the middle of a huge context.

The chart shows how that 18,370-token prompt divides up.

Figure 13.1: The fix itself is one line, but the model needs about 18,000 tokens of context to write it confidently.

Agents manage long runs with a few standard tricks: dropping or shortening old tool outputs, asking the model to summarise earlier steps into a compact note, and fetching only the code that is needed rather than whole directories. Article 14 covers that last idea in depth.

Prompt engineering that actually works

"Prompt engineering" has picked up a reputation for magic phrases. Most of what reliably helps is plain clear writing, the same things that help a new colleague do a task well.

Say exactly what you want. "Fix the bug" is weaker than "Find why login fails for emails with uppercase letters, change as little code as possible, and add a pytest test that reproduces the bug."

Give the context the task needs. The model cannot ask a colleague what the codebase conventions are. If emails are stored in lowercase, say so, or make sure the code that shows it is in the prompt.

Separate the parts clearly. Mark where instructions end and data begins, for example with headings or tags:

<ticket>
BUG-1423: Login fails when the email has uppercase letters.
</ticket>
<file path="auth/login.py">
...
</file>
Task: explain the cause, then propose the smallest fix.

This also helps the model avoid treating instructions hidden inside data as commands, an important defence covered in Article 19.

Put long material first and the question last. For long prompts, some providers, Anthropic for example, recommend placing documents near the top and the specific instruction at the end. Check your provider's prompting guide, since advice differs between models.

Let it think before answering. Asking the model to work through the problem before giving a final answer often improves results on multi-step tasks. Many current models have a built-in reasoning mode that does this automatically (Article 8).

Specify the output format. If a program will read the answer, say exactly what shape it must take, or use the provider's structured output feature.

Give it a way out. "If you cannot find the cause, say so and list what you checked" reduces confident guessing.

Tricks like flattery ("you are the world's best engineer") or capital-letter shouting have inconsistent effects on modern models. Clarity and relevant context do far more.

Few-shot examples

Instructions describe what you want. Examples show it. Including a few worked input and output pairs in the prompt is called few-shot prompting, and it is one of the most reliable techniques there is, especially for formats and judgement calls that are hard to describe.

Suppose the agent first triages tickets by area:

Classify each ticket as AUTH, BILLING or UI.

Ticket: Password reset email never arrives.
Area: AUTH

Ticket: Invoice total ignores the discount code.
Area: BILLING

Ticket: Save button overlaps the footer on mobile.
Area: UI

Ticket: Login fails when the email has uppercase letters.
Area:

The most likely next token is now AUTH, and just as importantly, the answer comes back in exactly the format the code expects. Three to five varied examples usually beat a paragraph of rules. Make them varied, so the model does not copy one accidental pattern, such as always answering with the first category.

Prompt caching

Look again at the token budget. On every call, the system prompt, tool definitions and ticket are identical. Without caching, the provider runs prefill on all of them every time.

Prompt caching keeps the processed form of a prompt's beginning, essentially its KV cache (Article 8), for a short time. If the next request starts with exactly the same tokens, the provider reuses that work instead of recomputing it. Cached input is billed at a discount and the first token arrives sooner.

The key word is exactly. The cache matches a prefix: the shared beginning of the prompt, token for token. Change one character early on, such as a timestamp in the system prompt, and everything after it misses the cache. So the rule is: stable content first, changing content last.

The diagram shows the order that makes caching work for the agent.

flowchart TD
    S["System prompt"]:::io --> T["Tool definitions"]:::io
    T --> K["Ticket BUG-1423"]:::io
    K --> H["Earlier steps"]:::io
    H --> N["Newest tool result"]:::warn
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 13.2: Teal parts repeat from the previous call and can be served from the cache. Only the coral part at the end is new each time.

Worked example: what caching saves

The prices here are illustrative, though the structure mirrors how several providers price caching: as of October 2026, some charge extra to write to the cache and charge a small fraction of the normal price to read from it. Assume:

• Normal input: $3.00 per million tokens
• Cache write: $3.75 per million (25% more)
• Cache read: $0.30 per million (90% less)

The agent makes 30 calls for one ticket. To keep the arithmetic simple, assume each call has the same 15,000-token stable prefix plus 3,000 new tokens. (In reality the prefix grows a little each step and the cache extends with it.)

Without caching: 30 × 18,000 = 540,000 tokens at $3.00 per million = 0.54 × $3.00 = $1.62.

With caching:

• First call writes the prefix: 15,000 tokens at $3.75 per million = $0.056.
• 29 later calls read it: 29 × 15,000 = 435,000 tokens at $0.30 per million = $0.131.
• New tokens on all 30 calls: 30 × 3,000 = 90,000 tokens at $3.00 per million = $0.27.
• Total: $0.056 + $0.131 + $0.27 ≈ $0.46.

That is about 72% cheaper for the same work, and each call starts faster. By default, caches typically expire after about five minutes without use, and some providers sell longer lifetimes for a higher write price. Short lifetimes suit agents well because their calls come in quick succession.

The diagram shows two consecutive agent calls with caching.

flowchart TD
    C1["Call 1: full prompt"]:::io --> P1["Prefill, cache the prefix"]:::infra
    P1 --> R1["Reply: run pytest"]:::io
    R1 --> C2["Call 2: prefix + test output"]:::io
    C2 --> P2["Reuse cache, prefill new part"]:::warn
    P2 --> R2["Reply: the fix"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 13.3: The second call only pays full price for the new test output.

Where this shows up in our bug-fix agent

• Budgeting. Each step of fixing BUG-1423 needs about 18,000 tokens of context. The harness decides what goes in: the ticket, auth/login.py, the relevant test file, the pytest output, but not the whole repository.
• Structure. The system prompt tells the agent the house rules: minimal changes, always add a test, use the PR title format "Normalise email case on login (BUG-1423)". Files go inside clearly labelled tags.
• Few-shot. Example tool calls in the system prompt show the model the exact format the harness can parse.
• Caching. System prompt, tools and ticket come first and never change during the run, so 29 of 30 calls read them from the cache.
• Long runs. If the agent goes past a few dozen steps, the harness summarises old steps so the important facts, like "emails are stored in lowercase", do not get lost in the middle.

Common misconceptions

"With a million-token window I can just paste everything." You can, but you pay for it on every call, wait longer for every answer, and the model may overlook what matters. Relevant context beats more context.

"The model remembers our earlier conversations." The model is stateless. Any memory comes from the application re-sending history or storing notes and adding them back to the prompt.

"Prompt engineering is about secret magic words." The reliable gains come from clarity, the right context, good structure and examples. Phrasing tricks are inconsistent across models.

"Prompt caching caches the answers." It caches the processed input prefix, not the response. Every call still generates a fresh answer. Caching whole responses is a different technique, covered with gateways in Article 17.

Key takeaways

• The context window is everything the model can see in one request. The model has no memory beyond it, so applications re-send history every time.
• Filling the window costs money, time and focus. Budget it like any limited resource.
• The techniques that work are clear instructions, needed context, clearly separated sections, examples, room to reason, and a defined output format.
• Few-shot examples are one of the most reliable ways to get consistent formats and judgements.
• Prompt caching reuses an identical prompt prefix. Put stable content first to make agents much cheaper and faster.

Glossary

• Context window: the maximum number of tokens a model can take into account in one request, including its reply for most models.
• Context: everything included in a request: instructions, documents, history and tool results.
• Stateless: keeping no memory between requests.
• Token budget: a plan for how a limited context window is divided between the parts of a prompt.
• Prompt engineering: designing prompts so a model performs a task reliably.
• Few-shot prompting: including a few input and output examples in the prompt to show the desired behaviour.
• Prompt caching: reusing the processed form of an identical prompt prefix across requests.
• Prefix: the shared beginning of two prompts, token for token.

Next up

Our token budget only worked because the harness already knew which files mattered. In Article 14: RAG from first principles, we look at how a system finds the right few pieces of information out of millions, using chunking, embeddings, vector databases and re-ranking.