A traditional program either passes its tests or it doesn't. An LLM system is different: the same input can produce different outputs, a new model can be better on average and worse on the cases you care about, and a small prompt edit can quietly break something three steps later. Without a way to measure quality and a way to see what happened inside each run, you are shipping changes on gut feeling.

This article covers both halves. Evals answer "is it good?" Observability answers "what did it actually do?"

Why normal tests are not enough

Unit tests check exact behaviour: given this input, expect exactly this output. That still works for the code around the model, but not for the model's output itself.

Ask the bug-fix agent to fix BUG-1423 twice and you may get two different but equally correct patches. One adds email = email.strip().lower(). Another lowercases the email inside the database query. Both fix the bug. An exact-match test would fail one of them.

So instead of checking for one exact answer, you check whether the output has the properties you care about. Did the tests pass? Did it change only the files it needed to? Is the explanation in the pull request accurate? That shift, from exact outputs to measurable properties, is what an eval is.

Golden datasets

An eval starts with a golden dataset: a fixed set of example inputs where you know what a good result looks like. For our agent, each example is a past bug ticket together with the repository as it was before the fix, and a way to check the result.

A good golden dataset has three qualities:

  • It is real. Use actual tickets from your history, not ones you invented in an afternoon. Invented cases tend to be easier than real ones.
  • It covers the categories you care about. Simple one-line fixes, bugs that span several files, tickets that are vague or wrong, tickets the agent should refuse.
  • It stays fixed. If you change the dataset every week, you can't compare this week's score with last week's.

You don't need thousands of examples to start. Fifty well-chosen cases tell you far more than none. Grow the set over time, and turn every production failure into a new case.

Grading the results

Each example needs a grader, the check that decides pass or fail. There are three kinds, and good evals use all three.

Code-based checks. Deterministic and cheap. For the bug-fix agent: do all tests pass, including the new test_login_with_uppercase_email? Did the agent touch only files under auth/ and tests/? Does the diff stay under 50 lines? These should be your first line of grading, because they are fast, free and never disagree with themselves.

LLM-as-judge. Some qualities are hard to check with code. Is the pull request description accurate and clear? Did the agent explain why the bug happened? For these, you can ask a second model to score the output against a written rubric. This scales well, but it has real weaknesses, covered below.

Human review. A person reads a sample of outputs and scores them. Slow and expensive, but it is the ground truth that the other two are calibrated against.

A worked example: comparing two versions

Say you want to switch the agent from its current prompt to a new one that asks it to write a failing test before fixing the code. You run both versions on a golden dataset of 50 past tickets.

Step 1: raw pass rates. A pass means all tests pass and the diff stays in scope.

  • Current prompt: 42 of 50 pass. 42 ÷ 50 = 0.84, so 84%.
  • New prompt: 45 of 50 pass. 45 ÷ 50 = 0.90, so 90%.

The new prompt looks 6 points better. But that is a difference of only 3 tickets.

Step 2: look at the changes case by case. Overall numbers hide movement. Compare the two runs ticket by ticket:

  • 40 tickets passed on both.
  • 5 tickets passed only on the new prompt.
  • 2 tickets passed only on the current prompt.
  • 3 tickets failed on both.

Check: 40 + 5 + 2 + 3 = 50. The new prompt fixed 5 cases and broke 2. Those 2 regressions matter. If they are both "multi-file" bugs, the new prompt might be worse at exactly the hardest category.

Step 3: is the difference real? With 50 cases, a swing of a few tickets can happen by chance, because the model's output varies from run to run. A practical way to check, without any statistics, is to run each version several times. If the current prompt scores 84%, 80% and 86% on three runs, and the new one scores 90%, 88% and 92%, the ranges don't overlap, and you can be fairly confident. If they overlap heavily, you don't have an answer yet. You need more cases or more runs. I am simplifying here: proper significance testing exists, but repeated runs are a good first habit.

The diagram shows where these comparisons sit in the release process.

flowchart TD
    A["Prompt or model change"]:::io --> B["Run golden dataset"]:::infra
    B --> C["Code checks"]:::infra
    B --> D["LLM judge"]:::model
    C --> E["Compare to baseline"]:::infra
    D --> E
    E -->|"no regressions"| F["Ship the change"]:::io
    E -->|"regressions found"| G["Block and investigate"]:::warn
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 18.1: An eval as a gate. Every prompt or model change runs against the same golden dataset and is compared with the current baseline.

Regression testing for models and prompts

This gate is regression testing, the same idea you already use for code. The difference is what triggers it. In an LLM system, a regression can come from:

  • A prompt edit. Even rewording one sentence in the system prompt.
  • A model change. Switching to a newer model, a cheaper one, or a provider silently updating the model behind the same name. Pin exact model versions where your provider allows it, so changes happen when you choose.
  • A change in tools or context. A new tool the agent can call, or a different way of retrieving files (Article 14), changes what the model sees.

Treat all three as code changes: reviewed, run through the eval suite, with results attached to the pull request. Editing prompts in production with no eval is deploying untested code.

LLM-as-judge, and its caveats

Using a model to grade a model sounds circular, and it does need care. Judges are useful for qualities that are hard to check with code, but they have known weaknesses:

  • They can prefer longer answers and answers that sound confident, even when a shorter one is better.
  • They can favour their own style. A judge from the same model family may rate that family's outputs higher.
  • They are sensitive to the rubric. "Rate quality from 1 to 10" gives noisy scores. A specific question works much better: "Does the PR description state the root cause, that emails were compared case-sensitively? Answer yes or no."

The fix is to calibrate the judge against people. Take 50 outputs, have a human grade them, have the judge grade them, and count how often they agree. If they agree on 44 of 50, that is 44 ÷ 50 = 88% agreement. If they agree on only 30, which is 60%, the judge isn't measuring what you think, and you should tighten the rubric before trusting it. Re-check the agreement whenever you change the judge model or the rubric.

Observability: seeing what the agent did

Evals tell you how often the agent succeeds. When it fails, you need to see why, and a single run can involve a dozen model and tool calls. Without a record of each one, debugging means guessing.

The standard approach comes from distributed systems: tracing. A trace is the record of one complete run. It is made of spans, one for each step, each with a start time, a duration, and details about that step. For an LLM call, the useful details are the model name, the number of input and output tokens, the cost, and the latency. For a tool call, they are the tool name, its arguments and its result. If you already use OpenTelemetry for your services, the same concepts apply, and there are conventions for recording LLM calls in it.

Here is a trace of one run of the agent on BUG-1423. The timings and token counts are illustrative, made up to show the shape of a trace.

Trace: BUG-1423 run, 33 s LLM plan search code read file LLM write fix run tests open PR 0 s 33 s

Figure 18.2: A trace of one agent run as spans on a timeline. Purple spans are model calls, teal spans are tool calls. In this illustrative run, the test suite takes most of the time.

A trace answers questions that are otherwise impossible. Why did this run take 33 seconds? Mostly the tests, not the model. Why did a run fail? Open the "read file" span and you might see the agent read auth/signup.py instead of auth/login.py.

Cost per run falls straight out of the trace. Say the run used 120,000 input tokens and 6,000 output tokens in total, and the model costs $3 per million input tokens and $15 per million output tokens. These prices are illustrative, not any provider's current rate.

  • Input: 120,000 ÷ 1,000,000 × $3 = $0.36
  • Output: 6,000 ÷ 1,000,000 × $15 = $0.09
  • Total: $0.36 + $0.09 = $0.45 per run

Input dominates because agents re-send the growing conversation on every step. Prompt caching (Article 13) exists largely for this reason.

Where this shows up in our bug-fix agent

The diagram shows how evals and observability wrap around the agent, before release and in production.

flowchart TD
    A["Golden tickets"]:::io --> B["Agent harness"]:::infra
    P["Production tickets"]:::io --> B
    B --> C["LLM and tools"]:::model
    C --> D["Traces: tokens, cost, time"]:::infra
    D --> E["Dashboards and alerts"]:::infra
    D -->|"failures become new cases"| A
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 18.3: The feedback loop. Every run is traced, and every production failure is added to the golden dataset so it can never silently come back.

In practice, for BUG-1423:

  • Before release, BUG-1423 itself becomes a golden case once it is fixed. The check is simple: the new test passes, the old tests still pass, and only auth/login.py and tests/test_login.py change.
  • During the run, every model call and tool call is a span. If the agent loops, calling "run tests" eight times, the trace shows it immediately.
  • In production, dashboards track the pass rate of the agent's PRs (how many get merged without changes), cost per ticket, and time per ticket. An alert fires if cost per run suddenly doubles, which often means the agent is stuck in a loop.
  • After a failure, the failing ticket goes into the golden dataset, so the next prompt change is tested against it.

Common misconceptions

"The new model scored higher on a public benchmark, so it will be better for us." Public benchmarks measure general tasks, not your codebase and your tickets. A model can top a benchmark and still be worse on your golden dataset. Only your own eval answers your question.

"An LLM judge is objective." It is consistent, which is not the same thing. It has biases, such as preferring longer answers, and they can point the wrong way. Measure its agreement with human reviewers before you rely on it.

"We'll add evals once the product is stable." The product becomes stable because of evals. Without them every prompt change is a guess. Start with 20 to 50 real cases on day one.

"Logging the final answer is enough." The final answer rarely tells you why a run failed. The cause is usually several steps earlier, in what the agent read or which tool it called. You need the full trace.

Key takeaways

  • An eval checks measurable properties of the output, such as tests passing and the diff staying in scope, rather than one exact answer.
  • Build a golden dataset from real cases, keep it fixed, and add every production failure to it.
  • Compare versions case by case, not just by overall pass rate, and run more than once before trusting a small difference.
  • Use code checks first, LLM judges for what code can't check, and human review to calibrate both.
  • Trace every agent run as spans with tokens, cost and latency. That is how you debug failures and control cost.

Glossary

  • Eval: a repeatable test of an AI system's output quality against a set of examples.
  • Golden dataset: a fixed set of real example inputs with a known way to judge a good result.
  • Grader: the check that decides whether one output passes, which can be code, a model, or a person.
  • Pass rate: the share of examples that pass, for example 42 of 50 is 84%.
  • Regression: a case that used to pass and now fails after a change.
  • LLM-as-judge: using a model, guided by a written rubric, to score another model's output.
  • Rubric: the written criteria a judge, human or model, uses to score an output.
  • Trace: the full record of one run of a system, made of spans.
  • Span: one step inside a trace, such as a single model call or tool call, with its timing and details.
  • Observability: the ability to understand what a system did from the data it records, such as traces, metrics and logs.

Next up

Measuring quality tells you how often the agent gets things right. It does not tell you what happens when someone tries to make it go wrong. In Article 19: Security and failure modes, we look at hallucination, prompt injection hidden in tickets and code comments, data leakage, and how to sandbox an agent so a bad decision can't do real damage.