The first AI feature in a company usually calls a model provider directly. By the fifth feature, each app has its own API keys, its own retry logic, no shared view of cost, and an outage at one provider takes several products down at once. An LLM gateway is the piece of infrastructure that fixes this, and if you have run an API gateway or a service mesh, most of the ideas will feel familiar.

What a gateway is

An LLM gateway is a service that sits between your applications and the model providers. Applications send every model request to the gateway instead of to a provider. The gateway decides where the request goes, applies company rules, and passes the response back.

Think of it as a reverse proxy that understands LLM traffic. Because it sees every request and every response, it is the natural place to put everything that should be the same for every app:

  • Routing: which model and provider serves this request.
  • Reliability: retries, fallbacks and circuit breakers when a provider fails.
  • Caching: avoiding paying twice for the same work.
  • Limits and cost control: rate limits and budgets per team or app.
  • Logging and audit: who used what, when, at what cost.
  • Privacy and safety: filtering personal data and risky inputs.

The diagram shows where the gateway sits.

flowchart TD
    A["Bug-fix agent"]:::io --> G["LLM gateway"]:::infra
    B["Support chatbot"]:::io --> G
    G --> P["Providers: A, B, local"]:::model
    G -.-> L["Usage and cost log"]:::infra
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 17.1: Every app talks to one gateway. The gateway talks to every provider and records every call.

Routing and aliases

The simplest useful feature is an alias. Instead of hard-coding a model name, an app asks for something like smart or fast. The gateway's configuration maps each alias to an actual model.

That indirection pays off quickly. When a better or cheaper model is released, you change one line of gateway config and every app moves at once, with no code deploys. You can also route by rule: send short classification requests to a small cheap model and long reasoning requests to a large one, or keep requests containing sensitive data on a model you host yourself (Article 12).

Most gateways expose a single API format, commonly the OpenAI-style chat format, and translate it into each provider's format behind the scenes. Apps write against one API and the providers become swappable.

Reliability: retries, fallbacks and circuit breakers

Model providers fail like any other remote service. They time out, return "overloaded" errors, or have outages. A gateway handles this in layers.

Retries. For a temporary error, wait briefly and try again. Good retries use exponential backoff with jitter: wait a little, then longer, with some randomness so that thousands of clients do not all retry at the same instant.

Fallbacks. Each alias maps to an ordered chain, for example Provider A's large model, then Provider B's, then a local model. If the first fails after retries, the gateway tries the next. The app gets an answer from a different model instead of an error.

Circuit breakers. If a provider has failed, say, five times in a row, there is no point sending it more traffic and making every user wait for a timeout. A circuit breaker "opens" and sends requests straight to the fallback for a cooling-off period, then lets a few test requests through to see if the provider has recovered.

Streaming (Article 8) adds one subtlety. Once the first tokens of an answer have reached the user, you cannot silently switch to another model, or the user would get half of one answer stitched to half of another. So gateways fall back freely before the first token, and after that report the error instead.

The diagram shows a fallback during a provider outage, step by step.

flowchart TD
    A["Agent asks for smart"]:::io --> G["Gateway tries primary"]:::infra
    G --> P["Provider A: 529 overloaded"]:::warn
    P --> F["Gateway falls back"]:::infra
    F --> Q["Provider B answers"]:::model
    Q --> R["Agent gets answer"]:::io
    classDef model fill:#2a2140,stroke:#a78bfa,color:#ece5ff
    classDef infra fill:#12302d,stroke:#2dd4bf,color:#d5fbf5
    classDef io fill:#1f1d1a,stroke:#8a8275,color:#ece5d8
    classDef warn fill:#3a1d18,stroke:#fb7a5e,color:#ffe1d9

Figure 17.2: The agent asked once and got an answer. The outage stayed inside the gateway.

Caching

There are two different kinds of caching worth separating.

Prompt caching happens at the provider. If many requests share a long identical beginning, like a big system prompt or a codebase summary, the provider can reuse its internal work on that prefix (the KV cache from Article 8) and charge less for it. Article 13 covers this. The gateway's role is to keep prompts structured so the shared part comes first, and to account for the cheaper cached tokens correctly.

Response caching happens at the gateway. If the exact same request arrives again, the gateway returns the stored answer without calling a model at all. A semantic cache goes further, using embeddings (Article 3) to return a stored answer for a question that is worded differently but means the same. Response caching suits FAQ-style traffic. It rarely helps an agent, whose every request includes a unique, growing history.

Rate limits and budgets

Without limits, one runaway batch job or one looping agent can exhaust your provider quota or your monthly budget.

Gateways issue their own API keys, one per app or team, and attach limits to each: requests per minute, tokens per minute, and a monthly spending cap. Token limits are tricky because you do not know how many tokens a request will use until it finishes. A common approach is to estimate up front from the prompt size and the maximum output allowed, reserve that amount, and correct to the real usage afterwards.

Worked example: what does one bug fix cost?

Here is how a gateway prices a request. The prices below are illustrative round numbers, not any provider's current price list. Real prices change often and vary by model.

  • Input tokens: $3 per million
  • Output tokens: $15 per million

The formula for one call is:

cost = (input tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price)

Suppose the BUG-1423 run makes 10 model calls (as traced in Article 16), averaging 8,000 input tokens and 400 output tokens each.

Input: 10 × 8,000 = 80,000 tokens. 80,000 ÷ 1,000,000 × $3 = $0.24

Output: 10 × 400 = 4,000 tokens. 4,000 ÷ 1,000,000 × $15 = $0.06

Total: $0.24 + $0.06 = $0.30 for the whole fix.

Now the budget. If the platform team gives the agent's key a $300 monthly budget, that covers $300 ÷ $0.30 = 1,000 runs like this one. If a bug in the agent makes it loop 200 times on a single ticket, the gateway's per-key budget and the harness's step limit (Article 16) stop it from quietly burning the month's money.

Notice that input dominates. Agents resend their whole history every step, which is why prompt caching matters so much for them.

Logging and audit

Because every call passes through it, the gateway can write one usage record per request: which key, which model, how many tokens, the cost, the latency, the status, and a request ID that links to application logs. That turns "the AI bill went up" into "the support bot's key doubled its token use on Tuesday".

Whether to log the prompts and responses themselves is a real decision. Full content is invaluable for debugging and evaluation (Article 18), but it is also a store of potentially sensitive data. Many teams log metadata by default and capture content only for sampled or opted-in traffic.

PII filtering

PII (personally identifiable information) means data that identifies a person: names, email addresses, phone numbers, ID numbers. Sending it to a third-party model provider may break your privacy commitments or regulations (Article 20).

A gateway can scan requests before they leave and replace PII with placeholders, using pattern matching for things like emails and card numbers, and sometimes a small model to spot names. It can also restore the original values in the response if needed.

Our running example shows why this needs care. The ticket contains a real customer's address, Maria@Example.com. Redacting it to [EMAIL_1] protects Maria, but it also hides the uppercase letters, which are the whole point of the bug. A smarter rule replaces it with a fake address of the same shape, like Xxxxx@Xxxxxxx.com, so the model still sees the pattern without seeing the person.

Building or buying one

There are open-source gateways, commercial ones, and gateway features built into cloud platforms. As of October 2026, widely used options include LiteLLM, Portkey, Kong's AI gateway features and Cloudflare AI Gateway, among others. I built one myself as a learning project, an open-source LLM gateway in Python with fallback chains, shared circuit breakers, per-key budgets and cost tracking. In its benchmarks it added about 3 ms of latency per request, and during a simulated provider outage only 3 requests reached the dead provider before the breaker opened, with no errors reaching clients.

Whichever you choose, the important decision is architectural: make the gateway the only way apps reach models, so the rules apply everywhere.

Where this shows up in our bug-fix agent

The agent's harness points at the gateway instead of a provider, using its own gateway key and the alias smart.

  • When Provider A is overloaded halfway through the run, the gateway retries, then falls back to Provider B. The agent never notices.
  • Every one of its ten calls is priced and logged against its key, so the team can see what BUG-1423 cost.
  • Its monthly budget caps the damage if it ever loops.
  • The PII filter masks Maria's real address while keeping its shape.
  • The platform team can move the agent to a newer model by editing gateway config, with no change to the agent's code.

Common misconceptions

"A gateway is just a proxy, so it adds a lot of latency." A well-built gateway adds milliseconds. Model responses take hundreds of milliseconds to many seconds, so the overhead is usually a tiny fraction.

"Fallbacks mean identical answers." A fallback model is a different model. Answers will differ in style and sometimes quality. Fallback chains should be chosen and tested, not assembled from whatever is available.

"Caching will cut our agent costs a lot." Response caching rarely helps agents, because every request is unique. Prompt caching of the shared prefix is what helps.

"If we use a gateway, privacy is handled." A gateway is a good enforcement point, but filters miss things. It is one layer, alongside data agreements with providers and not sending sensitive data in the first place.

Key takeaways

  • An LLM gateway is a single front door between all your apps and all model providers.
  • Aliases decouple apps from model names, so swapping models becomes a config change.
  • Retries, fallback chains and circuit breakers turn provider outages into slightly different answers instead of errors.
  • Per-key rate limits and budgets, with per-request cost records, make spending visible and bounded.
  • The gateway is the natural place to log usage and filter PII, but each is a trade-off, not a free switch.

Glossary

  • LLM gateway: a service between applications and model providers that routes, limits, logs and filters model requests.
  • Alias: a stable name like smart that the gateway maps to an actual model.
  • Fallback chain: an ordered list of models to try if earlier ones fail.
  • Exponential backoff with jitter: retrying after increasing waits, with randomness so clients do not retry in sync.
  • Circuit breaker: a mechanism that stops sending traffic to a failing provider for a cooling-off period.
  • Response cache: stored answers returned for repeated identical requests.
  • Semantic cache: a response cache that matches requests by meaning using embeddings.
  • Rate limit: a cap on requests or tokens per unit of time.
  • Budget: a cap on spending per key, team or period.
  • PII (personally identifiable information): data that identifies a person, such as an email address.
  • Redaction: replacing sensitive data with placeholders before it leaves your systems.

Next up

The gateway tells you what every call cost and how long it took. It does not tell you whether the answers were any good. In Article 18: Evals and observability, we look at how to measure quality, catch regressions when you change a prompt or a model, and trace an agent run step by step to see exactly where it went wrong.