Why I built it

I'm a platform engineer, and I wanted to learn AI engineering properly. So I built the thing platform teams end up needing anyway once a few apps start calling LLMs.

When every app talks to a provider directly, each one repeats the same work and hits the same problems. Every app carries its own provider SDK. An outage at one provider becomes an outage for your users. Nobody can say which team spent $4,000 last month, and one runaway batch job can burn the whole budget. A gateway moves all of that into one place.

What it does

Apps send requests to one endpoint, in either the OpenAI format or the Anthropic Messages format, so the OpenAI SDK, the Anthropic SDK, LangChain and Claude Code all work by changing a base URL. Instead of naming a model, an app asks for an alias such as fast, smart or local. Each alias maps to an ordered chain of models in a YAML file, so moving every app to a new model is a config change, not a deploy.

  • Reliability. Transient errors are retried with backoff, then the next model in the chain is tried. A circuit breaker per model, shared across replicas in Redis, stops traffic to a provider that is down.
  • Limits and money. Every key gets requests per minute, tokens per minute and a monthly dollar budget, and keys can belong to teams with their own budget. Each request is priced and written to a usage log in Postgres.
  • Choosing models. model: auto picks the cheapest, best or fastest model that can handle the request, and A/B variants let you test a model change on part of the traffic.
  • Extras. An optional response cache, a prompt-injection filter, and sampled LLM-as-judge scoring of answers.
  • Observability. Prometheus metrics, a Grafana dashboard defined as code, SLO burn-rate alerts, and JSON logs that never contain prompt content.

Behind the gateway it can reach Claude, OpenAI, Gemini, Grok, Mistral, DeepSeek, local models through Ollama, or any OpenAI compatible API. Claude and Ollama have been run live. The others are covered by mocked tests so far.

What I learned

Streaming changes error handling. Once the first token has reached the client, you have already sent part of an answer. Falling back at that point would splice two different models' answers together. So the gateway fails over freely before the first token and reports errors in-band after it. That one boundary shaped the router, the timeouts and the metrics.

You can't rate-limit tokens you haven't seen yet. The cost of a request is unknown until it finishes. The gateway estimates tokens up front, reserves them, and corrects to the real usage afterwards. With 50 concurrent streams racing for the last dollar of a budget, the first version overshot by 3.4%. Making the check and the hold one atomic step in Redis brought that down to 0.9%.

Measuring is harder than building. My first benchmark numbers were wrong. The mock ignored max_tokens, my load generator gave up before the gateway did, and the clients moved in lockstep. I rebuilt the rig with a mock that replays real Claude timing and paired requests so the mock's own noise cancels out.

Most of it is classic reliability work. Shared circuit breakers, dependencies that fail open, readiness checks, burn-rate alerts and graceful shutdown with open streams. None of that is specific to AI, and it turned out to be most of the work.

Results

  • About 3 ms of added latency at the median, which is 0.7% of a real model's time to first token.
  • One replica on one core holds 100 concurrent streams within 10% of a direct connection.
  • During a provider outage, 3 requests went to the dead provider before the breaker opened, and no errors reached clients.
  • Requests kept being served through provider, Redis and Postgres failures, and 97 of 97 open streams finished cleanly through a shutdown.
  • A 60 minute soak on two replicas served 259,511 requests with 100% success and flat memory.

It was built in phases, each with tests and a written decision record, and it ships as a public Docker image. Next is running it for real on a cloud deployment.