The AI Gateway Pattern

Three months after launch, your AI product's monthly invoice from OpenAI reads $47,000. Nobody knows which feature drove it. Your backend has five services, each calling the OpenAI API directly with hardcoded model strings. Last week, OpenAI deprecated the model ID two of those services use and they started returning 404s in production — at midnight. A competitor switched to Claude Sonnet because the pricing changed, cutting their inference bill by 40%. Meanwhile your team is booking a sprint to manually update the model string in each service.

This is the unglamorous reality of running LLMs in production without a gateway. None of these problems are hard. They are just expensive to fix retroactively.

What an AI gateway actually is

An AI gateway is a stateless proxy service that sits between your application code and every LLM provider API. Your services call the gateway; the gateway calls OpenAI, Anthropic, Mistral, or a self-hosted model, depending on routing rules you define in configuration. From the application's perspective, every LLM call goes to the same endpoint: http://gateway/v1/chat/completions with a logical model name.

This is the same pattern as the API gateway in microservices — but with LLM-specific extensions that a generic API gateway doesn't handle:

• Token cost tracking rather than just request counting
• Semantic deduplication of queries, not just exact-match caching
• Provider-specific error normalisation (every provider has different error codes, rate-limit headers, and retry semantics)
• Streaming passthrough of SSE chunks without buffering
• Model alias resolution from logical names to real provider model IDs

The difference from just sticking Nginx in front of your API calls is those last five items. A generic reverse proxy routes requests; a gateway understands what LLM requests are.

Model aliases: the feature that pays for the gateway on day one

Every LLM provider has deprecated model IDs that were previously in wide use. OpenAI has retired gpt-4-0613, gpt-4-32k, text-davinci-003, and gpt-3.5-turbo-0301. Anthropic similarly cycled through claude-1, claude-2, and specific dated snapshots. This is normal product behavior — models improve, old versions get sunset.

If your five services each hardcode "model": "gpt-4o-2024-08-06", a deprecation becomes a five-service code change, five pull requests, five deployments, ideally coordinated — but usually not, because someone is on PTO.

With model aliases, application code calls:

response = openai_client.chat.completions.create(
    model="chat-standard",  # logical alias, not a provider model ID
    messages=[{"role": "user", "content": user_message}]
)

The gateway resolves chat-standard from a config that looks like:

models:
  chat-standard:
    primary:
      provider: openai
      model: gpt-4o-mini
    fallback:
      - provider: anthropic
        model: claude-sonnet-4-5
      - provider: together_ai
        model: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo

  chat-premium:
    primary:
      provider: anthropic
      model: claude-opus-4-5
    fallback:
      - provider: openai
        model: gpt-4o

  embeddings-v1:
    primary:
      provider: openai
      model: text-embedding-3-small

Migrating from GPT-4o-mini to a new model means changing one line in this YAML and reloading the gateway. No application code changes. No coordinated deploys. This alone justifies the gateway for any team with more than two services.

Fallback chains and circuit breakers

LLM providers have outages. OpenAI's status page shows incidents almost every month; Anthropic's is similar. A provider degradation at 2am hitting your production traffic is not a hypothetical.

sequenceDiagram
    participant APP as Application
    participant GW as AI Gateway
    participant OAI as OpenAI
    participant ANT as Anthropic

    APP->>GW: POST /v1/chat/completions (model: chat-standard)
    GW->>OAI: forward request
    OAI-->>GW: 503 Service Unavailable
    GW->>OAI: retry 1 (100ms backoff)
    OAI-->>GW: 503 Service Unavailable
    GW->>OAI: retry 2 (200ms backoff)
    OAI-->>GW: 503 Service Unavailable
    Note over GW: retries exhausted, circuit opens
    GW->>ANT: failover to fallback provider
    ANT-->>GW: 200 OK, response body
    GW-->>APP: 200 OK (+ X-Gateway-Provider: anthropic header)
    Note over APP: app saw extra latency, no error

The gateway's fallback logic:

1. Attempt the primary provider.
2. On 5xx or timeout, retry with exponential backoff up to N times (configurable — typically 2-3 retries).
3. After retries exhausted, open the circuit breaker for that provider for a cooldown window (30-120 seconds).
4. Route to the next provider in the fallback list.
5. Log the failover event with provider, error code, and latency impact.

The application receives a response with slightly elevated latency. It does not see an error unless every provider in the fallback chain is simultaneously down — which would be newsworthy.

Circuit breakers matter here. Without them, the gateway continues hammering a degraded provider on every request until it recovers, compounding the latency impact. With a circuit breaker open, requests skip the failing provider immediately and go straight to the fallback — adding maybe 5ms routing overhead instead of a 10-second timeout per request.

Semantic caching

Exact-match caching is simple: hash the request body, return the cached response if the hash matches. But LLM applications generate queries with natural language variation — "what is your refund policy", "how do refunds work", "can I get my money back" are semantically identical questions that an exact-match cache treats as three different requests.

Semantic caching embeds the incoming query, searches a vector store for similar past queries, and returns the cached response if the similarity score exceeds a threshold.

# Pseudocode for semantic cache lookup in the gateway
def semantic_cache_lookup(query: str, threshold: float = 0.95) -> str | None:
    query_embedding = embed(query)  # ~5ms with a fast embedding model
    results = vector_store.search(query_embedding, top_k=1)
    if results and results[0].score >= threshold:
        return results[0].cached_response
    return None  # cache miss, call the LLM

def handle_request(request: ChatRequest) -> ChatResponse:
    cached = semantic_cache_lookup(request.last_user_message)
    if cached:
        return ChatResponse(content=cached, source="cache")

    response = call_provider(request)
    store_in_cache(request.last_user_message, response.content)
    return response

Back-of-envelope on cache savings.

FAQ support bot, 100k queries/day
Query diversity: assume 40% are semantically equivalent to a prior query
Cache hit rate: 40% × 0.85 (precision at threshold 0.95) ≈ 34%

Input tokens per query: avg 500 tokens × $0.15/1M (GPT-4o-mini, mid-2026 illustrative)
Cost without cache: 100k × 500 × $0.15/1M = $7.50/day = $225/month

Cache hit savings: 34% × $225 = $76.50/month
Embedding cost to check cache: 100k × 50 tokens × $0.02/1M = $0.10/day ≈ $3/month
Net saving: ~$73.50/month on input tokens alone.
Output token savings are proportionally larger.

For a high-diversity workload — code generation against unique user repos, creative writing — expect under 10% cache hit rates. The embedding overhead without cache hits costs money for no benefit. Know your query distribution before committing to semantic caching as your primary cost lever.

The threshold matters. At 0.90 similarity you'll get cache hits on queries that are semantically related but not actually equivalent, serving wrong answers confidently. At 0.98 you miss obvious matches. A threshold of 0.94-0.96 is a reasonable starting point; calibrate against a sample of your real query pairs.

{ "type": "token-cost", "title": "Gateway cost impact: with vs without semantic caching" }

Unified cost attribution

Without a gateway, understanding your LLM spend requires grepping logs across every service and correlating token counts from provider invoices. This is painful enough that most teams don't do it rigorously — and then wonder why the bill keeps growing.

A gateway solves this by requiring every request to carry attribution headers:

POST /v1/chat/completions HTTP/1.1
Host: ai-gateway.internal
Content-Type: application/json
X-Team: platform-search
X-Feature: query-expansion
X-User-Tier: free
X-Request-Id: 550e8400-e29b-41d4-a716-446655440000

The gateway extracts these headers, records the token counts from the provider response, and writes a cost event to a metrics store:

{
  "timestamp": "2026-07-02T14:23:01Z",
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "team": "platform-search",
  "feature": "query-expansion",
  "user_tier": "free",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "alias": "chat-standard",
  "input_tokens": 512,
  "output_tokens": 84,
  "cache_hit": false,
  "latency_ms": 843,
  "cost_usd": 0.0000961
}

From this stream, you can build dashboards that show daily spend by team, by feature, by user tier. When the search team's spend doubles week-over-week, you know immediately — before the bill arrives.

Budget alerts follow naturally: set a daily token budget per team and fire an alert when 80% is consumed. This is the LLM equivalent of AWS billing alerts, and it's nearly impossible to implement correctly without a centralised gateway receiving every request.

Attribution also fixes key custody. Without a gateway, every service holds its own copy of the provider API keys — five services, five places a key can leak, five rotations when one does. With a gateway, the real provider keys live only in the gateway's environment. Services authenticate with gateway-issued virtual keys: one per team or service, revocable individually, each carrying its own rate limit and budget. LiteLLM's proxy supports this directly — litellm_params hold the provider keys, and you mint virtual keys with per-key max_budget and rpm_limit/tpm_limit. When the search team's batch job goes rogue at 3am and starts sending 200 requests per second, the gateway throttles that key with a 429 before the provider throttles your entire org-level account — which would take every other team down with it. Per-team rate limiting at the gateway is the difference between one team having a bad night and everyone having one.

Model routing: sending the right query to the right model

Not every query needs a frontier model. A user asking "what are your store hours?" doesn't need Claude Opus 4 at $15/1M input tokens — GPT-4o-mini at $0.15/1M does the job. A user asking "review this 3,000-line PR and identify security vulnerabilities" probably does need the frontier model.

The gateway can implement routing rules based on request metadata:

flowchart TD
    REQ[Incoming request] --> CLASS{Complexity classifier}
    CLASS -->|"simple: short input,\nlow stakes"| CHEAP[Cheap model tier\ngpt-4o-mini / Haiku]
    CLASS -->|"medium: moderate\ncomplexity"| MID[Mid-tier model\nClaude Sonnet]
    CLASS -->|"complex: long context,\nhigh stakes, reasoning"| FRONTIER[Frontier model\nClaude Opus / GPT-4o]
    CHEAP --> RESP[Response]
    MID --> RESP
    FRONTIER --> RESP

    style CLASS fill:#0e7490,color:#fff
    style CHEAP fill:#15803d,color:#fff
    style MID fill:#0e7490,color:#fff
    style FRONTIER fill:#374151,color:#fff

Routing signals the gateway can use:

• Input token count: requests above a threshold get routed up the tier.
• Feature flag or endpoint: requests from /api/code-review always use the premium tier.
• User tier: free users get the cheap tier; paid users get the frontier model.
• Explicit override in request header: X-Model-Tier: premium.

{ "type": "model-router", "title": "Routing traffic across model tiers: cost vs quality tradeoff" }

Model routing at production scale has achieved 85% cost reductions for applications where the query mix skews toward simpler tasks. The sibling article on model routing and cascades goes deep on the algorithms; the gateway is the infrastructure that makes routing decisions actionable without application code changes.

What a gateway does not do

This matters because scope creep is how gateways become bottlenecks.

A gateway should not contain business logic. It does not know what a "good" response looks like for your specific use case. It does not implement your domain-specific guardrails (those belong in an output validation layer closer to your application, covered in the reference architecture). It does not evaluate response quality — that is LLM observability.

A gateway should not buffer streaming responses to apply post-processing. If you need to inspect or transform streaming output, do that in your application layer. A gateway that buffers streaming output before forwarding adds the full generation latency as perceived latency — defeating the point of streaming entirely.

What breaks

The cache serves a stale or wrong answer

Semantic caching's main failure mode is returning a cached response that is no longer accurate. Your company changes its refund policy on Monday; queries semantically similar to "what is your refund policy" get the pre-Monday cached answer for the cache TTL window. Fix this with explicit cache invalidation on knowledge base updates (call cache.invalidate(tag="refund-policy") from your content management system webhook) and a sensible TTL — 24 hours for most support content, hours for anything time-sensitive, and no caching at all for personalised queries that carry user context.

At the wrong similarity threshold, the cache also serves answers to related but not identical questions. A query about "returning a defective product" should not be answered with the cache hit for "cancelling a subscription," even if they score 0.91 similarity. Set your threshold empirically, not theoretically.

Fallback providers produce inconsistent output format

Your application parses a specific JSON structure from the primary provider. The fallback provider returns a different token for the same request — or more likely, returns structured output in a slightly different form because it handles your function-calling schema differently. This causes silent parse failures in the application layer.

Test your output parsing against the fallback model's responses before you rely on it in production. Fallback chains that have never been exercised are not fallback chains — they are confidence in a plan that has not been tested.

Cost attribution headers are missing

One service ships a change that drops the X-Team header on certain request paths. The gateway routes the request successfully, but the cost goes to an unknown bucket. Over time, unknown is 30% of your spend, the dashboard loses credibility, and you're back to guessing. Enforce required headers at the gateway with a 400 rejection if attribution is absent — painful for the team that forgot, but it keeps your cost data clean. Treat missing attribution like a linting error.

Gateway becomes a single point of failure

A single gateway instance in front of all LLM traffic is an outage at the worst possible time. Run multiple instances behind a load balancer, deploy across availability zones, and make the gateway genuinely stateless — routing config lives in a config file or etcd, not in-process memory. The semantic cache can be a shared Redis or Qdrant instance that all gateway instances connect to. Any instance can handle any request.

Streaming breaks under load

Many gateway implementations handle non-streaming requests fine but lose SSE chunks under high concurrency because the framework is buffering to apply middleware. Test your streaming path explicitly under load. A gateway that converts streaming responses to buffered responses is not serving streaming requests — it is serving delayed responses.

flowchart LR
    APP[Application] -->|"POST /chat\n(stream: true)"| GW[Gateway]
    GW -->|"forward with stream:true"| PROV[Provider]
    PROV -->|"SSE chunk 1"| GW
    GW -->|"SSE chunk 1 (passthrough)"| APP
    PROV -->|"SSE chunk 2"| GW
    GW -->|"SSE chunk 2 (passthrough)"| APP
    PROV -->|"SSE [DONE]"| GW
    GW -->|"SSE [DONE] + cost log"| APP
    Note1["Gateway logs cost\nafter [DONE] marker\n— never buffering"] -. annotation .-> GW

    style GW fill:#0e7490,color:#fff
    style PROV fill:#374151,color:#fff

The stream dies halfway

The fallback chain in the sequence diagram above only works because OpenAI failed before the first byte. The harder case: the provider accepts the request, streams 400 tokens of a 900-token answer, then drops the connection. The gateway cannot transparently fail over now — the client already has half an answer, and the fallback model would generate a different half. Silently stitching two models' outputs together is worse than an error.

Decide the policy explicitly. Either the gateway surfaces the truncated stream as an error and the application retries from scratch (discarding the partial output client-side), or you accept that failover is a before-first-token guarantee and document it that way. Cost logging is affected too: a truncated stream may never deliver the final usage frame, so the gateway must either count streamed tokens itself or flag the record as partial — otherwise your attribution data quietly undercounts exactly the requests that occurred during provider incidents, which is when you most want accurate numbers.

Three realistic options.

LiteLLM is the most widely deployed open-source gateway. It normalises the API surface across 100+ providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Together, Groq, Ollama, and more), handles retries and fallbacks, exposes a Prometheus metrics endpoint, and supports model aliases out of the box. Self-hosted, so your prompts and responses never leave your infrastructure. Operational overhead: you run it and patch it.

# LiteLLM proxy, minimal config
pip install litellm[proxy]
litellm --config litellm_config.yaml

# litellm_config.yaml (simplified)
model_list:
  - model_name: chat-standard
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-standard-fallback   # separate group — fallback target
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  num_retries: 3
  fallbacks: [{"chat-standard": ["chat-standard-fallback"]}]

general_settings:
  request_timeout: 30

Managed options — Portkey, Cloudflare AI Gateway, Kong AI Gateway — handle the operational burden and add UI dashboards for cost and observability. Portkey targets AI-native teams with semantic caching and prompt management built in. Cloudflare AI Gateway is attractive if you are already in their ecosystem; it sits at their edge with minimal added latency. Kong AI Gateway extends their existing API gateway product with LLM-specific plugins. These make sense when your team does not want to operate infrastructure.

Build your own if you have compliance requirements that prevent SaaS tooling (healthcare, finance, government), or if your routing logic is genuinely too complex to express in any of the above. A minimal gateway is a FastAPI or Express service with a routing table and a retry loop — maybe a week to build something that handles the basics. A production-grade one that handles streaming correctly, has semantic caching that doesn't serve wrong answers, and has tested fallback behavior takes 4-6 weeks. That engineering time has an opportunity cost.

Option	Setup time	Operational overhead	Data leaves your infra?	Semantic cache
LiteLLM (self-hosted)	4-8 hours	Medium (you run it)	No	Plugin (Redis-backed)
Portkey	1-2 hours	Low (managed)	Yes (their servers)	Built-in
Cloudflare AI Gateway	1-2 hours	Low (managed)	Yes (Cloudflare edge)	Built-in
Kong AI Gateway	4-8 hours	Medium	Depends on deploy	Plugin
Build your own	1-6 weeks	High	No	You build it

Deploying the gateway in practice

You have two deployment topologies:

Centralized fleet: a small cluster of gateway instances (2-4 nodes) behind a load balancer, shared by all application services. Simpler to operate, easier to monitor, easier to enforce consistent behavior. Adds one network hop between services and the gateway. This is the right choice for most teams.

Sidecar: a gateway instance runs alongside each application service, often as a container in the same pod. Removes the extra network hop; each service has its own scaling unit. Correct for high-traffic latency-sensitive workloads. Harder to operate because you now have N gateway instances to update when routing config changes — use a config push mechanism (etcd, Consul, a shared config map) so all instances pick up changes without restarts.

For the centralized fleet, the gateway itself should add no more than 5-10ms to the critical path under normal load. If it's adding more, profile the middleware chain — semantic cache lookup with a remote vector store is the usual culprit.

When to add a gateway

The honest answer: day one, or as close to it as possible. The retrofitting cost — identifying every service making direct LLM calls, changing clients, testing, deploying — grows linearly with the number of services and the age of the codebase. Adding it on day one is a half-day task. Adding it when you have fifteen services and a provider forcing a migration is a painful sprint.

If you are past day one and operating without a gateway, the right trigger is the first time you hit any of: a surprise LLM bill you cannot explain by feature, a provider outage that takes down a user-facing product, a forced model migration, or a compliance audit asking where your prompt data goes. All of these have happened to teams that thought the gateway was "something to add later."

The prompt versioning and CI/CD article covers the next layer up: once all your LLM calls go through a single point, you can gate prompt changes on automated evaluation before they reach production — another discipline that is much easier when traffic is centralised. And when you need to measure whether your gateway's routing is actually improving outcomes, A/B testing prompts and models describes how to run statistically valid experiments on real traffic without exposing all users to an untested variant.

The gateway is not the most interesting part of your AI product. It's infrastructure. But it's the infrastructure that determines whether a provider's 2am incident is a minor latency blip in your dashboards or a production outage in your support tickets.