The LLM Gateway Pattern: Rate Limiting, Cost Attribution, and Observability

Here is a pattern I have watched unfold at multiple companies. Six product features — Slack bot, PDF summarizer, customer support assistant, batch report generator, plus two more — share one API key and one application. A $22,000 monthly invoice arrives. Nobody can tell which feature drove it. An engineer in the dev environment was running 8,000-token prompts for every test, logging turned on, generating on gpt-4o. That engineer did not know the cost. There was no way for anyone to know.

The gateway pattern exists to make "there was no way to know" impossible.

Why a dedicated gateway layer

The naive approach — each service holds its own provider key and calls the API directly — works fine at one service and one team. It stops working when:

1. You have more than one team and need to enforce separate spend budgets without sharing the same key.
2. You need to rotate a compromised key and discover it is hardcoded in six services across three repos.
3. A provider goes down and you want fallback to happen transparently, not after six separate services are updated.
4. Finance asks for a cost breakdown by product feature, and the answer is "we don't have that."
5. You want to enforce rate limits to stay within a provider tier without building rate-limiting logic in every service.

The LLM gateway solves all five by being the only thing that holds the real provider credential, enforcing limits before calls go out, and tagging every request with the metadata needed for cost attribution.

The gateway is not a new idea. The API gateway pattern in the LLMOps section covers the broader architectural picture. This article is specifically about the operational mechanics: how rate limiting actually works, what to put on every span, and how to choose between the three build-vs-buy options.

Virtual keys and credential management

The gateway issues one virtual key per team, per application, or per environment — whatever granularity you need. Application code authenticates to the gateway with that virtual key. The gateway holds the real provider credential and swaps it in before forwarding the call.

# application code — never sees the real OpenAI key
import openai

client = openai.OpenAI(
    api_key="gw-vk-team-backend-prod-a1b2c3d4",  # virtual key issued by gateway
    base_url="https://llm-gateway.internal/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this document: ..."}],
    extra_headers={
        "X-Feature": "doc-summarizer",
        "X-User-Id": user_id,
    },
)

# gateway side (LiteLLM config, simplified)
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY  # real key, only in gateway env

general_settings:
  master_key: os.environ/GATEWAY_MASTER_KEY
  database_url: os.environ/GATEWAY_DB_URL

Virtual keys are not static config. LiteLLM creates them through its /key/generate endpoint, backed by the gateway database, which is what lets you issue and revoke them at runtime:

# issue a virtual key scoped to the backend team
curl https://llm-gateway.internal/key/generate \
  -H "Authorization: Bearer $GATEWAY_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "backend",
    "max_budget": 5000,
    "tpm_limit": 200000,
    "rpm_limit": 500,
    "models": ["gpt-4o", "claude-sonnet-4-5"]
  }'

When the virtual key is compromised — a developer accidentally commits it, a config file leaks — you revoke it in the gateway and issue a new one. The real provider key is unchanged. No credential rotation across six services. That change takes thirty seconds.

The spend budget field is the other benefit. Once a team's virtual key has hit its monthly limit, the gateway returns a 429 with a clear message. The team gets a budget conversation rather than a surprise on the company credit card.

Rate limiting: TPM + RPM, not one or the other

Most engineers think of LLM rate limits as a single number — tokens per minute. Provider dashboards show TPM prominently. But TPM alone leaves a gap.

Consider an attacker (or a buggy retry loop) sending 10,000 requests in one minute, each with a 1-token prompt: "Hi." TPM consumed: ~10,000 tokens — almost nothing against a million-token-per-minute limit. RPM consumed: 10,000 requests — potentially exhausting the provider's per-minute request limit and incurring costs from response overhead and connection setup even if token volume is tiny.

Dual enforcement at the gateway closes both dimensions:

flowchart TD
    REQ[Incoming request] --> COUNT["INCRBY estimated tokens + INCR request count\n(one atomic Redis pipeline)"]
    COUNT --> OVER{"Over TPM or\nRPM limit?"}
    OVER -->|Yes| RATE_429["429 rate limited\nRetry-After header set"]
    OVER -->|No| FORWARD[Forward to provider]
    style RATE_429 fill:#ff2e88,color:#111
    style COUNT fill:#15803d,color:#fff

The implementation in Redis uses two keys per virtual key, one for the minute-window token count and one for the minute-window request count, both with a 60-second TTL:

import redis
import time

def check_rate_limits(
    r: redis.Redis,
    virtual_key: str,
    estimated_tokens: int,
    tpm_limit: int,
    rpm_limit: int,
) -> tuple[bool, str]:
    now = int(time.time())
    window = now // 60  # 1-minute fixed window

    tpm_key = f"tpm:{virtual_key}:{window}"
    rpm_key = f"rpm:{virtual_key}:{window}"

    pipe = r.pipeline()
    pipe.incrby(tpm_key, estimated_tokens)
    pipe.expire(tpm_key, 60)
    pipe.incr(rpm_key)
    pipe.expire(rpm_key, 60)
    results = pipe.execute()

    current_tpm = results[0]
    current_rpm = results[2]

    if current_tpm > tpm_limit:
        return False, f"TPM limit exceeded ({current_tpm}/{tpm_limit})"
    if current_rpm > rpm_limit:
        return False, f"RPM limit exceeded ({current_rpm}/{rpm_limit})"
    return True, ""

The order matters: increment first, then compare. Check-then-deduct reads the counter and writes it in two steps, and under concurrency two simultaneous requests can both pass the check before either deducts. Increment-then-check keeps the counter honest — at the cost that rejected requests still consume a slot from the minute's budget, which is the right trade for a limiter.

A sliding window (using a sorted set with request timestamps) gives smoother enforcement but costs more Redis ops — one ZADD + ZREMRANGEBYSCORE + ZCARD per check versus a pipeline of INCRBYs. For most production gateways, fixed-window is fine; the boundary edge case (a burst that straddles two minute boundaries) is worth accepting in exchange for simpler, faster Redis operations.

The estimated_tokens value before you actually make the call is necessarily approximate. You can count the input tokens exactly (tiktoken for OpenAI, Anthropic's counting API), then deduct actual output tokens after the call completes. For rate limiting purposes the approximation is acceptable — the goal is to prevent runaway spend, not to count every token with billing precision.

Cost attribution: tagging as a first-class requirement

A cost rollup that says "total: $22,000" is not useful. A rollup that puts a dollar figure next to every feature and environment tells you exactly where to focus — the worked example at the end of this article shows what that looks like for this exact bill. Getting there requires that every request carries the tags before it hits the provider.

The canonical minimum tag set:

Tag	Why
user_id or session_id	Per-user cost breakdown; identify power users whose usage subsidizes the rest
feature	Per-feature cost; the primary optimization target
environment	Separate prod, staging, and dev; dev experiments should not inflate prod cost numbers
model	After fallback, the actual model used may differ from the requested model
routing_decision	If you use a router, record why each query went where it did

The gateway adds these to the span on every call. Application code passes them as custom headers (as shown in the code example above), and the gateway validates their presence — refusing requests that lack them during a grace period, then blocking them after. Yes, this requires discipline from callers. Yes, it is worth the friction; the alternative is six months of cost data you cannot use.

Downstream, tools like Langfuse and Braintrust ingest these tags natively. You get cost-by-feature dashboards without writing any aggregation code.

Retries, jitter, and provider fallback

Retries and fallbacks belong in the gateway, not in individual callers, for the same reason rate limiting does: implementing them once and getting them right once is better than implementing them many times and getting them wrong several times.

The retry rules are not complicated, but they matter:

• Retry on 429 and 5xx only. Never retry 400-level errors except 429 — a 400 Bad Request or 401 Unauthorized will not succeed on the second attempt.
• Use exponential backoff with full jitter: sleep(random(0, min(cap, base * 2^attempt))). Fixed delays create thundering herds when many clients hit a limit simultaneously and all retry at the same interval. The jitter spreads them out.
• Honor the Retry-After header in 429 responses — providers set it to tell you exactly how long to wait.

See Fallbacks, Retries, and Circuit Breakers for the full circuit breaker pattern. The gateway is the right place to implement that state machine because it has the global view of failure rate across all callers.

Provider fallback is the second layer:

# LiteLLM fallback config
router_settings:
  fallbacks:
    - {"gpt-4o": ["claude-sonnet-4-5", "gpt-4o-mini"]}
  context_window_fallbacks:
    - {"gpt-4o": ["claude-sonnet-4-5"]}  # 200k context vs 128k
  retry_after: 0.2
  num_retries: 3
  allowed_fails: 3
  cooldown_time: 60

One trap with fallback: context window mismatches. If your primary model has a 128k context window and your fallback model has a 32k window, a request with a 100k prompt will fail the fallback silently or with a confusing error. The gateway must check prompt length against the fallback model's context limit before routing there, and either truncate intelligently or fail loudly. This is one of the trickier things to get right when building your own gateway versus using a managed one that handles it.

What to put on every observability span

The gateway emits one trace span per request. That span is the primary artifact for every cost and reliability investigation. Here is what it must contain:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "timestamp": "2026-05-28T14:23:01.123Z",
  "duration_ms": 1847,
  "ttft_ms": 312,
  "virtual_key": "gw-vk-team-backend-prod",
  "feature": "doc-summarizer",
  "user_id": "usr_8f3a2b",
  "environment": "prod",
  "model_requested": "gpt-4o",
  "model_used": "claude-sonnet-4-5",
  "provider": "anthropic",
  "routing_decision": "fallback:openai_degraded",
  "input_tokens": 4217,
  "output_tokens": 389,
  "cache_hit": false,
  "estimated_cost_usd": 0.00612,
  "http_status": 200,
  "retry_count": 1
}

model_requested versus model_used matters. After a fallback, they differ. If you only log model_used, you lose the signal that your primary provider had a degradation event, and your cost rollup by model is wrong.

ttft_ms (time to first token) is the latency metric that actually affects user experience for streaming responses. Total duration tells you about server-side cost; TTFT tells you what the user felt. For interactive use cases, TTFT is the number to optimize. See Streaming LLM Responses for why the two metrics diverge and how to instrument them separately.

{ "type": "prompt-cache", "title": "Cache hit/miss spans: where caching shows up in cost", "scenario": "chatbot" }

Build vs buy: LiteLLM, Portkey, or your own

Three realistic options in 2026:

LiteLLM (Apache 2.0, self-hostable) is the default choice for routing and proxy use cases. It covers 100+ provider endpoints, handles virtual keys, supports the dual TPM/RPM model, and integrates with Langfuse and Braintrust for observability. The Python SDK means you can run it as a server or embed it as a library. For teams that want routing, fallback, and basic cost tracking without operational complexity on the gateway itself, this is the right starting point.

Portkey open-sourced its full AI gateway under Apache 2.0 in March 2026, including guardrails and PII redaction pipelines. If your use case requires input/output scanning (content policy, PII stripping before calls leave your network), Portkey gives you that in a single deployment rather than adding a separate guardrail layer. The managed cloud version reduces ops burden for teams below the data-residency threshold.

Roll your own only when you have unusual requirements — a custom wire protocol, deep integration with internal auth systems, or constraints that prevent running any third-party code. This path costs at minimum a couple of engineer-months to build something defensible, plus ongoing maintenance to keep provider SDK changes from breaking it. Below roughly $15,000 per month in API spend, building your own never wins on economics — LiteLLM or Portkey does the job for a fraction of the engineering cost. Above that, custom pays off only if you also have requirements neither tool covers.

	LiteLLM	Portkey (OSS)	Roll your own
Provider coverage	100+	250+	Whatever you build
Virtual keys	Yes	Yes	Yes
TPM + RPM limits	Yes	Yes	You implement
PII redaction	Plugin	Built-in	You implement
Guardrails	Plugin	Built-in	You implement
Self-hostable	Yes	Yes	Yes
Observability integrations	Langfuse, Braintrust	Langfuse, Braintrust, others	You wire up
License	Apache 2.0	Apache 2.0	N/A
Break-even vs custom	Below $15k/month	Below $15k/month	Above $15k/month with custom requirements

The decision at most companies is straightforward: start with LiteLLM, add Portkey if you need guardrails and PII handling, and revisit rolling your own only if you hit a specific wall that neither solves.

Worked numbers: what tagging actually saves

Suppose a team is spending $22,000 per month on a mix of features. Without tags, the only action available is "use cheaper models everywhere" — a blunt cut that may degrade quality across features that need it.

Tagged cost rollup (illustrative, mid-2026 pricing):

Feature           | Monthly calls | Avg tokens in | Avg tokens out | Cost/call | Monthly cost
doc-summarizer    | 45,000        | 8,000         | 400            | $0.031    | $1,395
slack-bot         | 280,000       | 850           | 120            | $0.0035   | $980
support-assistant | 12,000        | 2,200         | 600            | $0.0098   | $118
dev-env-logging   | 620,000       | 8,000         | 400            | $0.031    | $19,220
batch-reports     | 8,000         | 12,000        | 800            | $0.048    | $384
                  |               |               |                |           | --------
Total                                                                          | $22,097

There's the bill. The four production features account for $2,877 of it. dev-env-logging — the engineer from the opening anecdote, running 8,000-token test prompts on gpt-4o with logging enabled, 620,000 calls a month across CI and local loops — accounts for $19,220. It shows up as its own line only because the gateway tags calls by environment; on an untagged invoice it is invisible, smeared across the total. That $19,220 disappears almost immediately: switch dev to gpt-4o-mini or cap dev virtual keys at 10,000 tokens per day.

The slack-bot calls are cheap per call but high volume. Route those to a smaller model — see Model Routing and Cascades for the mechanics — and that $980 can drop to under $100.

The doc-summarizer has the highest per-call cost among production features because of the 8,000-token input. That's a caching candidate — if documents are re-processed frequently, prefix caching can cut the input token cost by 90%. See Prompt Caching Deep Dive.

{ "type": "token-cost", "title": "Where a monthly bill comes from: input vs output vs cached tokens" }

None of these optimizations are visible without the tags. That's the fundamental argument for getting tagging right from day one: cost data only becomes actionable when it is attributable.

What breaks

Rate limit Redis becomes a single point of failure. If the Redis cluster goes down, the gateway cannot check limits. Two options: fail open (allow all requests, accept the risk of temporary overage) or fail closed (reject all requests until Redis recovers). Fail open is usually correct for production — a few minutes of unchecked traffic is better than an outage. Make this a deliberate choice in your config, not an accident.

Context window mismatches on fallback. If your primary model has a 128k context and the fallback has 32k, prompts between 32k and 128k will fail the fallback with a confusing error. The gateway must check prompt length and either truncate or skip that fallback option. LiteLLM's context_window_fallbacks config handles this; hand-rolled gateways often get it wrong.

Streaming retries corrupt partial responses. If a streaming response fails after tokens have already been sent to the caller, retrying from the gateway generates a new completion that starts differently. The caller receives duplicate or contradictory content. The correct behavior is: once tokens have been streamed to the caller, do not retry transparently — surface the error so the caller can decide. This is one of the trickiest edge cases in gateway implementation. Streaming, Batching, and Throughput covers the underlying mechanics.

TPM estimation drift. You estimate tokens before the call, deduct estimated input tokens from the TPM counter, then deduct actual output tokens after. If your estimation is systematically low (common with long documents where input compression is nonlinear), you can consistently overshoot your actual TPM usage and hit provider limits even though your counter says you have headroom. Calibrate your token estimator periodically against actual provider token counts from the billing API.

Tag discipline erosion. The first team that bypasses the gateway "just for this one experiment" creates a hole in your cost data. Once the attribution data has gaps, the dashboard numbers become unreliable and the optimization conversations become political. Enforce tags at the gateway level — reject requests that lack required fields after a short grace period — or accept that your cost data will always be incomplete.

Virtual key leaks to logs. Virtual keys are credentials. If your application logs the full request headers (a common debugging setup), virtual keys end up in log files where they are accessible to anyone with log access. Scrub credentials from logs at the gateway; do not assume application logs are safe.

flowchart TD
    REDIS_DOWN[Redis outage] -->|fail open?| ALLOW[Allow all requests\nAccept overage risk]
    REDIS_DOWN -->|fail closed?| BLOCK[Block all requests\nOutage]
    CTX_MISMATCH[Prompt > fallback context] -->|no check| ERR[Confusing error to caller]
    CTX_MISMATCH -->|gateway checks length| SKIP[Skip this fallback\nor truncate]
    STREAM_FAIL[Stream fails mid-response] -->|retry transparently| CORRUPT[Duplicate/contradictory content]
    STREAM_FAIL -->|surface error to caller| CALLER_DECIDES[Caller handles gracefully]
    style REDIS_DOWN fill:#ffaa00,color:#0a0a0f
    style CTX_MISMATCH fill:#ffaa00,color:#0a0a0f
    style STREAM_FAIL fill:#ffaa00,color:#0a0a0f
    style ERR fill:#ff2e88,color:#111
    style CORRUPT fill:#ff2e88,color:#111
    style BLOCK fill:#ff2e88,color:#111

The decision in practice

When do you add a gateway versus keeping direct API calls?

Add a gateway when:

• More than one team shares a provider credential
• You have more than one provider and want unified fallback
• Finance or product has asked for cost attribution that you cannot currently provide
• You are hitting provider rate limits without knowing which service drove the spike
• Data residency requirements exist (self-hosted gateway keeps payloads in your perimeter)

Skip the gateway when:

• You have exactly one team, one provider, and one service
• You are in early prototype phase and the coordination overhead is not justified yet
• You are below roughly $500 per month in API spend — the investment in gateway setup does not pay back at that scale

The correct order of operations. Start with direct provider calls and a single shared key. The first time you have two teams or two providers, add LiteLLM. Configure virtual keys and add cost tags from day one — retrofitting tags into existing calls later is painful. Add RPM limits the same day you add TPM limits. Wire up Langfuse or Braintrust before your first production launch so you have cost-by-feature data from the start, not six months in when the bill has already surprised you.

The gateway is not the interesting part of building AI products. But it is the part that lets you see clearly what the interesting parts are actually costing you — and that visibility is what makes every subsequent optimization decision non-speculative. The teams that understand their token economics and have attribution data to act on cut bills by 3–5x. The teams that do not know which feature is expensive are guessing.