Cost Control and Reliability Engineering

The bill arrives in June. Sixty thousand dollars. You have ~30 engineers and an LLM-backed product that's gotten real traction, and you're staring at a Stripe charge that exceeds your entire cloud infrastructure spend. You trace it back: a system prompt sent fresh on every request, a frontier model handling trivial classification queries, no batching on your nightly summarization pipeline, and zero retry logic so every 429 spins up a retry storm that makes the rate-limit problem worse.

Three months later, the same traffic costs $14k. Same models, same quality. This module is the five levers that closed that gap.

Why the math surprises people

LLM API pricing has a structure that punishes naive usage at two places most engineers don't expect.

First: output costs more than input. Across OpenAI, Anthropic, and Google as of mid-2026, output tokens run 3-5× the per-token price of input tokens. Prefill is compute-bound and highly parallelizable — every prompt token is processed at once, with high arithmetic intensity per weight loaded. Decode is memory-bandwidth-bound: each generated token requires re-reading the full model weights (plus the growing KV cache), one sequential step at a time. That's the mechanism behind the output premium, and it means long system prompts are cheaper than verbose model outputs — the optimization focus should fall on output length control before anything else.

Second: multi-turn conversations compound. By turn 30 of a chat session, you're sending the entire conversation history as input on every call. If your average conversation starts at 500 tokens and each turn adds 200, by turn 30 you're sending 6,300 input tokens just for context. A user who has 100 turns sends 20,300 tokens of input on the final call alone — roughly forty times what they sent on turn 1.

# Back-of-envelope: why multi-turn sessions inflate fast

turn_1_input   = 500   # system prompt + first message
tokens_per_turn = 200  # average exchange adds this much history

turn_10_input  = 500 + (9 × 200)  = 2,300 tokens
turn_30_input  = 500 + (29 × 200) = 6,300 tokens
turn_100_input = 500 + (99 × 200) = 20,300 tokens

# At $3.00/M input tokens (GPT-4 class, illustrative):
# Turn 1:  $0.0015
# Turn 30: $0.019 per call — 12× more expensive
# Multiply by 10,000 daily active users × avg 20 turns...

{ "type": "token-cost", "title": "Where your monthly bill comes from" }

The interactive widget above lets you dial in your actual numbers — req/day, input/output token mix, cache hit rate, model tier — and watch the cost segments shift. Output tokens and long uncached inputs usually dominate.

Controlling output length

Since output carries the premium, control it deliberately instead of hoping the model is terse. Four mechanisms, in the order you should apply them:

• Set max_tokens per route. A classification endpoint needs 16 tokens; an extraction endpoint 256; a chat reply rarely needs more than 800. A hard cap turns worst-case cost into known cost, and a truncated response on a mis-sized route is a bug report you want.
• Ask for the length you want, in units the model can count. "Answer in at most three sentences" or "return at most 5 bullet points" works far better than "be concise" — models follow countable constraints and ignore vibes.
• Prefer terse output formats. A JSON schema with short keys and enums generates a fraction of the tokens of prose-wrapped answers, and structured outputs stop the model from padding with preamble ("Certainly! Here's the analysis you requested...").
• Use stop sequences to cut generation the moment the useful part ends — after a closing brace, a sentinel like ###, or the end of the first list.

The dollars are real: trimming average output from 500 to 350 tokens at $15/M across 4M requests/month is $9,000/month. That's a prompt edit and a max_tokens value, not an infrastructure project.

Lever 1: Prompt caching and prefix discipline

Provider-side prefix caching is the highest-ROI change for systems with stable system prompts or shared RAG context. It works on a simple principle: if your request starts with a byte-identical prefix that was cached from a recent prior request, the provider skips re-running prefill for those tokens and charges a fraction of the fresh rate.

Anthropic prefix caching (GA December 2024) costs a fraction of fresh prefill on cache reads — roughly 90% cheaper as of mid-2026 pricing — plus ~85% latency reduction because the KV cache is reused rather than recomputed. OpenAI's automatic caching gives a 50% discount on prefixes longer than 1,024 tokens, with zero code changes required on your end. (Both providers adjust pricing periodically; check their pricing pages for current rates.)

The catch is prefix discipline. The cache key is an exact prefix match. This means:

• Your system prompt must be byte-identical across requests. No timestamps, no random.seed()-dependent content, no "Today is {date}" in the system prompt unless you want zero cache hits.
• Cache-stable content must come first. Put your system prompt, then your RAG documents (if they're shared across requests), then few-shot examples, and only then the dynamic user message.
• Shuffled tool lists break caching. If you include tools and shuffle their order, each combination is a cache miss. Sort tools deterministically.

# Bad: timestamp in system prompt busts cache every second
system_prompt = f"""You are a helpful assistant. Current time: {datetime.now().isoformat()}
Always be concise and accurate."""

# Good: date at a coarser granularity, or pushed to the user message
SYSTEM_PROMPT = """You are a helpful assistant.
Always be concise and accurate."""

# Then in the user message if needed:
user_message = f"[{date.today()}] {user_input}"

The prompt-cache visualizer shows the cost delta in real time as you toggle which segments are cache-stable versus dynamic. The lesson is visceral once you move a timestamp from the system prompt into the user message and watch the monthly estimate drop.

{ "type": "prompt-cache", "scenario": "agent", "title": "Cost delta from prefix discipline" }

Semantic caching is different and operates at the application layer. You embed the user query, search a vector store for a similar cached query, and if cosine similarity exceeds a threshold, return the cached response without a model call. At a 67% cache hit rate, you can achieve 60-65% latency reduction on those hits. The domains where this works well are narrow: FAQ systems, structured code generation tasks, anything where the query space clusters tightly.

Semantic caching has one dangerous failure mode: stale entries silently serve wrong answers. If your underlying data changes, a query that was correctly answered last week may return a confidently wrong cached response this week. You need explicit TTLs and invalidation hooks tied to your data update pipeline, not just accuracy-threshold tuning. A 2025 paper also demonstrated key-collision attacks on semantic caches — adversarial prompts can retrieve wrong cached responses by exploiting similarity thresholds. Your invalidation logic needs adversarial hardening too, not just accuracy tuning.

Lever 2: Model routing and cascades

Not every query needs Claude Opus or GPT-4. The question is how to route automatically without degrading quality on the hard queries that do.

Production traffic splits roughly into thirds: a third of queries are trivially simple (greeting, yes/no, formatting), a third are medium complexity (explanation, summarization, moderate reasoning), and a third are genuinely hard (complex reasoning, code with intricate edge cases, long-form synthesis). At scale, sending the first two thirds to a cheap model saves 70-80% of model cost.

{ "type": "model-router", "title": "Cost vs quality at different routing thresholds" }

The visualizer lets you shift the routing threshold and watch the cost/quality tradeoff update. The insight: there's a wide middle zone where you can save 60-70% of cost with less than 5% quality degradation — the frontier model isn't earning its premium on most queries.

Two routing architectures exist and have different tradeoffs:

flowchart TD
    Q[Incoming query] --> CLF{Classifier\n<10ms}
    CLF -->|simple| CHEAP[Cheap model\n$1/M tier]
    CLF -->|complex| FRONT[Frontier model\n$15/M tier]
    CHEAP --> OUT[Response]
    FRONT --> OUT

    Q2[Incoming query] --> S[Small model\nattempt]
    S -->|low confidence| M[Medium model\nattempt]
    M -->|still low| F2[Frontier model\nattempt]
    F2 --> OUT2[Response]

    style CLF fill:#00e5ff,color:#0a0a0f
    style S fill:#0e7490,color:#fff
    style M fill:#0e7490,color:#fff

Routing makes one upfront decision. A lightweight classifier (a 0.5B BERT-class model gets ~90% routing accuracy in under 10ms) examines the query and picks a tier. Total latency for simple queries is cheap-model latency only.

Cascading tries the cheap model first, then escalates if confidence is low. This adds one or more model calls for the queries that end up at the frontier — the exact queries you care most about latency on. Three-tier cascades can add 300-800ms to your hardest queries.

The practical verdict: start with routing, not cascading. A rule-based router (regex patterns for "translate this", "what is your name", length thresholds) can capture 30-40% of traffic immediately. Add a classifier once you have enough labeled examples from production logs. RouteLLM (open source) ships a ready-made classifier if your traffic looks like general-purpose chat.

One warning that's under-documented: never use LLM self-reported confidence as your escalation signal. Models produce fluent authoritative text while being factually wrong. Use token probability distributions from the logprobs API, calibrated against empirical accuracy on your evaluation set — not "I'm 90% confident."

Read the deep dive on model routing and cascades if you're implementing this — the calibration details matter and the surface-level description makes it sound easier than it is.

Lever 3: Batching and streaming choices

The batching decision depends entirely on whether the workload is interactive.

For user-facing features where someone is watching a cursor: stream. Streaming sends tokens via server-sent events as they're generated. Perceived latency drops dramatically because users see the first word in 300ms rather than waiting 8 seconds for the full completion. Streaming does not reduce compute, token count, or cost — it's purely a UX improvement.

For background pipelines — summarization, classification, embedding generation, nightly report generation: use the batch API. Both OpenAI and Anthropic offer async batch endpoints that process requests with a ~24-hour turnaround window at roughly 50% of the synchronous price. You get the same model, the same quality, half the cost. The only reason to use real-time endpoints for batch workloads is not knowing the batch API exists.

{ "type": "batching", "title": "Static vs continuous batching throughput" }

If you're self-hosting on vLLM or SGLang, continuous batching delivers 2-3× throughput over static batching by dynamically inserting new requests into in-flight batches. The catch is that aggressive batching raises p99 latency — interactive and background jobs need separate serving configurations.

One subtle hazard: don't retry a stream you've partially sent to the user. If a stream fails after 40 tokens have been displayed, a naive retry sends a completely different completion from token 1. The user sees a corrupted response. The safe options are: don't retry mid-stream (just show an error), or don't start streaming until you have a buffer large enough to absorb partial failures.

Lever 4: The gateway pattern

At some point you want all of this — caching, routing, retries, fallback chains, cost attribution — handled in one place rather than scattered across a dozen different service files. That's the gateway pattern.

flowchart LR
    SVC1[Service A] --> GW[LLM Gateway]
    SVC2[Service B] --> GW
    SVC3[Service C] --> GW
    GW --> OAI[OpenAI]
    GW --> ANT[Anthropic]
    GW --> VK[Virtual keys\ncost caps]
    GW --> CACHE[Semantic cache]
    GW --> OBS[Observability\ntags + cost]
    style GW fill:#00e5ff,color:#0a0a0f

The gateway sits between your services and providers, issuing virtual keys — one key per team or feature — so your actual provider credentials never leave the gateway. You can set per-team spend caps, kill a feature's access, and rotate provider keys without touching application code.

Tag every request at the gateway with user_id, feature, environment, and model. Without these tags, cost rollups are meaningless and you can't tell which feature is responsible for the June bill spike. This is the prerequisite for all other optimization — you cannot optimize what you cannot measure.

LiteLLM is the most widely deployed option for pure routing/proxy use cases. Portkey open-sourced its full gateway (Apache 2.0, March 2026) including guardrails and PII redaction — worth evaluating if you need those features built in. Build your own only when compliance requirements (data residency, custom auth flows) can't be satisfied with configuration alone. As a rough threshold: below $15k/month API spend, building your own gateway costs more in engineering time than it saves.

Read the LLM gateway pattern deep dive for the specifics on dual TPM+RPM rate limiting (you need both; tokens-per-minute alone leaves burst attack vectors open).

Lever 5: Reliability engineering

A 429 cascade is how LLM reliability problems die. One service hits a rate limit, retries immediately in parallel with all its other in-flight requests, multiplies the rate-limit pressure by 50, and suddenly everything is returning 429 simultaneously. You can spend ten minutes fixing something that didn't need to break at all.

Retry logic that doesn't make things worse

The three rules that prevent most self-inflicted outages:

Rule 1: Only retry 429 and 5xx. 400 Bad Request, 401 Unauthorized, 403 Forbidden, 422 Unprocessable — these will not succeed on retry regardless of wait time. Retrying them wastes budget and adds latency to an already-failing path.

Rule 2: Exponential backoff with full jitter. Not fixed delay (sleep(2)), not exponential without jitter (sleep(2^attempt)). Both cause synchronized retry storms when many clients hit a rate limit at the same moment.

import random
import time

from openai import (AuthenticationError, BadRequestError,
                    InternalServerError, RateLimitError)

def call_with_retry(fn, max_attempts=5, base=1.0, cap=60.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except (AuthenticationError, BadRequestError):
            raise  # Never retry these
        except (RateLimitError, InternalServerError) as e:
            # 429 and 5xx — the only errors worth retrying
            if attempt == max_attempts - 1:
                raise
            # Honor Retry-After if present
            retry_after = getattr(e, "retry_after", None)
            if retry_after:
                wait = float(retry_after)
            else:
                # Full jitter: random in [0, min(cap, base * 2^attempt)]
                wait = random.uniform(0, min(cap, base * (2 ** attempt)))
            time.sleep(wait)

Rule 3: Honor the Retry-After header. When a 429 includes it, that's the provider telling you exactly how long to wait. Ignoring it and using your own timer is how you turn a 5-second penalty into a 30-second retry storm.

Fallback chains

When OpenAI is returning 503s, you want traffic to move to Anthropic (or vice versa) automatically. A fallback chain is that logic:

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=messages,
    fallbacks=[
        {"model": "claude-sonnet-4-5"},
        {"model": "gemini-2.0-flash"},
    ],
    timeout=30,
)

The non-obvious hazard: check context window size before falling back. If your primary model has a 128k context window and your fallback has 32k, a long conversation that succeeds on the primary will silently truncate or raise a context-length error on the fallback. You need to check len(messages_tokens) < fallback_context_limit before switching, and if it fails, either truncate the history or skip that fallback tier entirely.

sequenceDiagram
    participant App
    participant GW as Gateway
    participant OAI as OpenAI
    participant ANT as Anthropic

    App->>GW: Request (8k tokens)
    GW->>OAI: Attempt 1
    OAI-->>GW: 503 Service Unavailable
    GW->>GW: Check: 8k < 200k Anthropic limit ✓
    GW->>ANT: Fallback attempt
    ANT-->>GW: 200 OK
    GW-->>App: Response (flagged: used fallback)

Circuit breakers

Retries and fallbacks handle transient failures. Circuit breakers handle sustained failures — when a provider is down for 10 minutes, you want to stop sending requests there immediately rather than accumulating timeout latency across thousands of calls.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure_rate > 50%\nin 60s window
    Open --> HalfOpen: timeout (30-120s)
    HalfOpen --> Closed: probe request succeeds
    HalfOpen --> Open: probe request fails
    Open --> [*]: serve degraded mode

Closed state: requests flow normally, failures are counted. Open state: requests fail immediately with a cached or degraded response; no upstream calls made. Half-open state: one probe request goes through. If it succeeds, circuit closes. If not, back to open.

The circuit breaker is what prevents a degraded OpenAI from taking down your product with timeout accumulation. Without it, 200 concurrent requests × 30-second timeout = 6,000 seconds of accumulated wait time before anyone notices.

The $60k-to-$14k walkthrough

Here's what the math looks like when you apply the levers sequentially. These are illustrative numbers built from realistic patterns — your actual numbers depend on traffic mix.

Starting point: $60,000/month (illustrative — prices as of mid-2026)
  ~130k req/day → 4M req/month, everything on a frontier model, no caching:
    Avg input:  2,500 tokens (1,500 stable prefix + 1,000 dynamic)
                → 10B tokens/month × $3.00/M                  = $30,000
    Avg output:   500 tokens → 2B tokens/month × $15.00/M     = $30,000
  Half the bill is output — the premium-priced segment.

Step 1 — Prompt caching (Lever 1; stable 1,500-token prefix, 90% hit rate):
  Stable segment was: 6B tokens × $3.00/M                     = $18,000
  Now: 5.4B cached × $0.30/M + 0.6B fresh × $3.00/M           =  $3,420
  Saving: ~$14,600                     Remaining bill: ~$45,400

Step 2 — Model routing (Lever 2; 70% of requests → $0.40/M-in,
         $1.60/M-out tier):
  Cheap tier (2.8M req):  input $2,800 + output $2,240        =  $5,040
  Frontier (1.2M req):    cached prefix $1,030
                          + dynamic input $3,600
                          + output $9,000                     = $13,630
  Saving: ~$26,700 (net of the caching already applied)
                                       Remaining bill: ~$18,700

Step 3 — Batch API (Lever 3; 20% of traffic is async pipelines, 50% off):
  Saving: ~$1,900                      Remaining bill: ~$16,800

Step 4 — Output length control (max_tokens + length instructions,
         trim 30% avg output):
  Output spend after routing + batch: ~$10,100
  Saving: ~$3,000                      Remaining bill: ~$13,800

Final: ~$14,000/month. 77% reduction. Quality impact: <3% measured by eval.

The order matters. Caching comes first because it's nearly free to implement and the gains carry through every later step (note how Step 2's saving is computed against the already-cached frontier bill, not the raw one). Routing is the biggest single line item. Output length control is often neglected because engineers don't think about it as a cost lever — but at output-token prices 3-5× input, every output token you don't generate is worth 3-5 trimmed input tokens.

What breaks in practice

Cache hit rate is a lie when measured globally. An 80% global cache hit rate can coexist with a feature that gets 0% hits because its system prompt includes a session token. Measure per-route, per-feature. Global numbers hide the places that need fixing.

Routing without observability degrades silently. A router that sends a hard query to a weak model produces confident wrong answers. The user doesn't complain about quality; they just leave. You need to log every routing decision — query tier assigned, model used, latency, eval score — and actively monitor for quality degradation in the cheap-model tier.

Fallback models inherit the input. When you fall back from a 200k-context model to one with a 32k window — a self-hosted open-weights deployment, say — a 100k-token conversation silently truncates or errors. Pick the fallback by token count, with headroom below each limit:

def get_fallback_model(token_count: int) -> str:
    if token_count < 30_000:
        return "local-llama-70b"    # 32k window, cheapest tier
    elif token_count < 120_000:
        return "gpt-4o"             # 128k window
    else:
        return "claude-sonnet-4-5"  # 200k window; past that, truncate history

Semantic cache invalidation lags data updates. Your product just updated its return policy. The old policy is cached with high confidence. The first user who asks gets the wrong answer. Cache invalidation must be a first-class part of your data pipeline, not an afterthought.

Aggressive retry storms from naive retry logic. If you have 500 concurrent requests and all hit a rate limit at the same second, sleep(2) means they all retry at the same second + 2. You've just moved the thundering herd two seconds into the future. Full jitter breaks this up. The code above is not optional.

The measurement prerequisite

Nothing in this module works without observability. Every LLM call needs these tags before you can optimize anything:

span.set_attribute("llm.user_id", user_id)
span.set_attribute("llm.feature", feature_name)
span.set_attribute("llm.model", model_used)
span.set_attribute("llm.input_tokens", usage.input_tokens)
span.set_attribute("llm.output_tokens", usage.output_tokens)
span.set_attribute("llm.cached_tokens", usage.cache_read_input_tokens)
span.set_attribute("llm.cost_usd", estimated_cost)
span.set_attribute("llm.routing_tier", routing_decision)
span.set_attribute("llm.cache_hit", cache_hit)

Langfuse and Braintrust both support this natively. LiteLLM and Portkey emit cost telemetry automatically if you configure the tags at the gateway layer. The point is: set this up before you start optimizing, not after.

Read LLM observability and monitoring for what to alert on — cost drift, quality regression in routed tiers, cache hit rate drops.

Where to go next

Token economics deep dive — the full breakdown of where your bill comes from, including multi-turn compounding and the batch API pricing details that the marketing pages don't foreground.

Prompt caching in depth — covers both provider-side prefix caching and application-side semantic caching, with the security research on cache poisoning and how to tune invalidation.

Model routing and cascades — implementation details for RouteLLM, logprob-based confidence calibration, and how to measure routing quality without letting degradation hide in aggregate metrics.

Fallbacks, retries, and circuit breakers — the production failure patterns in detail: what the Retry-After header actually tells you, how to structure fallback chains across providers with different context windows, and circuit breaker threshold tuning.

The LLM gateway pattern — when to adopt a gateway, how to configure dual TPM+RPM limits, and what to instrument from day one so the June bill doesn't catch you off guard again.

LLMOps: shipping and operating AI products — the operational layer above cost optimization: deployment patterns, prompt versioning, CI for model changes, and the feedback loops that keep quality from drifting as you optimize.