Fallbacks, Retries, and Circuit Breakers: Building Reliable LLM Pipelines

At 2:47 AM on a Tuesday, your LLM-powered customer support tool started returning 503s from the primary provider. Your on-call alert fired at 2:48. But by then, your app had already queued 3,200 requests behind a retry loop that wasn't backing off — all hammering the same endpoint in a tight loop, blocking your thread pool, and preventing the fallback route from ever being used. The outage lasted 18 minutes. Your retry storm extended its blast radius for another 40.

This is the most common LLM production failure pattern. Not the outage itself — those are rare and short — but the retry implementation that turns a brief degradation into a full application freeze.

The three failure modes you're actually handling

Before writing any retry code, be precise about what you're trying to solve. LLM API errors fall into three categories that require completely different responses.

Transient errors are temporary conditions that resolve themselves: 429 rate limits when you've exceeded your tier's tokens-per-minute quota, 503s during provider capacity spikes, and intermittent network timeouts. These account for the vast majority of failures in a healthy system — typically 2–5% of requests at scale. They resolve within seconds to a few minutes, and retrying with appropriate backoff is exactly right.

Provider outages are sustained failures where retrying the same provider is futile. These are rarer (major providers like OpenAI and Anthropic publish >99.9% uptime SLAs) but longer — a real outage might last 15–90 minutes. For these, you need a fallback to a different provider or a degraded response mode, not endless retries against a dead endpoint.

Permanent errors will never succeed regardless of how many times you try. A 400 means your request is malformed. A 401 means your API key is wrong. A 403 means you don't have permission. A 422 or content-filter rejection means the specific content violated the provider's policies. Retrying any of these wastes money and time and never produces a success. The rule is: retry 429s and 5xx errors, fail fast on everything else.

flowchart TD
    ERR[API Error] --> IS4[4xx error?]
    IS4 -->|"400, 401, 403, 422"| FAST[Fail fast — do not retry]
    IS4 -->|429| CHECK["Check Retry-After header\nthen backoff + retry"]
    IS4 -->|No — 5xx| BACK["Exponential backoff + jitter\nretry up to max_attempts"]
    BACK --> LIMIT{Attempts exhausted?}
    LIMIT -->|Yes| FALL[Invoke fallback chain]
    LIMIT -->|No| BACK
    CHECK --> LIMIT2{Attempts exhausted?}
    LIMIT2 -->|Yes| FALL
    LIMIT2 -->|No| CHECK

    style FAST fill:#ff2e88,color:#fff
    style FALL fill:#ffaa00,color:#0a0a0f

Retry logic: backoff, jitter, and the Retry-After header

The mechanics of a correct retry implementation are well-understood in distributed systems, but they get misapplied in LLM contexts often enough to be worth covering precisely.

Exponential backoff multiplies the wait time by a factor (usually 2) on each attempt: 1s, 2s, 4s, 8s, 16s. The intuition is that if one retry didn't succeed after 1 second, the problem probably needs more than another second to resolve. Capping at some maximum (typically 60s) prevents a single stuck request from waiting indefinitely.

Full jitter adds a critical refinement. Without jitter, 500 clients that all hit a rate limit at the same moment all sleep for exactly 2^attempt × base_delay seconds and then all retry at the same moment. This recreates the same request spike that caused the 429 in the first place — a thundering herd. Full jitter randomizes the wait within [0, cap] where cap = min(max_delay, base × 2^attempt). The clients now spread their retries across a window instead of synchronizing them.

The Retry-After header on 429 responses is the provider telling you exactly when you can try again. Ignoring it and using only your own backoff logic is the wrong call — if the header says wait 47 seconds and you retry in 4, you restart the rate-limit clock. Always use the larger of your computed backoff and the header value.

Here's what a production-grade retry wrapper looks like:

import asyncio
import random
from typing import Any, Callable
import httpx

async def with_retry(
    fn: Callable[[], Any],
    max_attempts: int = 4,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    retryable_status: set[int] = frozenset({429, 500, 502, 503, 504, 529}),
) -> Any:
    """
    Retry with full jitter exponential backoff.
    Honors Retry-After on 429. Fails fast on non-retryable errors.
    """
    for attempt in range(max_attempts):
        try:
            return await fn()
        except httpx.HTTPStatusError as exc:
            status = exc.response.status_code

            if status not in retryable_status:
                # 400, 401, 403, 422 etc. — will never succeed
                raise

            if attempt == max_attempts - 1:
                raise  # exhausted retries

            cap = min(max_delay, base_delay * (2 ** attempt))
            wait = random.uniform(0, cap)
            if status == 429:
                retry_after = exc.response.headers.get("retry-after")
                if retry_after:
                    try:
                        wait = max(wait, float(retry_after))
                    except ValueError:
                        pass  # HTTP-date form — keep the jittered backoff

            await asyncio.sleep(wait)
        except httpx.TransportError:
            # timeouts, connection resets, DNS blips — transient, retry
            if attempt == max_attempts - 1:
                raise
            cap = min(max_delay, base_delay * (2 ** attempt))
            await asyncio.sleep(random.uniform(0, cap))

    raise RuntimeError("unreachable")  # max_attempts >= 1 always raises above

A few implementation details deserve attention. The frozenset of retryable status codes is explicit — nothing implicit — and includes one provider-specific entry: Anthropic returns HTTP 529 (overloaded_error) when its servers are at capacity, which is exactly the transient case this whole section is about; know your providers' non-standard codes and add them deliberately. The Retry-After branch on 429 takes the larger of the header value and the jittered backoff, falling back to jitter alone when the header is absent or unparseable. The TransportError branch catches timeouts and connection failures — the network blips the transient-error definition above promised to retry. And the early raise on non-retryable codes is the most important line in the function.

For the OpenAI and Anthropic Python SDKs specifically, both have built-in retry handling that you can configure:

from anthropic import Anthropic

client = Anthropic(
    max_retries=3,  # exponential backoff with jitter, built in
    timeout=30.0,
)

# Or disable it entirely if you implement your own:
client = Anthropic(max_retries=0)

from openai import AsyncOpenAI

client = AsyncOpenAI(
    max_retries=3,
    timeout=60.0,
)

The SDK defaults are reasonable starting points — three retries with backoff, and both SDKs also retry connection errors, timeouts, and 408s, not just 429/5xx — but they don't implement fallback chains or circuit breakers. For a single-provider integration without complex routing, the SDK defaults plus explicit 4xx handling is enough. When you're routing across providers, you need more.

Fallback chains: switching providers under load

A fallback chain tries Provider A, catches a failure threshold, and routes to Provider B. The concept is simple. The implementation details are where most teams make mistakes.

The most common mistake: context window mismatch. Your primary model has a 200,000-token context window. Your fallback model has 32,000. Your prompt is 95,000 tokens. The fallback call fails with a 400, and your retry logic catches it, doesn't know it's a permanent error for this specific prompt, and tries the fallback again. Now you've burned time and tokens on a guaranteed failure.

Before invoking any fallback model, check whether your prompt fits:

import tiktoken
from anthropic import Anthropic
from openai import AsyncOpenAI

PRIMARY_CONTEXT_LIMIT = 200_000   # Claude claude-opus-4-5 (illustrative)
FALLBACK_CONTEXT_LIMIT = 128_000  # GPT-4o (illustrative)
FALLBACK_SMALL_LIMIT = 32_000    # smaller fallback

def count_tokens_rough(text: str) -> int:
    """Rough token count using cl100k encoding; within ~10% for most models."""
    enc = tiktoken.get_encoding("cl100k_base")
    return len(enc.encode(text))

async def completion_with_fallback(
    prompt: str,
    system: str,
    anthropic_client: Anthropic,
    openai_client: AsyncOpenAI,
) -> str:
    prompt_tokens = count_tokens_rough(system + prompt)

    # Try primary
    try:
        resp = anthropic_client.messages.create(
            model="claude-opus-4-5",
            max_tokens=4096,
            system=system,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.content[0].text
    except Exception as primary_err:
        if not _is_provider_error(primary_err):
            raise

    # Fallback: check context window before attempting
    if prompt_tokens > FALLBACK_CONTEXT_LIMIT:
        # Truncate or summarize — never silently let the provider truncate
        prompt = _truncate_to_fit(prompt, FALLBACK_CONTEXT_LIMIT - len(system) // 4 - 4096)

    try:
        resp = await openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": system},
                {"role": "user", "content": prompt},
            ],
            max_tokens=4096,
        )
        return resp.choices[0].message.content
    except Exception:
        raise  # escalate — both providers failed

def _is_provider_error(err: Exception) -> bool:
    """True for errors that warrant fallback (5xx, sustained 429s)."""
    import httpx
    if isinstance(err, httpx.HTTPStatusError):
        return err.response.status_code in {429, 500, 502, 503, 504}
    return True  # network errors, timeouts → try fallback

def _truncate_to_fit(text: str, max_tokens: int) -> str:
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)
    if len(tokens) <= max_tokens:
        return text
    truncated = enc.decode(tokens[:max_tokens])
    return truncated + "\n[Content truncated to fit model context window]"

The _truncate_to_fit function is doing something important: it adds an explicit marker that the content was truncated. This is much better than silent truncation — the model knows it has incomplete input, and your observability layer can flag these events for later analysis.

For production systems with multiple providers and routing rules, LiteLLM handles this more elegantly:

import litellm

litellm.set_verbose = False

response = await litellm.acompletion(
    model="anthropic/claude-opus-4-5",
    messages=[{"role": "user", "content": prompt}],
    fallbacks=["openai/gpt-4o", "anthropic/claude-sonnet-4-5"],
    context_window_fallbacks=[
        # If prompt exceeds context window, fall back to a LARGER window —
        # a ~1M-token Gemini-class model (illustrative), not a smaller one
        {"anthropic/claude-opus-4-5": ["gemini/gemini-2.5-pro"]},
    ],
    num_retries=3,
    request_timeout=30,
)

LiteLLM's context_window_fallbacks handles the token-count check automatically — it catches the context-length error from the provider and routes to whatever you configured. It will not stop you from configuring a smaller-window target, so choosing a genuinely larger window (as above) is still your responsibility. This is the pattern the sibling article on the LLM gateway pattern covers in more depth.

{"type": "model-router", "title": "Routing + Fallback: cost vs quality trade-off"}

Circuit breakers: stopping the retry storm at the source

Retries are right for transient errors. But when a provider is genuinely degraded for minutes — not milliseconds — retrying each request individually is destructive. Each attempt occupies a thread or coroutine for the full timeout duration before it fails. At 1,000 req/s with a 10-second timeout, you accumulate 10,000 in-flight requests in the first 10 seconds. That's your thread pool, your connection pool, and your memory.

The circuit breaker pattern solves this by moving the decision about whether to attempt a provider out of the per-request hot path and into a shared state machine. The breaker has three states:

• Closed: everything normal, requests flow through
• Open: provider is unhealthy, requests fail fast without attempting the provider
• Half-open: after a cooldown period, one probe request is allowed; if it succeeds, the breaker closes; if it fails, the breaker re-opens

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : failure_rate > 50%\nover 30s window
    Open --> HalfOpen : timeout elapsed\n(e.g. 30s)
    HalfOpen --> Closed : probe request succeeds
    HalfOpen --> Open : probe request fails
    Closed --> Closed : request succeeds
    Open --> Open : fail fast\n(no attempt)

The threshold parameters — failure rate, window size, cooldown — are the design decisions that matter. A 50% failure rate over a 30-second window is a reasonable production default: sensitive enough to catch a real outage within 30 seconds, not so sensitive that a brief spike trips the breaker unnecessarily.

Here's a minimal circuit breaker implementation that composes with the retry logic above:

import time
from dataclasses import dataclass, field
from threading import Lock
from enum import Enum

class BreakerState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    name: str
    failure_threshold: float = 0.5   # 50% failure rate
    window_seconds: float = 30.0
    cooldown_seconds: float = 30.0

    _state: BreakerState = field(default=BreakerState.CLOSED, init=False)
    _lock: Lock = field(default_factory=Lock, init=False)
    _requests: list[tuple[float, bool]] = field(default_factory=list, init=False)
    _opened_at: float = field(default=0.0, init=False)
    _probe_in_flight: bool = field(default=False, init=False)

    def allow_request(self) -> bool:
        with self._lock:
            now = time.monotonic()
            if self._state == BreakerState.OPEN:
                if now - self._opened_at >= self.cooldown_seconds:
                    self._state = BreakerState.HALF_OPEN
                    self._probe_in_flight = True
                    return True  # exactly one probe
                return False
            if self._state == BreakerState.HALF_OPEN:
                return not self._probe_in_flight  # hold traffic until probe resolves
            return True

    def record_success(self) -> None:
        with self._lock:
            self._requests.append((time.monotonic(), True))
            self._trim_window()
            if self._state == BreakerState.HALF_OPEN:
                self._state = BreakerState.CLOSED
                self._probe_in_flight = False
                self._requests.clear()

    def record_failure(self) -> None:
        with self._lock:
            now = time.monotonic()
            if self._state == BreakerState.HALF_OPEN:
                # failed probe → straight back to open, restart the cooldown
                self._state = BreakerState.OPEN
                self._opened_at = now
                self._probe_in_flight = False
                return
            self._requests.append((now, False))
            self._trim_window()
            failures = sum(1 for _, ok in self._requests if not ok)
            rate = failures / max(len(self._requests), 1)
            if rate >= self.failure_threshold and len(self._requests) >= 5:
                self._state = BreakerState.OPEN
                self._opened_at = now

    def _trim_window(self) -> None:
        cutoff = time.monotonic() - self.window_seconds
        self._requests = [(t, ok) for t, ok in self._requests if t >= cutoff]

# Usage
breaker = CircuitBreaker(name="anthropic-primary")

async def call_with_circuit_breaker(prompt: str) -> str:
    if not breaker.allow_request():
        # Circuit is open — skip the attempt, use fallback immediately
        return await call_fallback(prompt)
    try:
        result = await call_primary(prompt)
        breaker.record_success()
        return result
    except Exception as exc:
        if _is_provider_error(exc):  # only 429/5xx/timeouts count against the breaker
            breaker.record_failure()
            if breaker._state == BreakerState.OPEN:
                return await call_fallback(prompt)
        raise  # permanent errors surface without touching breaker state

The minimum request count guard (len(self._requests) >= 5) prevents the breaker from opening on the first two failures in an otherwise healthy system. Without it, a cold-start burst of bad requests trips the breaker before you have any statistical signal.

What breaks

Streaming retries send duplicate or diverged completions

When a streaming response fails mid-stream — after tokens have already been sent to the client — retrying from scratch starts the generation over. The model uses different sampling paths and produces a different completion, so the user sees what looks like a corrupted or doubled response. Worse, the second completion might contradict the first.

The safe options:

1. Buffer the full stream before forwarding to the client. This eliminates the perception benefit of streaming but makes retries safe. Correct for async/background workloads.
2. Resume tokens: some providers support session-level resume, but this is not standard.
3. Explicit partial-failure UI: stop streaming, show "Generation interrupted — retry?", let the user decide. Correct for interactive chat where transparency matters.

Never silently restart a stream that has already emitted tokens. The streaming LLM responses deep dive covers this in detail.

Fallback "succeeds" on a prompt the primary would have rejected

Content filters vary across providers. A prompt that your primary rejects as policy-violating might succeed on the fallback — not because the fallback is more capable, but because it has different policy thresholds. This is usually undesirable. Distinguish between policy rejections (422 / content filter errors, which you should not fall back from) and capacity failures (429, 5xx, which you should).

Jittered retries still pile up if your client fleet is large

Full jitter solves the single-client thundering herd problem. But if 5,000 users all encounter 429s simultaneously (a traffic spike), even jittered retries from 5,000 independent clients produce a sustained elevated load during the backoff window. The right solution for very large fleets is a client-side rate limiter that caps outgoing requests before they reach the provider — a token bucket in your gateway layer. This is exactly what the LLM gateway pattern implements.

Circuit breaker opens on correlated deployments, not provider failures

If you roll out a bad prompt that causes 60% of requests to get 400s (content filter rejections) — which you are not retrying — the circuit breaker should not trip. But if your circuit breaker is counting all errors, it will. Always separate your circuit breaker's failure counting from permanent errors. Only 429s, 5xx, and timeouts should increment the failure counter.

Retry budget exhausted mid-pipeline

In a multi-step agent pipeline — query planning → retrieval → generation → postprocessing — exhausting retries at step 2 leaves the pipeline in a partially-completed state. The caller doesn't get a clean failure; it gets a half-populated response object with missing fields. Define clear failure contracts per step: what does a retry-exhausted failure look like, and what does the downstream step do when it receives one? Fail loudly with a typed exception rather than silently returning None.

The reason this deserves its own failure mode is that per-step reliability compounds: at 97% success per step, a five-step pipeline completes end-to-end only ~86% of the time (0.97⁵). Your retry budget is what holds each step's number up.

{"type": "error-compound", "accuracy": 0.97, "steps": 5, "title": "Reliability compounds across pipeline steps"}

Numbers: what this actually costs in production

Scenario: 100,000 LLM requests/day
Provider SLA: 99.9% uptime → ~525 down-minutes/year → 0.1% of requests hit outage
Rate limits: 2% of requests hit 429 at peak

Daily 429s: 2,000
Average retries per 429: 1.8 (most resolve on first retry)
Extra tokens from retries: 2,000 × 1.8 × 3,000 avg input tokens = 10.8M tokens
Cost at $0.40/M: ~$4.32/day = $1,574/year

Daily outage-related failures (0.1%): ~100 requests/day on average —
but bursty: zero most days, thousands during the one bad afternoon
With fallback: 100/day avg × fallback provider cost (assume 1.5× primary price)
Extra outage cost: negligible

Thundering-herd penalty without jitter:
2,000 429s → all retry at t+5s → second wave of 2,000 429s
Additional retries: 2,000 × 2 rounds × 3,000 tokens = 12M tokens
Cost of bad retry implementation: ~$4.80/day extra = $1,752/year wasted

The direct retry cost is real but modest — under $2k/year for a medium-scale deployment. The indirect cost of not having a fallback during a real outage (user-facing downtime, churned sessions, support volume) is typically 10–100× larger. The economics favor building the reliability layer correctly even if you never trigger the fallback.

At small scale — under 1,000 req/day — the SDK's built-in retry defaults plus manual fallback handling is fine. Above ~10,000 req/day, the complexity justifies using LiteLLM or Portkey rather than maintaining your own implementation. Above $15k/month, you probably want a self-hosted gateway that also handles cost attribution and rate limiting.

The decision in practice

The right reliability stack depends on where you are in the deployment lifecycle, not which patterns you find most interesting.

Day 1 — single provider, low traffic: use the Anthropic or OpenAI SDK's built-in retry (3 retries, default backoff). Add explicit handling to not retry 4xx errors other than 429. This takes 10 lines of code and covers 95% of your reliability needs.

Month 2 — growing traffic, hitting rate limits: add a fallback provider. If you're already on LiteLLM for model routing (see model routing and cascades), add the fallbacks parameter. If not, this is the moment to introduce it — the fallback chain configuration is much cleaner in LiteLLM than in hand-rolled code.

Month 4 — multiple providers, complex pipelines: add circuit breakers. At this point you're probably running a gateway (LiteLLM proxy or Portkey), and both support circuit-breaker configuration at the gateway level. If you're running agents where a mid-pipeline retry storm would be particularly damaging, add retry budgets per pipeline stage.

Scale — $15k+ monthly spend, data-residency requirements: self-hosted gateway with all three layers (retry, fallback, circuit breaker), dual TPM+RPM limits per virtual key, structured logging of every routing decision. This is the full reliability story, and the token economics article covers how to measure whether your reliability investment is actually paying off.

What you're optimizing for is not zero failures — providers will have incidents, rate limits will be hit — but bounded blast radius. A 15-minute provider outage should produce 15 minutes of degraded responses from your fallback, not 15 minutes of total unavailability followed by a 40-minute retry storm. The difference is three patterns and maybe 200 lines of carefully tested code.

One final thing worth being explicit about: the circuit breaker doesn't make your system more reliable by making it try harder. It makes it more reliable by making it try smarter — accepting that some providers are temporarily out of service, routing around them quickly, and letting them recover. That mindset — fail fast, fallback gracefully, recover cleanly — is the difference between a production LLM system and a demo that works until someone important is watching.