Reference architecture for a production LLM application

The invoice question arrives before any bug report. Finance wants to know why Tuesday cost $1,900 when a typical day runs $180. You pull the logs: one enterprise user pasted a 90-page contract into the chat, an agent spent four hours re-retrieving the same document set in a loop, and every single request returned HTTP 200. No errors. No alerts. Sub-400ms p50. The system did exactly what it was built to do — one successful model call at a time — and nothing in it could tell that anything was wrong.

This is the production LLM problem. Every individual call succeeding is not the same as the system behaving correctly. Building a system that can tell the difference — and act on it — requires a specific set of layers that most teams assemble reactively. This article describes them proactively, as a reference architecture you can build toward before the incidents.

Why the architecture exists

A naive LLM application looks like this: user sends a message, application appends it to a system prompt, calls the model API, returns the response. That works in a demo. In production at meaningful scale it breaks in at least six distinct ways:

1. The model provider goes down. Now your entire application is down.
2. A prompt change ships on Friday. Quality degrades silently until Monday's support tickets.
3. An agent gets into a reasoning loop and spends $40 on a single user session.
4. A user asks a question that retrieves an outdated document. The model answers confidently from stale data.
5. The model returns PII from another user's session that leaked into the context window.
6. A power user sends 50,000-token requests at scale, turning your $5k/month bill into $50k.

None of these scenarios produce HTTP 500s. They all produce HTTP 200 with a wrong, expensive, or unsafe response. The reference architecture exists to catch each of them at the layer where it belongs.

The seven layers

flowchart LR
    A["1 Ingress"] --> B["2 AI Gateway"]
    B --> C["3 Context Assembly"]
    C --> D["4 Model Execution"]
    D --> E["5 Output Validation"]
    E --> F["6 Observability"]
    F -.->|"async signal"| G["7 Feedback Store"]

    style A fill:#0e7490,color:#fff
    style B fill:#00e5ff,color:#0a0a0f
    style C fill:#a855f7,color:#fff
    style D fill:#15803d,color:#fff
    style E fill:#15803d,color:#fff
    style F fill:#0e7490,color:#fff
    style G fill:#ffaa00,color:#0a0a0f

Layer 1: Ingress

The ingress layer is everything that happens before your application sees the request. Authentication (JWT verification or API key lookup), coarse-grained rate limiting (per user, per org, per tier), TLS termination, and abuse protection (blocking obviously malicious payloads, enforcing maximum request body size).

This layer is table stakes and largely solved by existing infrastructure. If you're on AWS, API Gateway + Lambda Authorizer handles it. If you're self-hosting, Nginx or Envoy with a Redis-backed rate limiter. The key constraint: keep it fast and stateless. Every millisecond spent here is latency visible to the user, and every stateful dependency adds availability risk before you've even reached your application.

One LLM-specific addition worth making here: input size limits. A user who can send a 100,000-token message can run up a single-request bill of $3 at frontier pricing (illustrative, mid-2026). Your rate limiter should enforce token-budget limits by tier, not just request-per-minute limits.

Layer 2: The AI gateway

The AI gateway is the traffic router and abstraction layer for model providers. It does four things that your application code should not have to do: model routing, provider fallback, semantic caching, and cost attribution.

Model aliasing is the highest-value feature and the first thing to implement. Instead of hardcoding claude-sonnet-4-5 in your application code, you define a model alias like chat-standard. The gateway resolves that alias to a specific model+provider at call time, based on a config file. When Anthropic releases a new model or you want to test a cheaper option, you change one config entry and all services using chat-standard pick it up immediately — without a code deploy. With hardcoded provider model IDs, every deprecation becomes a multi-service find-and-replace and a coordinated deploy; with aliases, it's one config change.

Provider fallback handles the outage scenario. Your gateway maintains an ordered fallback chain — try the primary provider first, and if it returns 5xx or times out (configurable threshold, typically 30 seconds), retry on the secondary:

# Gateway routing config (simplified LiteLLM style)
ROUTING_CONFIG = {
    "chat-standard": [
        {"model": "anthropic/claude-sonnet-4-5", "weight": 1.0, "timeout": 30},
        {"model": "openai/gpt-4o",               "weight": 0.0, "timeout": 30},  # fallback
    ],
    "chat-fast": [
        {"model": "anthropic/claude-haiku-4",    "weight": 1.0, "timeout": 10},
        {"model": "openai/gpt-4o-mini",          "weight": 0.0, "timeout": 10},
    ],
    "reasoning": [
        {"model": "anthropic/claude-opus-4",     "weight": 1.0, "timeout": 60},
        {"model": "openai/o3",                   "weight": 0.0, "timeout": 90},
    ]
}

Semantic caching sits in the gateway and matches incoming queries to recent cached responses using embedding similarity rather than exact-string matching. A user asking "what is your refund policy?" and another asking "can I get a refund?" hit the same cache entry if their embeddings are within your similarity threshold. In production, semantic cache hit rates of 30-60% are realistic for customer-facing applications where query distribution is concentrated. Each cache hit saves the entire downstream pipeline cost.

The gateway is also where the AI gateway pattern — covered in depth in its own article — adds rate limiting at the model-provider level, which prevents you from inadvertently blowing through provider quotas in a traffic spike.

Layer 3: Context assembly

Context assembly is the layer with the biggest impact on quality and the one teams most often treat as an afterthought. It constructs the full prompt the model will see: system instructions, retrieved documents, conversation history, tool schema definitions, and memory injections.

In 2026, this is called context engineering — the deliberate design of what the model sees on each inference call. The shift in framing from "prompt engineering" matters. Prompt engineering was about writing better instructions. Context engineering is about architecting the entire information payload: what to include, what to exclude, in what order, and how to compress when you're over budget.

The token budget is your hard constraint. A 128,000-token context window sounds like infinite space until you do the arithmetic:

Context budget breakdown (illustrative, 128k context model)
  System prompt + instructions:     2,000 tokens
  Tool schema definitions:          3,000 tokens (10 tools × ~300 tokens each)
  Retrieved documents (RAG):       40,000 tokens (top-10 chunks × ~4,000 each)
  Conversation history:            20,000 tokens (last 20 turns)
  Output reserve:                  10,000 tokens
  ─────────────────────────────────────────────
  Total committed:                 75,000 tokens
  Remaining headroom:              53,000 tokens (41%)

  Problem: a 20-turn conversation with 10 tool calls grows at ~2,000 tokens/turn.
  After ~26 more turns, you're at the limit — before adding any new retrieved documents.

This is why context assembly needs compression logic: summarizing old turns, evicting low-relevance retrieved chunks, masking tool schemas not relevant to the current query. The context-window article covers the memory math; what matters architecturally is that this logic lives in one place, is versioned, and is testable independently of the model call.

The context assembly layer also handles just-in-time retrieval — fetching the right documents for this specific query rather than pre-loading the full knowledge base. The retrieval call goes out at assembly time, with caching for recent identical queries:

async def assemble_context(
    session: Session,
    query: str,
    tool_registry: ToolRegistry,
) -> ContextPackage:
    # Retrieve only what this query needs
    relevant_docs = await retriever.fetch(
        query=query,
        top_k=10,
        min_score=0.72,
    )
    
    # If history is over budget, swap in the compressed version.
    # Compression itself runs out-of-band, between turns, and caches
    # its result on the session — never inline in this hot path.
    history = session.history
    if token_count(history) > HISTORY_TOKEN_BUDGET:
        history = session.compressed_history  # written async by the summarizer
    
    # Mask irrelevant tools to save tokens and reduce tool confusion
    active_tools = tool_registry.filter_relevant(query, max_tools=8)
    
    return ContextPackage(
        system_prompt=SYSTEM_PROMPT,
        tools=active_tools.schemas(),
        documents=relevant_docs,
        history=history,
        query=query,
    )

One architectural decision worth making explicit: context assembly should make no synchronous model calls in the request path. It's a data assembly operation. Summarization and query expansion are model calls — so run them out-of-band: compress history asynchronously between turns and cache the result on the session, as the example above does. Putting those sub-calls inline turns context assembly into a latency multiplier — each one adds 300-800ms that the user feels on every request.

{ "type": "context-window", "window": 128000, "title": "Context budget under real load" }

Layer 4: Model execution

Model execution is the actual API call, and there is less to say about it than teams expect, because most of the interesting work has already happened in context assembly.

What belongs here: streaming response handling, retry logic for transient errors (429, 503), token counting on the response, and passing spans to the observability layer. What does not belong here: any business logic, response transformation, or validation — those go in the next layer.

Streaming matters more than most teams realize for perceived latency. A request that takes 8 seconds to generate 1,000 tokens feels slow if you wait for the full response; it feels fast if the first token arrives in 400ms. The engineering cost is real — you have to propagate the async stream through all downstream layers — but the UX improvement on long-form responses is typically the single highest-ROI latency optimization you can make.

async def execute_model(
    ctx: ContextPackage,
    gateway_client: GatewayClient,
    obs: ObservabilityClient,
) -> AsyncIterator[str]:
    span = obs.start_span("model_execution")
    
    try:
        stream = await gateway_client.stream(
            model="chat-standard",   # alias, not provider-specific name
            messages=ctx.to_messages(),
            tools=ctx.tools,
            max_tokens=4096,
        )
        
        total_tokens = 0
        async for chunk in stream:
            total_tokens += chunk.usage.output_tokens if chunk.usage else 0
            yield chunk.text
        
        span.finish(tokens=total_tokens, status="success")
    
    except ProviderError as e:
        span.finish(status="error", error=str(e))
        raise

The choice of max_tokens deserves attention. Setting it to the model's maximum (often 8,192-32,768 for current models) seems safe but has cost implications. Billing for output tokens runs 3-5× higher per token than input tokens at most providers (illustrative, mid-2026 pricing). An unconstrained agent that generates 10,000-token responses to simple questions will blow through your budget before you notice.

Layer 5: Output validation

Output validation is the layer most teams skip until it becomes a support ticket, a compliance finding, or a press story.

Three categories of checks run here:

Schema enforcement catches cases where the model was supposed to return structured JSON and returned something else — prose, partial JSON, or valid JSON with wrong field names. Use structured outputs where the provider supports it (Anthropic, OpenAI, Google all offer constrained decoding modes) and validate with Pydantic or equivalent as a backstop.

Guardrails check the response for policy violations: toxic content, competitor mentions if that's a business rule, responses that reference data from other users (a privacy red flag), or harmful instructions. NeMo Guardrails and Llama Guard are the two most common frameworks here, but you can also run a fast LLM-as-judge call against a policy rubric. The tradeoff: a guardrails LLM call adds 200-600ms and meaningful cost; a heuristic filter is fast but misses semantic violations. Most production systems run heuristics inline and the LLM guardrail call asynchronously for monitoring, catching violation patterns rather than blocking individual responses.

PII redaction is non-negotiable if your application could surface one user's data to another. Presidio from Microsoft is the standard open-source library; paid alternatives from Nightfall and Private AI offer higher accuracy on real-world PII patterns. Run this on output, not just input — the model can hallucinate plausible-looking but incorrect PII or reconstruct PII from partial hints in context.

What happens on validation failure is an architectural decision: fail closed (return an error to the user, log the attempt), fail open (return a degraded response with a disclaimer), or fail to a fallback (return a pre-written safe response for this category of query). There is no universally right answer; it depends on your application's tolerance for silence vs. incorrect responses. Pick a policy, write it down, and make it observable.

Streaming makes all of this harder, and the tension deserves naming: schema enforcement and PII redaction want the complete response, while streaming exists precisely so the user doesn't wait for one. Four workable strategies, in rough order of effort:

1. Buffer and flush. Collect the full response, validate, then send. Correct and simple, but you give back the 400ms-first-token win — acceptable for short responses (a couple of seconds of generation), painful for long-form.
2. Incremental checks on the stream. Run cheap validators (regex PII patterns, banned-string filters, running brace-balance for JSON) over a sliding window of tokens as they arrive, and kill the stream mid-response on a hit. Catches obvious leaks; misses anything semantic.
3. Stream now, judge async. Send tokens immediately, run the LLM guardrail after completion, and use hits for alerting and transcript redaction rather than blocking. This admits that some bad tokens reach users before you know. Most consumer chat products live here, usually combined with strategy 2 inline.
4. Don't stream structured outputs. A JSON payload has no partial-display value, so buffer it, validate against the schema, and return atomically.

Pick per endpoint, not globally: conversational endpoints stream with 2+3; anything machine-consumed uses 4.

Layer 6: Observability

Traditional APM gives you p50/p99 latency, error rate, and throughput. Those are necessary but not sufficient for an LLM application. A bot can sit at 0% error rate, 99.9% uptime, and 380ms p50 latency while every answer is confidently wrong. The observability layer needs four signal types that don't exist in traditional monitoring:

Output quality — faithfulness scores (does the answer match the retrieved documents?), relevance scores (does the answer address what was asked?), and user satisfaction proxies (thumbs-down rate, session abandonment after a response, follow-up clarification rate). LLM-as-judge evaluation gives you automated quality scores; LLM observability tooling covers the implementation in detail.

Behavioral drift — distribution shifts in response length, format adherence rate, refusal rate, or topic distribution. When these drift, it signals either a model version change, a prompt change that went unnoticed, or a shift in the query population. Detecting it requires baselining distributions at deploy time and alerting on deviation — standard for ML models but rarely applied to LLM applications.

Cost anomalies — per-request token spikes that indicate prompt injection attempts, runaway agent loops, or queries that hit a degenerate retrieval path and pull enormous documents into context. A per-request token histogram with p99 alerting catches these before they cause a budget overage.

Safety violation rate — guardrail trigger rates by category. An increase in toxic content attempts indicates an adversarial user or a compromised context source. An increase in PII detection hits suggests something upstream has started leaking sensitive data into the context pipeline.

The instrumentation approach that works in practice is OpenTelemetry spans per stage: one span for context assembly (with attributes for retrieved chunk count, token counts by section), one for model execution (with the model alias used, provider, and token usage), one for output validation (with pass/fail and which checks ran). This gives you distributed traces that work across the entire pipeline and integrate with existing observability stacks.

sequenceDiagram
    participant U as User
    participant GW as Gateway
    participant CTX as Context Assembly
    participant M as Model
    participant V as Validator
    participant OBS as Observability

    U->>GW: query
    GW->>OBS: span: gateway (cache miss, route=chat-standard)
    GW->>CTX: assemble
    CTX->>OBS: span: retrieval (12 chunks, 8,400 tokens)
    CTX->>M: assembled context
    M->>OBS: span: model (provider=anthropic, tokens=2,847)
    M->>V: token stream
    V->>OBS: span: validation (inline checks=pass, async guardrail queued)
    V->>GW: stream (inline-checked)
    GW->>U: streamed response
    OBS-->>OBS: async: quality score, cost attribution

Layer 7: Feedback store

The feedback store closes the loop. It collects signals — explicit (thumbs up/down, rating widgets, correction flows) and implicit (session abandonment, follow-up clarification queries, copy-paste of the response with edits) — and writes them to a durable store alongside the full request context.

The architectural requirement that most teams miss: the store needs to be queryable for training purposes, not just for dashboards. That means logging the assembled context (not just the user query), the raw model response before validation, validation results, and any user edits. Without that full tuple, the feedback data can't be used to generate training pairs for fine-tuning or preference optimization.

A feedback store that feeds nothing is expensive storage. The data flywheel — where feedback closes into fine-tuning signal that improves model quality, which attracts more users, which generates more feedback — is the compounding advantage that production LLM applications develop over time. The data flywheel article covers the curation and training side; architecturally, the point is that the store's schema and capture logic need to be designed for that downstream use from day one, not retrofitted six months later.

What breaks

Provider dependency without fallback

A single-provider setup with no fallback is a business continuity risk. Major LLM providers have had multi-hour outages. At 10,000 daily active users, a 4-hour outage during peak hours costs you 1,700 user sessions. With a fallback provider configured in the gateway, the same outage costs you the per-request latency increase (typically 100-400ms for secondary provider) while quality may vary slightly between models.

Context rot under growing history

Model quality measurably degrades as context window size grows, even on tasks the model handles perfectly with shorter context. This is "context rot" — not a theoretical concern but a documented production phenomenon. An agent that accumulates 80,000 tokens of conversation history over a long session will perform noticeably worse than the same agent at 10,000 tokens, even if the relevant information is in the most recent 5,000 tokens. The fix is context compression in the assembly layer: summarize and discard old turns, evict low-relevance retrieved chunks. Make context size a monitored metric, not just token cost.

Prompt changes without evaluation gates

Prompt changes — not model updates, not infrastructure changes — are the leading cause of silent quality regressions across large production LLM deployments (a pattern documented by ZenML across 1,200+ deployments in 2025). A prompt edit that seems obviously better in manual testing often has subtle effects on edge cases that only surface at scale. Every prompt change needs an evaluation gate before it reaches production users. The prompt versioning and CI/CD article has the full pipeline; the architectural point is that the observability layer needs to capture enough data — stored request-response pairs with quality scores — to run that evaluation gate.

Guardrails missing on the output path

Teams typically add input validation (checking user messages for obvious abuse) but skip output validation (checking model responses before returning them). This is backwards. The model is the more dangerous source: it can hallucinate sensitive data, produce policy-violating content in response to an innocuous-sounding query, or construct a response that technically answers the question but violates your legal compliance requirements. Output validation is not optional if you have any regulatory or brand-risk exposure.

The "single HTTP call" architecture at scale

Starting with a single module that calls the model directly is correct for a prototype. Staying there past the first 1,000 daily users creates problems that accumulate: scattered cost attribution, no fallback, no semantic caching, quality metrics only from occasional manual checks, no systematic feedback collection. Retrofitting the seven-layer architecture onto an existing application that has scattered direct API calls across multiple services is significantly more disruptive than building it in from the start. You don't need the full stack on day one, but the gateway and observability layers should be the first things you add after a working prototype.

Worked cost estimate

Here is a real sizing exercise for a mid-scale internal enterprise tool: 500 employees, 30 queries per day per active user, 60% daily active usage rate → 9,000 requests/day.

Naive architecture (single provider, no caching):
  Average input tokens per request:   3,500  (system prompt 500 + history 1,000 + RAG 2,000)
  Average output tokens per request:    800
  Input cost (illustrative mid-2026):  $0.003 / 1k tokens
  Output cost (illustrative mid-2026): $0.015 / 1k tokens
  Per request:  (3.5 × $0.003) + (0.8 × $0.015) = $0.0105 + $0.012 = $0.0225
  Daily:        9,000 × $0.0225 = $202.50
  Monthly:      ~$6,075

Optimized architecture (applied sequentially — each step only sees the
traffic the previous step didn't already remove):

  Step 1 — Semantic caching (30% hit rate on repeated queries):
    2,700 requests served from cache → $0 model cost
    Cache cost: ~$0.50/day (Redis)
    Paid traffic remaining: 6,300 requests/day

  Step 2 — Model routing (simple queries → fast model at $0.0003/1k input, $0.001/1k output):
    40% of remaining traffic classified as "simple" → 2,520 requests
    Fast model per request:  (3.5 × $0.0003) + (0.8 × $0.001) = $0.00185
    Fast-model spend:        2,520 × $0.00185 = $4.66/day
    Standard-model spend:    3,780 × $0.0225  = $85.05/day

  Step 3 — Prompt caching (system prompt 500 tokens, standard-model traffic):
    95% hit rate; cached tokens ~10% of input price at Anthropic/Google,
    so each hit nets a 90% saving on those tokens
    Net saving: 500 × 0.95 × 0.90 × 3,780 × $0.003/1k ≈ $4.85/day

  Optimized daily cost: $85.05 - $4.85 + $4.66 + $0.50 ≈ $85.40
  Monthly optimized cost: ~$2,560 (vs ~$6,075 naive)
  Effective reduction: ~58% (about 2.4x)

The ~58% cost reduction in this example requires roughly 6-8 weeks of engineering — about $3,500/month saved, forever. That math is almost always favorable. (Note the sequencing: sum each optimization against the full naive baseline instead and you'd claim ~73%, double-counting the overlap. Vendors do exactly this.)

A comparison of orchestration approaches

Not every LLM application needs the same level of complexity. Here is an honest map of what fits where:

Application type	Context assembly	State management	Orchestration needed
Single-turn Q&A / chatbot	Simple RAG retrieval + history	Session store (Redis)	Minimal — one model call per turn
Multi-step agent with tools	Tool masking, just-in-time RAG	Durable per-session state	LangGraph or equivalent for branching
Long-running background tasks	Full corpus access	Checkpoint-based (Temporal, Step Functions)	Workflow engine for reliability
Batch document processing	Document-specific context	Queue-based	Async queue + worker pool

The common mistake is reaching for LangGraph, Temporal, or a full agent framework for a single-turn chatbot. Those tools exist to solve state management for branching, multi-step workflows. For a turn-by-turn conversational assistant, they add complexity without corresponding benefit. Start with the simplest thing that works for your application type and add orchestration when you have multi-step workflows that need it.

{ "type": "model-router", "title": "Model routing: cost vs quality tradeoff" }

The decision in practice

The reference architecture is not a prescription to build all seven layers before shipping anything. It is a map of where things will break as you scale, so you can make deliberate choices about which layers to invest in and when.

On day one: build the model execution layer and a thin gateway with model aliasing. Cost: two to three days of engineering. Return: every future provider swap is a config change.

At 100 daily active users: add the observability layer, even if it's just request logging with token counts and a quality-score column you manually inspect weekly. You need the data before you need the alerting.

At 1,000 daily active users: add context assembly as a distinct, versioned module with its own tests. Add semantic caching. Add prompt versioning with evaluation gates — at this scale, a bad prompt deploy will reach hundreds of users before you notice.

At 10,000+ daily active users: the feedback store becomes a strategic asset. Instrument it properly, design the schema for training use, and start the data flywheel. The teams that do this at 10,000 users have a structural advantage over teams that try to retrofit it at 100,000.

The observation from production that should stick: the teams that ship lasting AI products are not the ones that pick the best model. They are the ones that build the architecture around the model — the layers that catch failures, measure quality, and compound improvement over time. The model is table stakes. The architecture is the product.