LLMOps: Shipping and Operating AI Products

Nobody on your team changed anything. Your provider quietly rolled a new model snapshot behind the API version you thought you had pinned, and your extraction pipeline's accuracy slid from 96% to 89% over a weekend. Your dashboards show 99.8% uptime, p99 latency under 400ms, zero 5xx errors — because every one of those wrong answers returned HTTP 200. The first alert was a customer escalation, three weeks later.

That gap between "the system is up" and "the system is correct" is where LLMOps lives. It is not a set of best practices to bolt on after launch. It is the engineering discipline that makes AI products operable at all.

This is the last module in this course, and it pulls everything together. By the end, every prior module maps onto a concrete layer in a reference architecture you can actually build.

The reference architecture

Before going layer by layer, the full picture:

flowchart TD
    Client([Client]) --> GW[AI Gateway<br/>routing · rate limits · caching · cost]
    GW --> CA[Context Assembly<br/>retrieval · memory · tool masking]
    CA --> ME[Model Execution<br/>provider API · streaming · retries]
    ME --> OV[Output Validation<br/>schema · guardrails · PII redaction]
    OV --> Client
    GW --> OBS[Observability<br/>traces · quality scores · cost metrics]
    CA --> OBS
    ME --> OBS
    OV --> OBS
    OBS --> FB[Feedback Store<br/>implicit signals · annotation queue]
    FB --> EP[Eval Pipeline<br/>CI gate · regression · prod sampling]
    EP --> CA

Seven distinct layers. Each one has a reason to exist independently of the others, which means each one can fail independently. That's the point.

The context assembly layer is separate from model execution because you need to test retrieval quality without calling a frontier model on every test case — and because retrievers, rerankers, and tool schemas get updated on their own schedules. The output validation layer is separate from model execution because guardrails and PII redaction need to run regardless of which model generated the output. The observability layer reads from every other layer because quality signals come from everywhere — a retrieval failure looks different from a model failure looks different from a prompt failure, and you need to distinguish them.

Let's build each layer.

The AI gateway: your first infrastructure decision

The biggest mistake AI teams make in the first month is hardcoding provider names into application code. client = openai.OpenAI() scattered across twelve services means any provider migration is a multi-service code change, cost attribution is guesswork, and retry logic gets reimplemented in every service differently.

The fix is a gateway: a thin proxy every service calls instead of calling providers directly.

flowchart LR
    S1[Service A] --> GW[Gateway<br/>LiteLLM / Portkey / Kong]
    S2[Service B] --> GW
    S3[Service C] --> GW
    GW --> OAI[OpenAI]
    GW --> ANT[Anthropic]
    GW --> GGL[Google]
    GW --> OSS[Self-hosted<br/>vLLM / SGLang]
    style GW fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f

Your application code calls chat-standard or chat-fast. The gateway resolves that alias to the current best option — today Sonnet 4.5, tomorrow maybe a fine-tuned open-weight model — via config, not a code deploy. When a provider has an outage, the gateway routes to a fallback automatically. When you want to know which team is burning $8K/month on frontier model calls, the gateway has the attribution.

Semantic caching sits here too. Before forwarding a request, the gateway embeds the query and checks whether a semantically equivalent query was answered recently. In documented production deployments this cuts costs 30-60% for apps with repetitive query distributions — FAQ bots, support agents, internal search tools. The cache has no value for creative or personalized generation.

For open-source options, LiteLLM covers most of this. For managed options, Portkey and Cloudflare AI Gateway are reasonable. The mechanism matters more than the vendor here — read the AI gateway pattern article for the full decision matrix.

Context assembly: the 2026 discipline

Context engineering is the 2026 term for what used to be called prompt engineering. The shift is real, not just renaming. When your system is retrieving documents, managing conversation history, routing between tools, and assembling a final prompt, the question is no longer "how do I word this instruction?" It's "what does the model actually see on this call, and why?"

The context assembly layer is responsible for that composition. At inference time it:

1. Pulls the right documents from your vector store (retrieval coverage problem)
2. Compresses or selects from conversation history (context window budget problem)
3. Masks tools that aren't relevant for this query (tool confusion problem)
4. Injects fresh memory or facts that override stale knowledge (knowledge currency problem)

Each of these is independently testable, which is why the layer exists. You can run a retrieval quality eval — measuring faithfulness and relevance — without calling the model at all. That eval costs orders of magnitude less and runs in seconds.

The failure mode this layer is designed to prevent is context rot. As context grows, model quality measurably declines even on tasks the model handles well with smaller context. The inflection point varies by model and task type, but empirically you see quality drop before you exhaust the context window. Track context token count as a metric in your observability layer and plot it against quality scores. You'll find the cliff. Cap retrieval at that number, not at the technical maximum.

For a deep look at this, see context window management for agents and the module on RAG in one pass.

Prompts as code: versioning, CI, and staged rollout

Here is a claim that feels obvious in retrospect but gets ignored constantly: a prompt change is a deployment. It changes system behavior for every user. It can cause regressions that return HTTP 200. It needs review, testing, and staged rollout — exactly like code.

The process that works:

flowchart LR
    PR[Prompt PR<br/>git diff] --> EVAL[Eval gate<br/>score vs baseline]
    EVAL -->|passes| SHADOW[Shadow test<br/>dual-run, no user exposure]
    SHADOW -->|passes| CANARY[Canary rollout<br/>5-10% traffic]
    CANARY -->|passes| PROD[Full rollout]
    EVAL -->|fails| BLOCK[Blocked]
    SHADOW -->|degrades| ROLLBACK[Rollback]
    style BLOCK fill:#ff2e88,color:#000
    style ROLLBACK fill:#ff2e88,color:#000
    style PROD fill:#22c55e,color:#0a0a0f

The eval gate runs your candidate prompt against a golden dataset and scores outputs with an LLM-as-judge or rule-based metric. If the score falls below the baseline — even by one percentage point on a faithfulness metric — the PR is blocked. No exceptions for "minor wording changes." Every prompt change is a hypothesis. The eval gate is the test.

Shadow testing runs both the current and candidate prompts on every request but shows users only the current output. It is the highest-confidence stage because it exposes the candidate to real query distribution without real user impact. The cost is roughly doubled LLM spend during the shadow period — worth it before any user-facing exposure.

A concrete implementation in Python using the Anthropic SDK:

import anthropic
from eval_harness import score_outputs  # your internal scorer

# Point the SDK at your gateway, not the provider — the gateway
# resolves the "chat-standard" alias to whatever model is current.
client = anthropic.Anthropic(base_url="https://ai-gateway.internal")

def score_shadow_sample(
    queries: list[str],  # sampled from live traffic by the shadow harness
    current_prompt: str,
    candidate_prompt: str,
    threshold: float = 0.0,  # any drop vs current blocks; loosen only with a written reason
) -> dict:
    """Scoring core of a shadow test.

    The shadow harness dual-runs both prompts on live requests (users see
    only the current output) and periodically hands a sample here to decide
    whether the candidate is holding up.
    """
    current_outputs, candidate_outputs = [], []

    for query in queries:
        for prompt, outputs in [
            (current_prompt, current_outputs),
            (candidate_prompt, candidate_outputs),
        ]:
            response = client.messages.create(
                model="chat-standard",  # gateway alias, never a hardcoded model name
                max_tokens=1024,
                system=prompt,
                messages=[{"role": "user", "content": query}],
            )
            outputs.append(response.content[0].text)

    current_score = score_outputs(queries, current_outputs)
    candidate_score = score_outputs(queries, candidate_outputs)
    delta = candidate_score - current_score

    return {
        "current_score": current_score,
        "candidate_score": candidate_score,
        "delta": delta,
        "approved": delta >= -threshold,
    }

LLM-as-judge scorers agree with human raters roughly 80% of the time at orders-of-magnitude lower cost per eval. That tradeoff is worth taking for CI — you use human review for the golden dataset curation, automated scoring for the gate. The eval-first module and the prompt versioning article go deeper on building the golden dataset.

{"type": "eval-pipeline", "title": "The five-stage eval safety net"}

A/B testing prompts and models

Once you have a shadow testing infrastructure, A/B testing is the same mechanism with traffic split instead of full duplication. You route some percentage of real traffic to the treatment variant, score both in near-real time with automated evaluators, and declare a winner when you have statistical significance.

The catch: statistical significance for LLM quality metrics requires more samples than traditional feature flags because output variance is high. A 2-3 percentage point quality delta — meaningful in production — at 95% confidence requires thousands of samples, not hundreds. Plan test durations accordingly.

The metrics that matter for LLM A/B tests are not what you'd instrument for a button color:

Signal	Measure with
Answer quality	LLM-as-judge faithfulness + relevance score
Task completion	Session-level outcome (resolved / escalated / abandoned)
Cost efficiency	Tokens per session × model price
Latency	p50, p95 time-to-first-token and total
Safety	Guardrail trigger rate

Run infrastructure and quality metrics together. A variant that improves quality by 5% but costs 3× more might be the wrong tradeoff. A variant that cuts cost by 40% but increases session abandonment rate is a worse tradeoff. The A/B testing article has the full sample size math.

Observability: monitoring quality, not just uptime

Traditional APM tells you the call succeeded. LLM observability tells you whether the answer was any good. These are orthogonal signals and you need both.

The four signal types unique to LLM systems:

Output quality — faithfulness (does the answer match the sources?), relevance (does it answer the question?), and task completion rate. Measured with LLM-as-judge on a sampled fraction of production traffic — 5-10% is usually sufficient for trend detection.

Behavioral drift — the distribution of responses shifting over time. This happens when the query distribution drifts (new topics users are asking about), when retrieved document content changes, or when a model provider silently updates a model version. Embedding-based cluster analysis on response distributions is the practical detection method.

Cost anomalies — per-user, per-feature, and per-model token consumption. A single misguided agent loop that runs unconstrained can generate five-figure API bills overnight. Token spike alerting should fire in minutes, not at the end-of-month billing reconciliation.

Safety violations — guardrail trigger rate by category. Spikes in injection attempts or refusal rates are production signals, not just security events.

OpenTelemetry-based tracing works here: a span per prompt, per retrieval call, and per tool invocation gives you distributed traces through multi-step pipelines. LangSmith, Langfuse, Arize, and Helicone all instrument at this granularity. Datadog's LLM Observability product covers this if you're already standardized on Datadog for APM.

{"type": "token-cost", "title": "Monthly cost by input / output / cached segments"}

The feedback loop and the data flywheel

Every interaction your system has is a weak training signal. Most teams collect none of it. The ones that do usually fail to close the loop from feedback to training.

The data flywheel in production:

flowchart LR
    U[User queries] --> OUT[Model output]
    OUT --> FB[Feedback<br/>implicit + explicit]
    FB --> CUR[Curation<br/>dedup · filter · weight]
    CUR --> FT[Fine-tuning / DPO]
    FT --> MDL[Improved model]
    MDL --> OUT
    style CUR fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f

Implicit signals — session abandonment, follow-up clarifications, thumbs-down interactions, escalation to human support — are 10-100x more abundant than explicit ratings. Instrument both. Implicit signals need interpretation (a follow-up question might mean the answer was incomplete, or it might just be a new question), but volume matters for fine-tuning and preference optimization datasets.

The honest note: the bottleneck is annotation infrastructure. Collecting raw feedback is straightforward. Turning it into usable training signal requires deduplication (near-duplicate queries with identical answers add no signal), difficulty weighting (easy examples the model already handles correctly contribute less than near-misses), and coverage balancing (you want representative coverage of failure modes, not just the most frequent query type). Most teams treat annotation as a one-time step for launch and never rebuild it as the query distribution drifts.

The data flywheel pattern has been documented in several published production case studies showing 90%+ cost reduction when replacing frontier-API solutions with smaller domain-fine-tuned models after sufficient feedback loops — with quality equal to or better than the original. See the data flywheel article for the pipeline design.

What breaks in production

You now have all the pieces. Here is what actually goes wrong.

Prompt drift without evals. A team ships a prompt change on Friday, confident it's "just a wording fix." By Monday, session abandonment is up 12%. Nothing in the monitoring fired because quality metrics weren't instrumented. The first signal was support tickets. This is the most common production failure mode in LLM systems and it is entirely preventable with eval gates.

Hardcoded model names. model="gpt-4o" in twelve services. A provider deprecates that version, or a cheaper alternative appears. Every service needs a code change, a PR, a deploy. With a gateway and model aliases, it's a config update. The teams that learned this lesson did so the expensive way.

No cost attribution. The API bill arrives. It is $80K. You cannot tell which team, which feature, or which agent loop is responsible. With a gateway, every request carries team and feature metadata. Without one, you're guessing.

Eval dataset rot. The golden dataset was built at launch from 200 curated examples. The product has been running for eight months. The query distribution has shifted substantially — users are now asking about features that didn't exist at launch. The evals still pass. The evals are measuring the wrong thing. Rebuild your eval dataset on a cadence tied to query distribution drift, not the calendar.

Context rot. Retrieval returns 20 chunks. The model answers correctly on 10 of them. Quality monitoring shows p75 scores fine. p95 is quietly declining. No one notices until a customer escalates. Set hard limits on context size based on empirical quality inflection points, and monitor context size as a metric.

The feedback loop that doesn't close. User feedback is collected and stored in a database. No one has ever looked at it. The fine-tuning pipeline was built "for later." Later is now 14 months ago. The model is the same as launch. This is not a data problem; it's a process problem. Assign ownership of the flywheel explicitly.

The prototype-to-production checklist

Before you call something production-ready:

• Gateway in place — no service calls providers directly
• Model aliases configured — app code references chat-standard, not claude-sonnet-4-5
• Eval gate in CI — prompt changes blocked without baseline eval pass
• Golden dataset — at minimum 200 examples, covering known failure modes
• Observability instrumented — quality score sampling, cost by feature, context size tracking
• Guardrails deployed — input and output validation in code, not in prompts
• Cost alerting — per-user and per-feature token spike alerts with human-review thresholds
• Feedback collection — implicit signals instrumented and stored
• Rollback plan — previous prompt version tagged and deployable in under 5 minutes
• Runbook — what to do when quality drops, costs spike, or a provider goes down

The prototype to production article has the scaling decisions — caching strategy, async batching, model routing by complexity — that come after this checklist.

Mapping this course to the architecture

This is module 14, which means you now have the full vocabulary. Here is how every prior module maps onto the reference architecture:

Layer	Course coverage
AI Gateway	Cost control and reliability engineering — routing, fallbacks, circuit breakers
Context Assembly	RAG in one pass, Embeddings and vector databases, Context engineering
Model Execution	The prediction loop, Transformers without the math, Tokens, costs, context window
Output Validation	Guardrails and security — input/output guardrails, PII redaction
Eval Pipeline	Eval first — golden datasets, LLM-as-judge, CI gates
Observability	Cost control and reliability engineering, this module
Feedback Store	This module — the flywheel and annotation infrastructure
Agent orchestration	Your first production agent — loop, memory, tool use
Model selection	The model landscape, Reasoning models, Fine-tuning decision framework

None of these layers are optional in production. Some can be simpler early on — a gateway can start as LiteLLM with two provider configs; an eval pipeline can start as fifty hand-labeled examples and a one-line pytest assertion. But each layer needs to exist, because each one fails independently.

Where to go next

LLM app reference architecture — the canonical seven-layer blueprint with decision criteria for each layer's design. Read this before finalizing any production architecture.

Prompt versioning and CI/CD for LLM changes — the full eval-gated deploy pipeline, with tooling comparison across Braintrust, LangSmith, and Agenta.

LLM observability: monitoring quality, cost, and drift — OTEL-based tracing implementation and the four signal types in more detail, including hallucination detection patterns that go beyond keyword matching.

A/B testing prompts and models in production — the sample size math, the statistical pitfalls specific to LLM quality metrics, and the tooling that integrates with existing feature flag infrastructure.

The data flywheel: closing the feedback loop — the annotation infrastructure design, the curation pipeline, and how to connect implicit feedback to preference optimization.

The AI gateway pattern — gateway implementation details, semantic caching configuration, and the cost attribution model.