Offline vs Online Evaluation: Building Your Two-Layer Safety Net

On a Tuesday in March, your support bot's answers get worse. Not broken — worse. Slightly off-topic, oddly hedged, missing the clarifying question it used to ask. You deployed nothing that week. Latency is fine, error rate is zero, CI is green. Two weeks later a provider changelog entry confirms a model revision rolled out behind the alias you call, and you realize you cannot say which of the 40,000 conversations since then went badly, because nothing you run scores production traffic.

That is not a monitoring problem. That is an eval infrastructure problem. Standard observability catches when your system breaks. Eval catches when your system quietly gets worse — which is the more common failure mode for LLM applications.

The answer is two evaluation layers: one that runs before you ship, one that runs after.

What offline evaluation is and what it is not

Offline evaluation is a fixed test suite — a golden dataset — that you run against your full system stack before any change lands in production. Input goes in, output comes out, a scorer compares the output to a rubric or reference answer, and if the aggregate score drops below a threshold the PR is blocked.

The word "offline" here means the data is fixed, not that the evaluation happens on a laptop. You run it in CI on your actual deployment infrastructure, hitting your actual retrieval stack and your actual model. The "offline" refers to the data being historical or hand-crafted, not sampled from real-time traffic.

# Minimal offline eval harness (illustrative structure)
import asyncio
from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalCase:
    input: str
    expected_criteria: str  # rubric, not exact match
    tags: list[str]

async def run_offline_eval(
    cases: list[EvalCase],
    system_under_test: Callable[[str], str],
    judge: Callable[[str, str, str], float],
    threshold: float = 0.75,
) -> dict:
    scores = []
    failures = []

    for case in cases:
        output = await system_under_test(case.input)
        score = await judge(case.input, output, case.expected_criteria)
        scores.append(score)
        if score < threshold:
            failures.append({"case": case, "output": output, "score": score})

    aggregate = sum(scores) / len(scores)
    return {
        "aggregate_score": aggregate,
        "passed": aggregate >= threshold,
        "failure_count": len(failures),
        "failures": failures,
    }

Offline eval is specifically good at catching regressions you introduce: a system prompt change that accidentally truncates instructions, a chunking config tweak that degrades retrieval, a model version upgrade that changes tone in ways your product specs prohibit. Every time a developer makes any of these changes, the CI gate runs the golden dataset and flags the drop before it ships.

What it cannot catch: anything that changes in production without a corresponding code change. And there is more of that than most teams expect.

What breaks in production without a code change

This section could just be titled "things that humiliated teams who only did offline eval." Three categories dominate:

Model provider silent updates. Undated model aliases — gpt-4o, gemini-2.5-flash-style names that float to the latest revision — are moving targets: providers roll safety tuning passes, RLHF updates, and alignment changes behind them without changing the name you call. Dated snapshots like gpt-4o-2024-11-20 are the mitigation, and providers treat the pinned weights as a contract — but serving-stack changes (quantization, batching, decoding infrastructure) have occasionally shifted behavior even on pinned versions. Changes show up in provider changelogs after the fact, if at all. If you are not watching production quality trends, you find out when a user screenshots something odd.

Input distribution shift. Your golden dataset was constructed based on the queries your early users sent. Three months later, a TechCrunch article runs, a different user cohort finds your product, and they phrase things completely differently. Your eval suite keeps passing — those inputs are not in it. Real-user quality is declining. The only way to see this is to score actual traffic.

Compounding retrieval drift. For RAG applications, the documents in your index change. New policies are added, old ones are archived, the writing style of your knowledge base shifts. Retrieval that worked well on the old corpus starts returning marginally worse chunks. No code changed. No model changed. The answer quality degrades anyway.

None of these are hypotheticals. They are the reason that teams who skip online evaluation consistently report "we didn't know anything was wrong until a user complaint spike."

Building the offline eval layer

The offline layer has three components: a golden dataset, a scorer, and a CI integration.

The golden dataset

The golden dataset article covers construction in depth, but the eval-layer perspective is: you need coverage across the types of failures that matter for your product, not just a sample of typical happy-path inputs.

Three source types compose a healthy dataset:

1. Hand-crafted edge cases — inputs that probe known failure modes: adversarial phrasing, ambiguous queries, inputs that trip up your retrieval, multi-step questions your model handles poorly. These are the cases a human expert writes specifically to find weaknesses.

2. De-identified production samples — real user queries, stripped of PII, labeled with expected outputs or rubrics. These capture the actual distribution your users send, including phrasings you would never think to write.

3. Synthetic expansion — programmatically generated variants of known hard cases to fill coverage gaps without the cost of full human annotation.

The minimum viable size is around 50 examples for detecting major regressions (>15% quality drop). Two hundred examples lets you detect smaller shifts and segment by failure type — you can see that your retrieval-heavy cases degraded while your direct-answer cases stayed stable, which tells you where to look.

Version-control the dataset alongside your code. When a test case becomes irrelevant (the product no longer does that thing), retire it explicitly rather than silently deleting it. A shrinking test suite with a rising score is a red flag, not a win.

The scorer

For most non-code, non-structured-output tasks, you need an LLM judge. The LLM-as-judge article covers calibration in depth — but the short version is: your judge needs to be calibrated against domain expert verdicts on at least 30 examples before you trust it to gate production. An uncalibrated judge with a Pearson correlation below 0.7 against human ratings is generating noise, not signal.

For offline eval specifically, a few additional properties matter:

• Consistency across runs. Set judge temperature to 0 and use deterministic sampling. Offline eval needs to reproduce: if the score changes between runs with no code changes, you cannot tell signal from variance.
• Rubric specificity. Vague rubrics ("is this a good answer?") produce noisy scores. Criteria derived from your actual product spec ("the answer cites a policy from the provided context and does not add information not present in the retrieved documents") are far more reliable.
• Different model family from the subject. If you are evaluating GPT-4o outputs, judge with Claude or Gemini. Zheng et al. (2023) measured this self-preference bias directly: GPT-4 favored its own outputs by ~10%, Claude-v1 by ~25%. The magnitudes vary by model generation, but the direction — judges prefer their own family — has held. Judge with a different family.

For structured-output tasks — JSON extraction, classification, code generation — skip the LLM judge and use deterministic scorers: exact match, schema validation, unit tests. These are cheaper, faster, and perfectly reliable. LLM judges are for the tasks where "correctness" has no single right answer.

Wiring into CI

The CI integration is simpler than most teams expect. The key decisions:

• Blocking threshold. Set a threshold you would actually block on (e.g., aggregate score < 0.75 or more than 10% of cases below 0.6). If you set it too low to ever block, you have a metric, not a gate.
• Fast enough to fit in PR review. At 200 test cases, a judge call per case, with a fast judge model and async execution, you are looking at 2–5 minutes of wall-clock time. That is acceptable. At 2000 test cases with a slow judge, you get a 40-minute CI step that developers start skipping.
• Segment the report. A single aggregate score hides where the regression is. Report pass rates by tag (e.g., "retrieval-heavy" vs "direct-answer"), so the developer who broke the retrieval config can see it immediately.

sequenceDiagram
    participant DEV as Developer
    participant CI as CI Pipeline
    participant SUT as System Under Test
    participant JUDGE as LLM Judge
    participant GH as GitHub / PR

    DEV->>GH: open pull request
    GH->>CI: trigger eval workflow
    CI->>SUT: run 200 test cases (async batch)
    SUT-->>CI: 200 outputs
    CI->>JUDGE: score each output vs rubric (async)
    JUDGE-->>CI: 200 scores
    CI->>CI: aggregate by tag, compare to threshold
    alt score >= 0.75
        CI-->>GH: check pass — allow merge
    else score < 0.75
        CI-->>GH: check fail — block merge, post failure report
        GH-->>DEV: PR blocked, here are the failing cases
    end

Tools like Braintrust, Langfuse, and Promptfoo all have native CI integrations that manage this loop — they store the dataset, run the eval, compare to a baseline, and post the diff to the PR. The observability tooling comparison covers tradeoffs between these platforms.

Building the online eval layer

Online evaluation samples a fraction of live production traffic — typically 1–10% depending on volume — scores each sampled trace asynchronously, and feeds a quality dashboard you watch for trends.

The architecture has four components: sampling, async scoring, storage, and alerting.

Sampling strategy

Pure random sampling is fine for catching broad quality trends. But for catching rare failure modes — the long-tail queries that happen 0.1% of the time but fail spectacularly — stratified sampling by input characteristics (query length, topic cluster, user segment) catches problems that random sampling at the same rate would miss.

At 10,000 requests per day with 10% random sampling, you score 1,000 traces per day. That is enough to detect a 10% quality drop within 24–48 hours. At 1 million requests per day with 1% sampling, you score 10,000 traces — better statistical power than most offline datasets, running continuously.

The sampling decision should happen at the gateway layer, not inside your application. Intercept the request and response at the LLM gateway or observability SDK, tag it for eval, and emit it to an async queue. The application never blocks on eval.

Async scoring

The scoring loop is intentionally asynchronous — you never want eval latency on the critical path of a user request. The typical pattern:

# Async online eval consumer (illustrative)
import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic()

async def score_trace(trace: dict) -> dict:
    """Score a sampled production trace offline."""
    judge_prompt = f"""You are evaluating a customer support response.

User query: {trace['input']}
System response: {trace['output']}

Criteria:
- Does the response directly address the user's question? (0-1)
- Is every claim grounded in the retrieved context? (0-1)  
- Is the tone appropriate for customer support? (0-1)

Return JSON: {{"address": 0-1, "grounded": 0-1, "tone": 0-1, "overall": 0-1, "reason": "..."}}"""

    response = await client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=256,
        temperature=0,
        messages=[{"role": "user", "content": judge_prompt}],
    )
    # parse and return score
    return {"trace_id": trace["id"], "scores": parse_json(response.content[0].text)}

User signals — thumbs up/down, follow-up clarification requests, escalations to human agents — are a free online eval signal. They are noisy and sparse, but they are ground truth in a way that judge scores are not. Wire them in alongside judge scoring and weight them heavily when they appear.

What to watch on the dashboard

The most useful signal is not the absolute score but the trend. A quality score that was 0.81 last week and is 0.74 this week is a problem even if 0.74 sounds acceptable in isolation. Set alert thresholds on the delta as much as on the absolute value.

Segment the trend line the same way you segment your offline report. If overall quality is stable but quality on "policy questions" dropped 12 points, the retrieval index probably has a problem. If quality on "short queries" dropped while "long queries" held, something in your prompt handling changed.

The eval-pipeline visualizer shows the five-stage structure interactively:

{ "type": "eval-pipeline", "title": "Five-Stage Eval Safety Net" }

Canary and shadow evaluation for risky changes

There is a gap between "CI passed" and "deploy to 100%" that the two layers alone do not cover. The drift-detection math above cuts both ways: at full traffic, a 10% quality drop takes 24–48 hours to surface on the dashboard, and by then every user has seen the regressed behavior. For risky changes — a model version upgrade, a prompt rework, a new retrieval strategy — route a canary slice first: send 5% of traffic to the new version and score both arms with the same online judge rubric. At 10,000 requests per day, a 5% canary scored at 100% sampling gives you ~500 scored traces per day per arm — enough to spot a meaningful quality gap within a day or two while 95% of users stay on the known-good version. Promote on parity or better; roll back on a gap.

For changes too risky even for a canary, shadow traffic goes further: send each request to both versions, serve the old response to the user, and score the new version's answer async. Zero user exposure, at the cost of doubling inference spend on the shadowed slice. Either way, there is nothing new to build — it is the sampling, judging, and dashboard segmentation you already have, pointed at two arms instead of one.

The feedback loop that closes the gap

The two layers are not just additive — they compound when you wire them together correctly.

When online eval detects a new failure pattern — a class of queries your judge consistently scores below 0.6 — you:

1. Pull a sample of those traces from the observability store.
2. De-identify them (strip any PII, user identifiers, session data).
3. Validate the expected behavior against your product spec.
4. Add them to the golden dataset.
5. The next PR that touches anything related to those queries will trigger the CI gate on these new cases.

This is how the offline suite grows to match your actual user base rather than the imaginary users you had at launch. It is also why the golden dataset needs versioning — you want to know which cases were added when, so you can attribute a CI failure to a specific new case rather than a general regression.

flowchart TD
    ONLINE["Online eval\ndetects new failure pattern"]
    SAMPLE["Pull sampled traces\nfrom observability store"]
    DEIDENT["De-identify\n(strip PII)"]
    VALIDATE["Human validates\nexpected behavior"]
    ADD["Add to golden dataset\n(versioned)"]
    CI["CI gate now covers\nthis failure class"]

    ONLINE --> SAMPLE
    SAMPLE --> DEIDENT
    DEIDENT --> VALIDATE
    VALIDATE --> ADD
    ADD --> CI
    CI -->|next regression caught before ship| ONLINE

    style ONLINE fill:#0e7490,color:#fff
    style CI fill:#15803d,color:#fff

The loop takes discipline to maintain. Teams that let it slip — adding cases only when something catastrophically fails, letting the golden dataset stagnate at its launch composition — find their offline CI gate becoming increasingly irrelevant. The score stays high while real-world quality diverges.

What breaks: failure modes for each layer

Offline eval failure modes

Goodhart's Law in action. When the eval score becomes the explicit goal — teams tune prompts against the test set rather than against the actual task — the score rises while real quality doesn't. The fix is test set opacity: developers should not see individual test cases, only aggregate failure reports. Treating the golden dataset as a private fixture like a held-out test set, not a public benchmark to optimize against.

Static dataset, drifting world. A golden dataset that was built eighteen months ago and never updated is evaluating a product that no longer exists against users who no longer represent your user base. Coverage gaps accumulate silently. This is not a hypothetical — it is the most common way that "we have an eval suite" becomes "our eval suite is useless."

Threshold set too low to matter. If your CI gate only blocks at aggregate score < 0.50 and your system routinely scores 0.82, the gate will never fire. Set thresholds that are at the upper end of your observed score distribution, not at an absolute floor.

Slow CI step that gets skipped. A 45-minute eval suite becomes the thing that "only runs on main, not on PRs." Test suite speed is a product feature. Use a fast judge model, async batching, and aggressive parallelism to stay under 5 minutes.

Online eval failure modes

Sampling bias toward easy cases. If your sampling is purely random and your traffic is 80% simple queries, you will over-represent simple queries in your online eval. The failure patterns hiding in the 5% complex multi-hop queries need stratified sampling or explicit filtering to surface.

Judge drift. The judge model you use for online eval can itself get silently updated by its provider. If you use gpt-4o as your judge and OpenAI updates it, your quality trend line now has noise from two sources: your system and your evaluator. Pin the judge model version explicitly. Use a versioned snapshot endpoint where available.

Alert fatigue from noisy scores. If your judge produces high variance — the same trace scores 0.6 one day and 0.8 the next — alerts on daily averages will misfire constantly. Smooth with a rolling 7-day average, and alert on the rolling average dropping below a threshold rather than on any single day's number.

PII leakage into the eval store. Production traces contain user data. A pipeline that naively stores them for async scoring can inadvertently build a secondary store of sensitive user queries that was never in scope for your privacy review. De-identification must happen before storage, not after.

Integrating with your observability stack

Eval data flows through the same observability infrastructure as traces, metrics, and logs. The emerging standard is OpenTelemetry GenAI semantic conventions (gen_ai.* spans), which stabilized in 2025 with MCP tool-call observability added in OTel v1.39. These conventions define span attributes like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.response.finish_reason in a vendor-neutral format that tools like Langfuse, Arize Phoenix, and Braintrust all consume.

The practical consequence: if you instrument your LLM calls with OTel GenAI attributes, your traces flow into any of these platforms without vendor lock-in. You can switch from Langfuse to Arize without re-instrumenting your application. You just point the OTel exporter at a different endpoint.

flowchart LR
    APP[Application] -->|OTel GenAI spans| OTEL[OTel Collector]
    OTEL --> LF[Langfuse]
    OTEL --> ARIZE[Arize Phoenix]
    OTEL --> BT[Braintrust]
    LF --> EVAL_STORE[Eval store + judge scoring]
    ARIZE --> EVAL_STORE
    BT --> EVAL_STORE
    EVAL_STORE --> DASHBOARD[Quality dashboard]
    EVAL_STORE --> CI_GATE[CI offline gate]

    style OTEL fill:#0e7490,color:#fff
    style EVAL_STORE fill:#15803d,color:#fff

Instrumenting your application to emit these spans is a one-time setup cost. Every LLM call, retrieval step, and tool invocation becomes a traced, queryable event. You can pull any production trace for manual review, filter by score bucket to find the worst-performing calls, or replay a trace against a new model version to see if it fixes the problem. The observability tooling comparison walks through how Langfuse, Braintrust, Arize, and LangSmith differ in practice.

The decision in practice: what to build first

If you are at an early stage with your first LLM feature in production, the priority order is:

Build offline eval first. A 50-case golden dataset with an LLM judge and a CI gate takes a day to set up and immediately eliminates an entire class of "we didn't notice the prompt change broke everything" incidents. Tools like Promptfoo, Braintrust, and Langfuse all have reasonable free tiers to start.

Add user signal collection in parallel. Thumbs up/down, escalation events, clarification follow-ups — wire these into your database before you need them. They cost nothing to collect and become invaluable when you need ground-truth signal.

Build online eval scoring once you have traffic. At fewer than a few hundred requests per day, the statistical power of online eval is too low to be useful. Wait until you have enough volume for a 5–10% sample to give you a few dozen scored traces per day. At that point, set up async LLM judge scoring against the same rubrics you use offline.

Close the feedback loop once both layers are running. Review the online eval dashboard weekly. Pull the lowest-scoring traces. Add the representative ones to the golden dataset. The offline gate gets stronger, the online eval surface area shrinks, and you ship with more confidence over time.

This is the pattern described in eval-first thinking: the sooner you build the infrastructure, the more quality history you have banked before your first real scaling crisis — data you cannot retroactively collect. Teams that wait until they have a problem to measure find themselves triaging quality issues without the historical data to understand when they started.

The eval discipline is what separates AI products that improve over time from ones that slowly drift until users stop trusting them. Offline and online together form a closed loop — a two-layer safety net that catches the regressions you introduce and the ones the world introduces for you.