Eval First: Building the Safety Net Before You Need It

You changed one word in the system prompt on Tuesday. By Friday, refund approvals had doubled. Nobody noticed until finance flagged the anomaly.

This is not a hypothetical. It's the canonical form of silent regression in LLM applications. No exception was thrown. No error rate spiked. The latency numbers looked fine. The model just started interpreting "generous" differently — a word that crept into the rewritten prompt because it sounded friendlier. The old eval, which checked only whether responses were valid JSON, passed with flying colors.

The worst part: this failure mode isn't detectable with any of the instruments you're used to. Error rate, p99 latency, uptime — they all measure whether your system is responding. They don't measure whether it's doing the right thing. LLM quality is invisible to the observability stack you built for your API.

This module is about building the layer that makes quality visible, before you have a finance department calling.

Why this problem is actually hard

Classical software testing has a nice property: given the same input, you get the same output, and you can write assert result == expected and be done. LLM outputs have neither of those properties.

The outputs are non-deterministic. Run the same prompt twice and you get two slightly different responses. At temperature 0 they stabilize, but the temperature you're using in production is probably not 0.

There is no single right answer. Ask the model to summarize a policy document and there are thousands of acceptable summaries. "Correct" requires judgment, which brings you to the third problem: two trained domain experts looking at the same response will agree on its quality only about 81% of the time. That's not measurement error — that's irreducible subjectivity baked into the task.

And there are silent regressions. When a model provider does a silent update (they do, even when they promise they won't), when a new user cohort starts phrasing things differently, when a downstream context shift changes what "comprehensive" means — the regression doesn't announce itself. Your system keeps responding. Quality quietly erodes.

flowchart TD
    A[Prompt / model change] --> B{Offline CI eval}
    B -- "pass rate drops > threshold" --> C[Block merge]
    B -- "pass rate OK" --> D[Deploy]
    D --> E[Production traffic]
    E --> F{Online sampling + scoring}
    F -- "drift detected" --> G[Alert + investigation]
    F -- "metrics stable" --> H[Continue monitoring]
    style C fill:#ff2e88,color:#111
    style G fill:#00e5ff,color:#0a0a0f
    style H fill:#22c55e,color:#111

The fix isn't one layer. It's two: an offline gate that catches regressions you introduce, and an online monitor that catches what happens to you. Most teams have neither when they first ship.

The five-stage eval pipeline

Before building anything, it helps to have a mental map of the complete discipline. There are five layers, each catching different failure classes.

{ "type": "eval-pipeline", "title": "The five-stage eval safety net" }

Stage 1 — Unit evals during development. You write a test before you write (or change) the prompt, exactly like TDD. These are fast, cheap, run locally, and catch obvious failures: wrong format, missing required fields, hallucinated entity names you can verify.

Stage 2 — Pre-release adversarial testing. Before a feature ships, a small set of deliberately tricky inputs: ambiguous phrasing, long edge cases, inputs that broke the old system. You run these manually or in a small automated batch to probe failure modes the unit tests don't cover.

Stage 3 — CI regression gate. Every PR that touches a prompt, a model version, or retrieval logic triggers a run against your golden dataset. The score is compared to the main branch baseline. Below a threshold, the merge is blocked. This is the layer that would have caught the refund approval regression — if the golden dataset included refund scenarios.

Stage 4 — Production sampling. A fraction of live traffic (5–10% is typical) is scored asynchronously — either by a lightweight LLM judge or by collecting user signals like thumbs-down rates. This catches what offline tests cannot: new input patterns, model provider updates, distribution shifts as your user base grows.

Stage 5 — Runtime guardrails. Output filtering and validation that runs synchronously on every response. Not eval in the analytical sense — more of a hard stop on specific failure classes (PII in output, refusals incorrectly formatted, malformed JSON). Covered in depth in the guardrails module.

The mistake most teams make is thinking one layer is enough. Offline CI catches what you break; online monitoring catches what you didn't know was fragile. You need both.

Building a golden dataset that actually works

Here's the dirty truth about synthetic eval datasets: they're easy to build, they make your CI look thorough, and they catch the regressions you imagined having rather than the ones you'll actually encounter.

A synthetic dataset starts with your ideal user. The real dataset starts with your actual users. The gap between them is where silent regressions live.

The three-source mix for a useful golden dataset:

1. Hand-crafted edge cases — sit down with the product team and enumerate the boundary conditions: the longest reasonable input, the shortest, the one where the right answer contradicts the training distribution, the one where tone matters more than content. These take hours to write properly and they're worth it.

2. De-identified production samples — once you've been in production long enough to collect them, real user inputs are irreplaceable. Strip PII, then label each one with the expected behavior. These encode patterns you'd never invent.

3. Synthetic expansion for coverage gaps — after you've done steps 1 and 2, use a model to generate variants of underrepresented scenarios. Not as the foundation — as targeted gap-filling.

How many do you need? 50 examples is enough to catch a major regression with high confidence. 200 examples lets you detect a 5-percentage-point drop at statistical significance. 5,000 synthetic examples may detect nothing useful if they're all centered on the same distributional mode as your training inputs.

The other thing the synthetic-first approach misses: reference outputs that capture judgment. For each example, you don't just need an input — you need either a reference answer or a set of criteria that make evaluation possible without human review on every run. Writing those criteria is the hard part. It's also where the product team earns their keep.

Versioning and aging

Golden datasets get stale. As your product evolves, the examples that represented the typical input in month one become a poor proxy for the distribution in month twelve. The discipline is:

• Version your dataset (a YAML or JSON file in the repo, or a named snapshot in your eval tool).
• On each product milestone, review which examples are still representative and add examples from recent production traffic.
• Retire examples only when the feature they test no longer exists — old failure cases often resurface.

LLM-as-judge: making it trustworthy

Manually reviewing 200 outputs before every PR merge is not sustainable. The solution is using a separate LLM to score the responses — but an uncalibrated judge is worse than no judge, because it gives you false confidence.

As of 2026, a well-calibrated LLM judge achieves roughly 85% alignment with human reviewers — higher than the ~81% inter-human agreement rate. That's good enough to use in CI. The word "calibrated" is doing a lot of work there.

The five bias failure modes you need to know:

Bias	What it does	Mitigation
Narcissistic / self-preference	GPT-4 favors GPT-4 outputs ~10%; Claude v1 ~25%	Never use same family as judge and subject
Position bias	First response in a pair scores higher	Randomize or separately score each response
Verbosity preference	Longer answers score higher regardless of quality	Include conciseness criteria in rubric
Non-determinism	Same judge, same input, different score	Average 2-3 independent runs; use temperature 0
Coarse-grained collapse	Judges cluster around 3/5 or 4/5	Use binary or 1-5 with forced distributions

The most impactful change is moving from holistic scoring to chain-of-thought rubrics. Instead of asking "rate this response 1-5," you break the judgment into a sequence of binary questions: Does it answer the stated question? Does it cite only information from the provided context? Does it match the required tone? Each binary answer is auditable; the aggregate is interpretable.

JUDGE_PROMPT = """
You are evaluating a customer support response.
Answer each question with YES or NO, then give a one-sentence reason.

Response to evaluate:
<response>
{response}
</response>

Context provided to the model:
<context>
{context}
</context>

Evaluation criteria:
1. Does the response directly address the user's question? YES/NO — reason:
2. Does every factual claim in the response appear in the provided context? YES/NO — reason:
3. Is the tone professional and non-dismissive? YES/NO — reason:
4. If a refund or policy decision is mentioned, does it match the policy in the context? YES/NO — reason:

Final score: (count of YES answers) / 4
""".strip()

Calibration is non-negotiable. Before you trust any judge pipeline, you need to verify it agrees with your domain experts. The process:

1. Sample 30–50 diverse examples from your golden dataset.
2. Have 2–3 domain experts rate each one.
3. Run your LLM judge on the same set.
4. Compute Pearson correlation between judge scores and expert averages.
5. If r < 0.7, the judge is unreliable. Revise the rubric and repeat.

Few-shot examples in the judge prompt raise GPT-4 judge consistency from roughly 65% to 77.5% in practice. Add three to five well-chosen examples (including at least one where the right answer is "bad" and one where a short answer is "good") before you calibrate.

Wiring eval into CI: a runnable sketch

The following is a minimal pytest-based golden eval runner. It uses anthropic for the judge, but the pattern works identically with openai. This is meant to be runnable, not production-perfect.

import json
import pytest
import anthropic

client = anthropic.Anthropic()

# Load from your versioned golden dataset file
with open("evals/golden_dataset.json") as f:
    GOLDEN_CASES = json.load(f)

JUDGE_PROMPT = """
Evaluate whether the RESPONSE correctly and faithfully answers the USER_QUERY
given only the CONTEXT provided. Score each criterion YES or NO.

USER_QUERY: {query}
CONTEXT: {context}
RESPONSE: {response}

Criteria:
1. Addresses the question directly? YES/NO
2. Grounded in provided context only? YES/NO
3. No hallucinated facts? YES/NO

Reply as JSON: {{"scores": [true/false, true/false, true/false], "score": 0.0-1.0}}
""".strip()

def judge_response(query: str, context: str, response: str) -> float:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=256,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                query=query, context=context, response=response
            )
        }]
    )
    result = json.loads(msg.content[0].text)
    return result["score"]

@pytest.mark.parametrize("case", GOLDEN_CASES, ids=[c["id"] for c in GOLDEN_CASES])
def test_golden_case(case, your_system_under_test):
    response = your_system_under_test(
        query=case["query"],
        context=case["context"]
    )
    score = judge_response(case["query"], case["context"], response)
    # Hard floor: every case must pass at 0.67+ (2/3 criteria)
    assert score >= 0.67, (
        f"Case {case['id']} scored {score:.2f}\n"
        f"Query: {case['query']}\n"
        f"Response: {response[:200]}"
    )

The CI gate compares the aggregate pass rate against the main branch baseline. In GitHub Actions, you'd collect scores as JSON artifacts, then fail the job if the mean drops by more than your configured threshold (3 percentage points is a reasonable start).

One thing the sketch glosses over: the gate is itself non-deterministic. The system under test produces a different response on every run, and the judge scores it with its own noise — so a hard per-case assert will flap, and the same PR can go red on Tuesday and green on rerun. Three adjustments make the gate stable enough to trust. Gate on the aggregate pass rate, not on individual cases — or at minimum allow one retry per failing case before counting it. Pin the judge at temperature 0, or average 2–3 judge runs per case (the same mitigation from the bias table applies here). And respect the statistics: at an 85% pass rate over 200 cases, a single run's rate has a standard error of roughly 2.5 percentage points, which puts a 3-point drop barely above the noise floor. Treat a small drop as a trigger to rerun, and block only when it survives the rerun or clears twice your threshold.

The cost is real: at ~$0.003 per judge call, running 200 golden cases costs about $0.60 per PR. That's not nothing, but compare it to the cost of a refund policy regression running unchecked for three days.

The offline/online split: what each layer actually catches

flowchart LR
    subgraph "What you introduce"
        A[Prompt change]
        B[Model version bump]
        C[Retrieval config change]
    end
    subgraph "What happens to you"
        D[Provider silent update]
        E[New user cohort]
        F[Seasonal input shift]
    end
    A --> CI[Offline CI eval]
    B --> CI
    C --> CI
    D --> PROD[Online sampling]
    E --> PROD
    F --> PROD
    style CI fill:#0e7490,color:#fff
    style PROD fill:#a855f7,color:#fff

Your offline CI suite is a regression test. It answers: "did I break something?" It cannot answer: "is something breaking because of inputs I've never seen?" That's what online monitoring is for.

Online eval setup in practice:

1. Sample 5–10% of production requests (or 100% if volume is low enough).
2. For each sampled request, run the LLM judge asynchronously — never in the hot path.
3. Track the rolling pass rate over a sliding window (24 hours works for most products).
4. Alert when the rolling rate drops by more than your threshold versus the trailing 7-day baseline.

User signals — thumbs down, correction requests, escalations — are complementary, not replacements. They're noisy (most users don't rate anything) but they catch failure modes your judge rubric doesn't enumerate.

sequenceDiagram
    participant User
    participant App
    participant SampleQueue
    participant JudgeWorker
    participant Dashboard

    User->>App: Request
    App->>User: Response
    App->>SampleQueue: Enqueue 10% of requests
    SampleQueue->>JudgeWorker: Dequeue sample
    JudgeWorker->>JudgeWorker: Score with LLM judge
    JudgeWorker->>Dashboard: Write score + metadata
    Dashboard->>Dashboard: Update rolling pass rate
    Dashboard-->>App: Alert if drift detected

Tracing: the substrate everything runs on

Every eval system needs a substrate for collecting what actually happened — the full prompt sent, the model response, any retrieval context, tool calls in an agent chain, and the latency of each step. Without traces, you're scoring blindly.

Honest one-liners on the four dominant tools (specifics drift fast; check current docs):

• LangSmith — zero friction for LangChain teams; free tier (5K traces/month) exhausts in a day or two if you're sampling even modestly; the tight coupling is friction if you're not on LangChain.
• Langfuse — self-hostable, OpenTelemetry-compatible, ships with prompt versioning and inline LLM-as-judge scoring baked in; operationally heavier to run yourself; the open-source option is real.
• Braintrust — the most end-to-end: dataset management, scoring, CI gating, and production monitoring in one system; best fit for iterative prompt engineering workflows where you want one tool instead of three.
• Arize Phoenix — fully open-source, OTel-native, pairs with Promptfoo for a $0 eval stack; the choice if vendor lock-in is a hard constraint.

The underlying standard is OpenTelemetry GenAI semantic conventions — gen_ai.* spans that capture model, prompt, response, and now (as of OTel v1.39, 2025) MCP tool call observability. If your tracing tool emits these, you can switch between observability backends without rewing your instrumentation. That matters more than any feature comparison.

A minimal trace looks like this with the Langfuse Python SDK (v3, the OTel-based rewrite — the older langfuse.trace() API was removed):

import anthropic
from langfuse import Langfuse

langfuse = Langfuse()
client = anthropic.Anthropic()

def traced_completion(prompt: str, user_id: str) -> str:
    with langfuse.start_as_current_generation(
        name="support-response",
        model="claude-sonnet-4-5",
        input=prompt,
    ) as generation:
        generation.update_trace(user_id=user_id)

        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}]
        )
        output = response.content[0].text

        generation.update(
            output=output,
            usage_details={
                "input": response.usage.input_tokens,
                "output": response.usage.output_tokens,
            },
        )
        return output

That trace is now visible, searchable, and scoreable. The async judge worker picks it up, scores it, and writes the score back to the trace. From there, it feeds the rolling dashboard and eventually gets promoted into the golden dataset when it represents a new failure pattern.

The eval-driven development loop

The sequence matters as much as the tools. Here is the loop that makes eval a development habit rather than a one-time setup:

flowchart TD
    SPEC[Write the spec / criteria first] --> TEST[Add a golden case for the scenario]
    TEST --> IMPL[Implement the prompt change]
    IMPL --> CI[Run golden eval in CI]
    CI -- "regression" --> DEBUG[Debug: trace inspection + judge reasoning]
    DEBUG --> IMPL
    CI -- "passes" --> SHIP[Ship]
    SHIP --> PROD[Monitor online metrics]
    PROD -- "drift alert" --> TRIAGE[Triage: sample review + new golden case]
    TRIAGE --> TEST
    style SPEC fill:#00e5ff,color:#0a0a0f
    style SHIP fill:#22c55e,color:#111
    style TRIAGE fill:#ff2e88,color:#111

The key discipline: write the golden case before you write the prompt, for every new feature scenario. This is TDD for LLM systems. It forces you to define what "correct" means before you've anchored on any particular model output.

The feedback loop from production is equally important. When a live trace shows a new failure mode — one that your CI suite didn't catch — that trace (de-identified) becomes a new golden case. The dataset grows from production experience rather than developer imagination, and it stays representative of the inputs that actually matter.

What breaks in practice

Goodhart's Law, reliably. Once a team optimizes for the eval score rather than the task, prompts get tuned to pass the eval while actual user experience degrades. The symptom: eval pass rate climbs steadily while user satisfaction scores don't. The fix is rotating judge rubric examples periodically, adding blind human spot-checks, and treating any rapid jump in eval scores with suspicion.

Same-family judge inflation. In the original LLM-as-judge study (Zheng et al., 2023), self-preference was measured at ~10% for GPT-4 judging its own outputs and ~25% for Claude v1. Nobody has published equivalent numbers for every newer pairing — assume same-family pairings inflate similarly until you've measured yours. The mitigation is model-diverse judging or a fine-tuned open-source judge like Prometheus-7B for domains where you can afford the setup cost.

Golden dataset staleness. A dataset curated at launch represents the launch-era user. Six months later, new user cohorts phrase things differently, use your product in ways you didn't design for, and have different background knowledge. A static golden set develops blind spots in silence. Schedule quarterly reviews — it's one hour of work that has caught real regressions in every team that does it.

Offline-only blindness. Teams that only run CI evals miss provider silent updates entirely. Model provider version increments do not guarantee unchanged behavior — they've never guaranteed it. Online sampling is the only instrument that catches this class of regression.

Agent trajectory gaps. End-to-end task success is not enough for agents. A correct final output can mask broken intermediate reasoning. An agent that calls three unnecessary tools before arriving at the right answer looks identical to an efficient one in an end-to-end metric — until the chain is longer and the accumulated errors compound. This is covered in the agent evaluation deep-dive.

{ "type": "error-compound", "title": "Why agent step accuracy compounds to failure", "accuracy": 0.92, "steps": 8 }

At 92% per-step accuracy across an 8-step agent chain, end-to-end success rate is 0.92^8 ≈ 51%. That's not a bad agent — that's a good one. The math forces trajectory-level eval.

A worked cost estimate

Before you dismiss this as overhead: here's what the eval infrastructure actually costs at moderate scale.

Assumptions:
- 10,000 requests/day in production
- 10% sampled for online eval = 1,000 judge calls/day
- Golden dataset: 200 cases, run on every PR
- 5 PRs/week → 1,000 judge calls/week from CI
- Judge model: claude-haiku-3-5 at ~$0.0008/1K input tokens
- Average judge prompt: ~800 tokens input, ~100 tokens output

Online eval: 1,000 calls/day × $0.00064 input + $0.0004 output ≈ $1.04/day → ~$32/month
CI eval: 1,000 calls/week × $0.00064 + $0.0004 ≈ $1.04/week → ~$4.50/month

Total: ~$36/month for a real two-layer eval system.

The finance call from the refund regression probably cost more than that in one afternoon.

Where to go next

Why Eval Is Hard — the measurement problem in depth, including why benchmark scores are not application proxies and the Goodhart trap in detail.

LLM-as-Judge: Calibration — the five bias modes with calibration procedures, DAG decision trees, and fine-tuned open-source judge options for cost-sensitive deployments.

Golden Datasets — the complete recipe for sourcing, labeling, versioning, and growing a dataset that stays representative as your product evolves.

Offline vs. Online Evaluation — statistical sampling strategies, how to set drift alert thresholds, and what model provider silent updates look like in your metrics.

Tracing and Observability Tooling Compared — a current comparison of LangSmith, Langfuse, Braintrust, and Arize Phoenix with actual pricing and integration complexity, not vendor summaries.

Prompt Versioning and CI/CD — treating prompts as code: diff, review, gate, rollback. The engineering discipline that makes eval actionable rather than ornamental.

RAG Evaluation with Ragas — if your system retrieves before it generates, you need separate retrieval and generation metrics. Ragas gives you faithfulness, context precision, context recall, and answer relevance without requiring ground-truth answers for most of them.