Golden Datasets: Building the Ground Truth Your Evals Depend On

The email arrived on a Tuesday morning. An enterprise customer's support team reported that the AI assistant had stopped citing policy numbers in its answers — instead it was paraphrasing them correctly but without the specific codes the compliance team needed for their audit trail. Nothing crashed. Error rate: 0%. Latency: normal. The regression had been live for three weeks, introduced when someone updated the system prompt to "sound more natural." There was no golden dataset. There was no CI gate on prompt changes. There was no way to know.

This is not an unusual story. It is the default story when teams treat eval as something to add later.

What a golden dataset actually is

A golden dataset is a curated, versioned collection of (input, expected-behavior) pairs that defines "correct" for your specific application. The word "golden" signals that it is the fixed reference — not something regenerated on each run, not the leaderboard benchmark your model provider already optimizes for, not a randomly sampled slice of your logs. You built it deliberately, you know what it covers, and you can reason about its gaps.

The contrast with general benchmarks matters. MMLU tests breadth of world knowledge. HumanEval tests code generation accuracy. Neither tells you whether your compliance assistant correctly cites policy codes for the questions your specific enterprise customers actually ask. Benchmark scores measure model capability in the abstract; your golden dataset measures product correctness for your users on your task. The gap between the two is where silent regressions live.

A golden dataset entry typically looks like this:

# One row in your dataset (JSONL format, Git-tracked)
{
  "id": "policy-cite-0042",
  "version": "v3",
  "input": {
    "system": "You are a compliance assistant for Acme Corp...",
    "user": "What is the data retention policy for EU customer records?"
  },
  "reference_answer": "EU customer records must be retained for 7 years under GDPR Article 5(1)(e) per Acme policy POL-DPR-2024-03.",
  "evaluation_criteria": [
    "Answer must include the retention period (7 years)",
    "Answer must cite GDPR Article 5(1)(e)",
    "Answer must include the Acme internal policy code POL-DPR-2024-03",
    "Answer must not contradict the reference policy document"
  ],
  "tags": ["policy-citation", "eu-compliance", "critical"],
  "added_by": "human",
  "added_date": "2026-02-14",
  "source": "production-sample",
  "fail_threshold": "any"  # fail if any criterion is unmet
}

The evaluation criteria field is where specification-driven evaluation lives. Rather than a vague holistic "is this good?", each criterion maps to a specific clause in your product spec. When a criterion fails, you know exactly which part of the specification is violated.

The three-source construction model

No single source covers all the failure classes you care about. Golden datasets built from only one source type develop systematic blind spots.

Source 1: Hand-crafted edge cases

These are written by engineers and product owners who understand what your system promises. They start from your product specification: what does the system guarantee to do? What are the boundary conditions? What are the inputs that stress those boundaries?

For a customer support assistant: What happens when the user asks about a policy that changed last quarter? What if the user's question is ambiguous between two policies? What if the user asks in the wrong language? What if they paste a URL into the question?

Hand-crafted cases are slow to produce — expect 30-60 minutes per high-quality example including writing the criteria — but they are the only source that systematically covers intentional product guarantees. They encode knowledge that is not in your logs because these are the failure modes that have not happened yet or have happened but were never logged.

A useful starting structure is to derive cases from a simple spec statement expansion:

Product spec statement: "The assistant must cite the specific policy code for any claim about internal procedures."

Edge cases to write:
  - User asks about a procedure that exists → must cite code ✓
  - User asks about a procedure that changed last quarter → must cite NEW code, not old
  - User asks about a procedure in a jurisdiction we don't cover → must say so, not hallucinate a code
  - User's question mentions the wrong procedure name → must clarify, not silently use the wrong one
  - Two procedures apply → must cite both, not arbitrarily pick one

Five spec statements with 4-6 cases each gives you 20-30 solid hand-crafted examples to start with.

Source 2: De-identified production samples

Production traffic shows you what users actually ask, which is reliably different from what you imagined they would ask. After a few weeks in production, you will have examples your hand-crafted set never anticipated: unusual phrasings, questions that span two product domains simultaneously, questions submitted in French when you only tested English, questions that accidentally trigger your guardrails, questions that reveal a product knowledge gap.

The collection pipeline works like this:

sequenceDiagram
    participant P as Production Traffic
    participant S as Sampling Filter
    participant H as Human Reviewer
    participant D as Golden Dataset

    P->>S: Continuous stream of traces
    S->>S: Sample by: low-scoring (auto-judge <0.7), flagged by users, random 1%
    S->>H: Candidate examples (with PII stripped)
    H->>H: Review: is this example representative and correctly labeled?
    H-->>D: Approved examples added with criteria
    Note over D: Git commit, versioned

PII removal is not optional — you need a scrubbing step before any production sample enters your dataset. The standard approach: regex for obvious patterns (email, SSN, phone, credit card), followed by an NER pass to catch names and addresses, followed by human spot-check on a 10% audit sample. Tools like Presidio handle the automated layer; the human audit is what catches the edge cases the automation misses.

The budget for this collection process matters. At human reviewer cost of roughly $50/hour and 10 minutes per example: 200 production-sourced examples costs about $1,700 in reviewer time. That is a one-time investment that pays for itself the first time it catches a regression before it ships.

Source 3: Synthetic expansion

Your hand-crafted and production-sampled sets will still have coverage gaps: scenarios that are plausible but rare, adversarial inputs you want to stress-test, linguistic variations you want to confirm work. Synthetic expansion fills these gaps by generating variations programmatically.

import anthropic

client = anthropic.Anthropic()

EXPANSION_PROMPT = """
You are helping build an evaluation dataset for a compliance assistant.

Here is a representative example:
Input: {original_input}
Criteria: {criteria}

Generate 5 variations of this input that test the same underlying criteria but with:
1. Different phrasing / sentence structure
2. The same question with typos, abbreviations, and inconsistent capitalization
3. A user who seems more expert (uses technical jargon)
4. A user who seems less expert (uses colloquial language)
5. A multi-part question that bundles this with an unrelated question

Output as a JSON array with fields: input, notes_on_variation.
Do NOT change the underlying criteria these test.
"""

def expand_example(example: dict, n_variants: int = 5) -> list[dict]:
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": EXPANSION_PROMPT.format(
                original_input=example["input"]["user"],
                criteria="\n".join(example["evaluation_criteria"])
            )
        }]
    )
    # parse and return variants
    import json
    variants = json.loads(response.content[0].text)
    return [{
        **example,
        "id": f"{example['id']}-synth-{i}",
        "input": {**example["input"], "user": v["input"]},
        "added_by": "synthetic",
        "synthetic_notes": v["notes_on_variation"]
    } for i, v in enumerate(variants)]

Notice that every variation rephrases the question without changing what it asks. That constraint is load-bearing: the code above copies the original reference_answer and evaluation_criteria into each variant, so a domain-shifted variant (an HR-policy question inheriting criteria that demand data-retention policy code POL-DPR-2024-03) would be a permanently failing, semantically broken row. If you want coverage in a new domain, write a new example with its own criteria and put it through the same human review as a production sample.

One warning about synthetic expansion: the model generating variants and the model you are evaluating should not be the same. If you expand examples with GPT-4o and evaluate with GPT-4o, you will find that the examples happen to match GPT-4o's output distribution and you will overestimate performance on natural user traffic. Use a different model family for generation.

Sizing your dataset

The question everyone asks: how many examples do I actually need?

The answer depends on the regression size you want to detect and your tolerance for false positives in CI.

Statistical power calculation (one-sided binomial test, α=0.05, power=0.80):

Baseline pass rate: 95%
Minimum detectable regression: 10% → need N ≈ 50 examples
Minimum detectable regression: 5%  → need N ≈ 175 examples
Minimum detectable regression: 2%  → need N ≈ 1,000 examples

Practical implication:
  50 examples  → catches "the system is clearly broken"
  200 examples → catches "something meaningful regressed"
  1,000+ examples → fine-grained quality monitoring; probably overkill for most teams

Start at 50. Run CI for two months. Sample interesting failures from production.
You will naturally grow to 150-200 before you notice you are doing it.

The number matters less than the coverage. A 50-example dataset that covers all the product spec's critical paths is more valuable than 500 examples that all test variations of the same happy path.

Tag your examples by category (policy-citation, multi-turn, adversarial, language-X) so you can run slice-level analysis. "Overall pass rate 93%" is less actionable than "policy-citation pass rate 87%, adversarial pass rate 91%, multi-turn pass rate 95%."

Evaluation criteria: specification-driven vs ad hoc

The weakest version of a golden dataset has a column called expected_output containing the answer you want. This fails quickly: exact string matching is brittle, paraphrases get marked wrong, and the criteria are opaque — you cannot tell from a failure which requirement the output violated.

The stronger version derives evaluation criteria from formal product specifications. If your spec says "the assistant must always cite the source document when making a factual claim," that becomes a testable criterion in every example that involves factual claims. When a criterion fails, you can trace it back to the spec clause, file a regression ticket, and explain to stakeholders exactly what broke and why it matters.

# Specification-driven criteria evaluation with LLM judge
import anthropic

client = anthropic.Anthropic()

JUDGE_PROMPT = """
You are evaluating an AI assistant response against specific criteria.

User input: {input}
Assistant response: {response}
Reference answer: {reference}

Evaluate each criterion below. For each, output:
- verdict: PASS or FAIL
- reasoning: one sentence explaining why (cite specific text from the response)

Criteria:
{criteria}

Respond as JSON: {{"results": [{{"criterion": "...", "verdict": "...", "reasoning": "..."}}]}}
"""

def evaluate_example(example: dict, response: str) -> dict:
    result = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                input=example["input"]["user"],
                response=response,
                reference=example["reference_answer"],
                criteria="\n".join(
                    f"{i+1}. {c}" 
                    for i, c in enumerate(example["evaluation_criteria"])
                )
            )
        }]
    )
    import json
    eval_results = json.loads(result.content[0].text)
    
    failed = [r for r in eval_results["results"] if r["verdict"] == "FAIL"]
    
    # "any" threshold: fail if any criterion fails
    # "majority" threshold: fail if more than half fail
    if example.get("fail_threshold") == "any":
        overall = "FAIL" if failed else "PASS"
    else:
        overall = "FAIL" if len(failed) > len(eval_results["results"]) / 2 else "PASS"
    
    return {
        "example_id": example["id"],
        "overall": overall,
        "criteria_results": eval_results["results"],
        "failed_criteria": [r["criterion"] for r in failed]
    }

See LLM-as-Judge: Calibrating Your Automated Evaluator for how to validate that your judge actually agrees with domain experts before trusting it in CI — the calibration step is not optional.

Wiring into CI

The golden dataset only earns its keep when it runs automatically. The pattern:

flowchart TD
    PR["PR opened\n(prompt / model / retrieval change)"] --> TRIGGER["CI trigger"]
    TRIGGER --> RUN["Eval job\n• load golden dataset\n• run each example against new system\n• score with LLM judge"]
    RUN --> SCORE["Compute aggregate pass rate\nper tag and overall"]
    SCORE --> CHECK{">= threshold?"}
    CHECK -->|"yes (≥92%)"| PASS["Status: ✓ PASS\nPR unblocked"]
    CHECK -->|"no (<92%)"| FAIL["Status: ✗ FAIL\nPR blocked + failure report"]
    FAIL --> REPORT["Report: which examples failed,\nwhich criteria, what the output was"]
    style FAIL fill:#ff2e88,color:#fff
    style PASS fill:#15803d,color:#fff
    style TRIGGER fill:#00e5ff,color:#0a0a0f

A minimal CI eval script:

# eval_ci.py — runs in CI, exits nonzero on regression
import json
import sys
from pathlib import Path
from your_app import run_pipeline  # your LLM application
from eval import evaluate_example   # the judge function above

GOLDEN_DATASET = Path("evals/golden-dataset-v3.jsonl")
PASS_THRESHOLD = 0.92
CRITICAL_TAGS = {"policy-citation", "compliance-critical"}

def run_ci_eval() -> int:
    examples = [json.loads(l) for l in GOLDEN_DATASET.read_text().splitlines()]
    
    results = []
    for ex in examples:
        response = run_pipeline(ex["input"])
        result = evaluate_example(ex, response)
        result["tags"] = ex.get("tags", [])
        results.append(result)
    
    overall_pass = sum(1 for r in results if r["overall"] == "PASS") / len(results)
    
    # Per-slice analysis — a critical slice failing blocks the PR
    # even when the overall rate stays above threshold
    slice_failed = False
    for tag in CRITICAL_TAGS:
        tag_results = [r for r in results if tag in r["tags"]]
        if tag_results:
            tag_pass = sum(1 for r in tag_results if r["overall"] == "PASS") / len(tag_results)
            print(f"  {tag}: {tag_pass:.1%} ({len(tag_results)} examples)")
            if tag_pass < PASS_THRESHOLD:
                print(f"  CRITICAL SLICE FAILURE: {tag}")
                slice_failed = True
    
    print(f"\nOverall pass rate: {overall_pass:.1%} (threshold: {PASS_THRESHOLD:.0%})")
    
    if slice_failed or overall_pass < PASS_THRESHOLD:
        failures = [r for r in results if r["overall"] == "FAIL"]
        print(f"\nFailed examples ({len(failures)}):")
        for f in failures[:5]:  # show first 5 in CI output
            print(f"  [{f['example_id']}] Failed: {f['failed_criteria']}")
        return 1  # exit code 1 blocks the PR
    
    return 0

if __name__ == "__main__":
    sys.exit(run_ci_eval())

Handling run-to-run variance

A single-run hard gate at 92% on 200 examples will flap, and the first week of any CI eval rollout is usually spent discovering why. Both the system under test and the judge are stochastic: sampling temperature, provider-side nondeterminism, and judge parse failures each move the measured pass rate a point or two between runs on an unchanged system. Four mitigations, in order of payoff:

1. Pin temperature to 0 (or the lowest your app allows) for both the system under test and the judge. Runs still will not be bit-identical — batched inference is not deterministic — but it removes most of the variance.
2. Re-run failures 2-3× before counting them. Require a majority verdict per example. This converts "flaky example" into "consistently failing example" at the cost of a few extra calls on the failing tail only.
3. Retry judge parse errors once. A judge returning malformed JSON is a plumbing failure, not a quality regression. Count persistent parse failures separately from FAILs so they surface as infrastructure noise instead of blocking the PR.
4. Gate with a margin. On 200 examples one example is half a point, and the 95% confidence interval around a measured 92% is roughly ±4 points. Either set the threshold 2-3 points below the level you actually care about, or block only when the confidence interval sits entirely below your target.

The run itself is cheap enough not to argue about. At Sonnet-class pricing (illustrative, mid-2026), 200 examples × (one app call at ~1,500 input / 400 output tokens + one judge call at ~1,200 / 300) comes to about $4 per CI run. Wall clock at 20-way parallelism: under two minutes. The expensive part was never the compute — it is the reviewer time that built the dataset.

The eval-pipeline visualizer shows this five-stage safety net interactively:

{ "type": "eval-pipeline", "title": "Where golden dataset CI fits in the eval safety net" }

The versioning and retirement problem

This is where most teams get into trouble. They build a golden dataset, wire it into CI, and then start making exceptions: "This example is failing but we changed that behavior intentionally, so let's just remove it." Six months later the dataset is 30% smaller and covers none of the behaviors that have been "intentionally changed."

The rule: retire examples only when the underlying product spec changes, and document why.

// evals/retired/policy-cite-0038.jsonl
{
  "id": "policy-cite-0038",
  "retired_date": "2026-05-01",
  "retired_reason": "Product spec updated: citations now only required for Level 1 policies, not Level 2. This example tested a Level 2 policy citation requirement that no longer exists.",
  "spec_change_ticket": "PROD-4421",
  ...original example data...
}

Keep retired examples in a separate file with the reason documented. This creates an audit trail: if a future regression brings back the original behavior, you can reactivate the example. It also makes visible when a team is abusing the retirement path to suppress legitimate failures.

Version the dataset itself with a clear versioning strategy:

evals/
  golden-dataset-v1.jsonl   # initial 50 examples at launch
  golden-dataset-v2.jsonl   # +80 production samples after month 2
  golden-dataset-v3.jsonl   # +70 examples after spec update in month 4
  retired/                  # retired examples with documentation
  CHANGELOG.md              # what changed in each version, why

Your CI job should pin to a specific version. When you release a new version, run both the old and new versions in parallel for one sprint to catch any examples the new version incorrectly retired.

The collection flywheel

A golden dataset built once and never updated is a liability disguised as an asset. After six months, your user base has evolved, your product has added features, and the static dataset covers none of the new territory. You are catching regressions in the old product, not the current one.

The flywheel is the mechanism that keeps the dataset current:

flowchart LR
    PROD["Production\ntraffic"] -->|"1% random sample\n+ low-score filter"| CAND["Candidate\nexamples"]
    CAND -->|"PII removal\n+ dedup"| REVIEW["Human review\n(30 min/week)"]
    REVIEW -->|"approve + label criteria"| GD["Golden Dataset"]
    GD -->|"CI eval on every PR"| CI["CI gate"]
    CI -->|"failures surfaced\nfor review"| REVIEW

Thirty minutes per week with a human reviewer converting production failures into dataset examples keeps coverage current without requiring a dedicated team. The CI failures are the most valuable source: when a real prompt change causes a regression in production-sampled examples, the failure report tells you exactly what broke and why — feed that directly back into the dataset.

Braintrust, Langfuse, and Arize Phoenix all automate parts of this flywheel: they capture production traces, allow inline scoring, and can push scored examples directly into a dataset via API. The observability tooling comparison in Tracing and Observability Tooling covers the specifics of each platform's dataset management capabilities.

What breaks

Goodhart's Law corruption. When the team treats golden dataset pass rate as the goal rather than a proxy, prompts get tuned to pass the eval. The tell: eval score climbs every sprint, but user satisfaction metrics stay flat or degrade. The fix: maintain a held-out set (10-15% of examples) that is never shown to anyone writing prompts. Run it monthly. If the held-out score diverges from the main dataset score, you have optimized the eval instead of the product.

Homogeneous sourcing. A dataset built entirely from hand-crafted examples misses real production query patterns. A dataset built entirely from production samples misses intentional edge cases. A dataset built entirely from synthetic generation misses both. Teams that skip one of the three sources routinely discover blind spots the hard way.

Stale coverage. New feature ships in March. Golden dataset last updated in January. The new feature has no coverage, regresses in April, CI passes because there are no examples for it. The flywheel is the only prevention.

Inconsistent human labeling. Human annotators agree on LLM output quality at roughly 81% inter-annotator agreement in general; for domain-specific applications the rate is higher when annotators have domain expertise and lower when they do not. If you have multiple reviewers labeling examples, measure inter-annotator agreement and calibrate before adding their labels. Inconsistently labeled examples add noise to your CI signal — failing examples do not actually represent regressions; passing examples mask real problems.

Too-coarse thresholds. A single aggregate pass rate hides per-slice regressions. If your policy-citation slice degrades from 95% to 78% but overall pass rate stays above 90% because the easy cases still pass, your CI gate will not catch it. Always run per-tag slice analysis alongside the aggregate.

Judge bias from family matching. If you evaluate Claude outputs using Claude as judge, self-preference bias inflates scores. Zheng et al. (2023) measured Claude-v1 favoring its own outputs by roughly 25% in pairwise win rate — the exact number will not transfer to current models, but the mechanism does, so treat same-family judging as suspect. Use a different model family for the judge than for the system under evaluation. See the calibration methodology in LLM-as-Judge: Calibrating Your Automated Evaluator.

The decision in practice

The question is not whether to build a golden dataset — the question is where to start and how small you can go before you have something useful.

Week 1: 50 hand-crafted examples from your product spec's critical paths. Wire CI. Accept that coverage is incomplete; you now have a working pipeline and a floor. Any regression in your critical paths is caught. Anything else still slips through.

Month 1-2: Collect production traffic. Sample 10 traces per week from your observability platform. Review for 30 minutes. Strip PII. Add 40-80 examples. Coverage gap starts closing.

Month 3+: Synthetic expansion for underrepresented scenarios. Slice analysis by tag. Tune your CI threshold based on empirical false-positive rate (a threshold that blocks too many good PRs loses team buy-in). Retire only with documented spec changes.

The teams that skip this and treat eval as something to add after the product is stable learn, eventually, that there is no stable to wait for. The product is always changing. The model provider silently updated last month. The user base added a new cohort in Q3 with query patterns you never anticipated. The eval infrastructure is what tells you when any of that matters.

The compliance assistant story from the opening had a three-week regression with no CI gate. With 50 examples covering citation requirements and a CI job that runs on every prompt change, the regression would have been caught the day the "more natural" system prompt was proposed. The PR would have failed. Someone would have written a better prompt. That Tuesday email would not have happened.

Your eval is only as good as the ground truth it runs against. Build the ground truth first.

---

Related: Why Eval Is Hard: The Measurement Problem in LLM Applications — the case for why you need this before you can trust any metric. Offline vs Online Evaluation — how golden datasets fit into the two-layer safety net alongside production sampling. Prompt Versioning and CI/CD for LLM Changes — the workflow for treating prompt changes like code changes, with eval gates built in.