A/B Testing Prompts and Models in Production

Your new prompt variant looks great on the eval set. You ship it to production on a Friday afternoon. By Monday morning, your support queue has doubled. The prompt phrasing that felt more natural also turned out to be dramatically worse at following complex multi-part instructions, which are rare in your golden dataset but common in production queries from power users. Nothing crashed. HTTP 200 throughout. The system had no idea anything was wrong.

This is the default failure mode of LLM changes shipped without experimentation infrastructure. Prompt quality doesn't raise exceptions. Model swaps don't trip circuit breakers. The regression is silent until it isn't.

Why LLM experiments are different from feature flag A/B tests

Traditional A/B testing works when you have a clear binary or count metric — clicks, conversions, signups — and outputs have low variance. You split traffic 50/50, wait for significance, ship the winner. The entire apparatus built up over the last decade assumes this shape.

LLM experiments break three of those assumptions simultaneously.

Quality is multi-dimensional. There is no single "did it work" boolean. Relevance, faithfulness, tone, format compliance, safety, instruction-following accuracy — each is a separate signal, they can move in opposite directions, and you have to decide upfront which ones you weight. A prompt that improves helpfulness scores by 8% while degrading safety classifier pass rate by 2% is not obviously a win.

Variance is much higher. A conversion rate of 3% has tight binomial variance at scale. LLM output quality on a 1-5 rubric has standard deviation around 0.7-1.0 — high enough that you need hundreds of samples to detect a 0.3-point improvement. Teams routinely underestimate this and declare winners on 50-100 samples, which produces false positives at rates far above 5%.

The scorer is also probabilistic. You cannot afford human evaluators at production query volumes. So you use LLM-as-judge, which adds its own noise on top of the already-noisy output. A GPT-4-class judge agrees with expert human raters about 80% of the time on well-calibrated rubrics, but the remaining 20% is correlated noise — the judge has systematic biases that compress your effective sample size. You need to account for this in your power calculations.

The three-phase experimentation pipeline

Phase 1: Shadow testing

Shadow mode runs the treatment variant on every request alongside the control, but only the control response reaches the user. The treatment response is evaluated silently and discarded.

sequenceDiagram
    participant U as User
    participant R as Router
    participant C as Control (v1)
    participant T as Treatment (v2)
    participant J as Async Judge

    U->>R: query
    R->>C: forward request
    R->>T: forward request (shadow)
    C-->>R: control response
    T-->>R: treatment response
    R-->>U: control response (user sees this)
    R->>J: [control response, treatment response, query]
    J-->>J: score both
    J-->>J: log to experiment store

The economics are blunt: shadow testing doubles your LLM call volume and therefore your LLM cost for the duration of the test. This is the right trade when the downside of a bad rollout is severe — a model swap, a major prompt rewrite, a change to RAG retrieval strategy. For a minor rephrasing tweak on a low-stakes use case, shadow may be overkill.

What shadow gives you is real-distribution data with zero user exposure. Your offline eval set was sampled from historical queries. Production traffic in the next two weeks will have queries you didn't anticipate. Shadow testing on live traffic finds the edge cases your golden dataset missed, before real users see the treatment.

Two caveats before you wire this up. If the flow calls tools — writes to a database, sends an email, hits a metered external API — a naive shadow lane executes every side effect twice. Stub or sandbox side-effecting tools in the shadow path, and shadow only the generation and read-only retrieval steps; a duplicate confirmation email to a customer is a pager story you don't want to explain. And shadow validates single-turn quality only: treatment responses never reach users, so every follow-up query in a conversation was shaped by the control's output. Multi-turn dynamics — whether the treatment's phrasing invites more clarification round-trips, say — stay invisible until the canary.

Run shadow long enough to collect at least 200-300 treatment samples before you look at results. Resist the urge to check daily. Set a calendar reminder for when you will look.

Phase 2: Canary rollout

If shadow passes your quality thresholds, promote the treatment to a real canary: route 5-10% of live traffic to the treatment variant, with both responses being real user-facing outputs.

Why 5-10% rather than the 50% common in UI A/B tests? Two reasons. First, LLM quality regressions can be subtle and accumulate — a prompt that's 8% worse on faithfulness will slowly build negative reputation before it shows up in business metrics. You want to contain exposure. Second, at 5-10% canary, collecting 500 treatment samples requires 5,000-10,000 total requests. At a product with 2,000 daily active requests, that's 2.5-5 days — an acceptable experiment window.

The canary must run with automated scoring from day one. There's no point routing real traffic to the treatment if you're not measuring it. The minimum viable scorer is:

• Format compliance rate: does the output match the required structure (JSON schema, required sections, character limits)?
• LLM-as-judge quality score: a 1-5 rubric covering the dimensions you care about most.
• Safety classifier pass rate: does the output pass your content safety filters?
• Cost per request: average input + output tokens × price, tracked per variant.
• P95 latency: some prompts make the model work harder, increasing latency on long-tail requests.

{ "type": "eval-pipeline", "title": "Eval stages in a prompt canary rollout" }

Phase 3: Full rollout or rollback

You have a decision to make when your pre-registered criteria are met: minimum sample size reached, statistical test run, metrics within acceptable ranges.

Three outcomes are possible. The treatment wins — quality up, no cost blowout, no latency regression — and you promote to 100% traffic and deprecate the control. The treatment loses — quality down or cost up beyond threshold — and you kill the canary and roll back. The result is inconclusive — no statistically significant difference — which usually means you should either run longer or accept that the change doesn't matter and revert to the simpler variant.

Inconclusive is actually fine. An experiment that found no improvement saved you from shipping a change that might have slightly degraded quality in ways you didn't detect. Treat no-result as a success of the experimental apparatus, not a failure of the prompt engineer.

Statistical rigor: the parts most teams skip

Pre-register your stopping rule

The most common statistical mistake in LLM experimentation is peeking at results and stopping when p < 0.05 first appears. This is the same problem that plagued clinical trials before pre-registration became standard. If you check results daily and stop at first significance, your actual false-positive rate is not 5% — it's roughly 14% after five peeks, and it climbs past 30% if you monitor continuously.

The fix is to set your stopping criteria before the first treatment request is served. Write them down:

• Minimum samples per variant: N
• Primary metric and significance threshold: p < 0.05, two-tailed
• Secondary metric guardrails: cost cannot increase > 15%, latency P95 cannot increase > 200ms
• Hard stop date: if N not reached by [date], declare inconclusive

If you genuinely need early stopping capability — to kill a regression fast — use sequential testing (SPRT or alpha-spending methods) rather than naive peeking. These are designed for the interim analysis problem.

Sample size calculation

Model: LLM-as-judge quality score, 5-point Likert scale
Typical σ (output variance): 0.8 – 1.1
Effect you want to detect: δ = 0.2 points (meaningful but modest)
Power: 80%
α: 0.05 (two-tailed)

Standard sample size formula for two-sample t-test:
n = 2 × σ² × (z_α/2 + z_β)² / δ²
  = 2 × 0.81 × (1.96 + 0.84)² / 0.04
  = 2 × 0.81 × 7.84 / 0.04
  ≈ 317 per variant (ignoring judge noise)

Judge noise correction: judges disagree with each other ~15-20% of the time,
effectively inflating σ to ~1.0-1.2. Multiply by 1.5-2×:
→ Practical target: 500 – 700 per variant for δ = 0.2

Detecting δ = 0.1 (subtle improvement): ~2,000 – 3,000 per variant
Detecting δ = 0.5 (obvious improvement): ~50 – 100 per variant

This math has a practical consequence: at low traffic volumes, you may not be able to run statistically valid experiments at all. 200 daily requests × 10% canary = 20 treatment samples per day → 25-35 days to significance for a modest effect. Either accept longer experiment windows, increase canary percentage, or accept that you're operating below the statistical detection threshold and rely more on shadow testing and offline eval.

Multi-metric testing and the Bonferroni correction

You are almost certainly tracking multiple metrics simultaneously — quality score, cost, latency, safety pass rate, format compliance. Running five independent hypothesis tests at α=0.05 each means your per-experiment false positive rate is approximately 1 - (0.95)^5 ≈ 23%. Apply Bonferroni correction (divide α by the number of tests) or use Holm-Bonferroni stepdown. In practice, designate one primary metric and treat the rest as guardrails with softer thresholds, which sidesteps the correction overhead while preserving rigour where it matters.

Building the LLM-as-judge scorer

LLM-as-judge is the practical standard for automated quality scoring at production volumes. A GPT-4o or Claude Sonnet 4 call costs $0.005-0.015 per evaluation versus $20-100 per human annotation, and achieves ~80% agreement with expert raters on well-specified rubrics. The key word is "well-specified" — a vague rubric yields a vague judge.

A minimal evaluation prompt looks like this:

import json

from openai import AsyncOpenAI

client = AsyncOpenAI()

JUDGE_SYSTEM_PROMPT = """You are an expert evaluator for an AI assistant that
answers questions about software documentation.

Score the following response on a scale of 1-5 for each dimension:
- relevance: Does the response directly address the user's question? (1=off-topic, 5=directly answers)
- faithfulness: Are all factual claims consistent with the provided context? (1=contradicts context, 5=fully grounded)
- completeness: Does the response cover all parts of the question? (1=major gaps, 5=fully complete)
- conciseness: Is the length appropriate? (1=padded/bloated, 5=right length)

Output ONLY a JSON object with keys: relevance, faithfulness, completeness, conciseness (each 1-5).
No commentary outside the JSON."""

async def judge_response(query: str, context: str, response: str) -> dict:
    """Score one response in isolation. Call once per variant —
    the judge never sees control and treatment in the same prompt."""
    user_prompt = f"""Query: {query}

Context provided to the model:
{context}

Response:
{response}"""

    result = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(result.choices[0].message.content)

Positional bias is the judge's tendency to favor whichever response it evaluates first (when shown two). Cancel it by evaluating each response independently, not as a pair. The code above does this — one call per response, not one call with "Response A vs Response B."

Verbosity bias is the judge's tendency to rate longer responses higher, independent of quality. Counter it with an explicit conciseness rubric that penalizes unnecessary length, as above.

Self-preference bias is the tendency of a model to prefer its own outputs. If your treatment model is the same as your judge model, add a secondary human spot-check on 2-3% of cases, or use a judge model from a different provider.

The model-swap experiment: special considerations

Swapping the underlying model — from GPT-4o to Claude Sonnet 4, or from a frontier to a cheaper tier — is qualitatively different from a prompt change. The response distribution shifts more dramatically, the quality dimensions that differ are harder to anticipate, and the cost and latency change alongside quality.

The model routing article covers query-by-query routing in depth. For a wholesale model swap, the key additions are:

flowchart TD
    SHADOW["Shadow test\n(both models, all traffic)"] --> QUAL{Quality parity\n≥ 95% of control?}
    QUAL -->|"Yes"| CANARY["Canary 5%\ntrack cost + latency delta"]
    QUAL -->|"No"| TUNE["Prompt-tune\nfor new model"]
    TUNE --> SHADOW
    CANARY --> COST{Cost acceptable?\nLatency within SLA?}
    COST -->|"Yes"| ROLLOUT[Full rollout]
    COST -->|"No"| TRADEOFF{Quality delta\njustifies cost?}
    TRADEOFF -->|"Yes, explicit sign-off"| ROLLOUT
    TRADEOFF -->|"No"| REVERT[Keep control model]
    style SHADOW fill:#a855f7,color:#fff
    style ROLLOUT fill:#15803d,color:#fff
    style REVERT fill:#dc2626,color:#fff
    style TUNE fill:#00e5ff,color:#0a0a0f

The "prompt-tune for new model" branch is real and common. Models have different instruction-following conventions, different tendencies around verbosity, different defaults for how to handle ambiguous instructions. A prompt written for GPT-4o may perform noticeably worse on Claude Sonnet without minor adaptation. Build this into your experiment plan — a model swap experiment is typically three experiments: control prompt on control model vs control prompt on treatment model vs treatment-optimized prompt on treatment model.

There's a statistical wrinkle specific to cost-driven downgrades. When you swap to a cheaper model, the question is not "is the new model better?" but "is it not worse than a margin we can live with?" — a non-inferiority test, not the superiority test everything above assumes. That "quality parity ≥ 95%" gate in the diagram is only meaningful if you make it concrete: pre-register a parity margin (say, judge score within 0.1 points of control), then run a one-sided test that the treatment's deficit is smaller than the margin. The sample-size math bites here — a 0.1-point margin puts you in the same 2,000-3,000-samples-per-variant regime as detecting a 0.1-point improvement, so a tight margin can make the "cheap" swap experiment the most expensive one you run. Pick the margin from the business side (how much quality loss the cost savings buys) before you see any data, not after.

The cost half of the tradeoff is easier to pin down. Plug your own request volume and token mix into the model below — the monthly bill delta between tiers is the number the swap has to justify.

{ "type": "token-cost", "title": "Cost delta of a model-tier swap" }

What breaks

Novelty effect

Users respond differently to changes that are new. A response style that feels fresher in week one may not sustain that edge in week four. For aesthetic changes (tone, length, formatting), run the experiment for at least two weeks before declaring a winner. For factual quality improvements (faithfulness, accuracy), novelty is less of a concern.

Query distribution shift

Your canary runs against tomorrow's query distribution, which may be different from the queries in your golden dataset or even from last week's traffic. A prompt that excels on the current query mix may perform poorly on the query patterns that arrive next month after a product launch changes who your users are and what they ask. Treating your eval set as a fixed artifact is the slow poison of LLM quality management — see the LLM observability article for how to monitor distribution drift.

Judge calibration drift

LLM judges are not stable evaluators over time. Provider model updates change the judge's behavior without you changing the judge prompt. A judge that agreed 82% with human raters in January may agree only 74% in July after a provider model update. Spot-check your judge's calibration monthly against a held-out human-labeled set. If agreement drops significantly, recalibrate.

Silent cost regressions

A treatment prompt that uses richer chain-of-thought reasoning may improve quality but add 300 output tokens per request. At 100k req/day on gpt-4o ($0.015/1k output tokens), that's $450/day of additional cost — $164k/year. Track cost per request as a first-class metric in every experiment. Set a hard threshold: if treatment cost exceeds control cost by more than X%, require explicit budget sign-off regardless of quality improvement.

Worked example — cost regression check:
  Control: avg 850 input + 220 output tokens, gpt-4o pricing (illustrative)
  Treatment: avg 850 input + 520 output tokens (CoT reasoning added)
  
  Δ output tokens = 300
  At $0.015/1k tokens output: Δ cost = $0.0045/request
  At 100k req/day: Δ daily cost = $450
  Monthly: $13,500
  
  Quality improvement: 0.3 points on 5-point scale (6%)
  
  Is 6% quality improvement worth $162k/year? 
  That's a business decision, not a technical one. 
  The experiment must surface this number, not bury it.

Imbalanced variant assignment

Feature flag systems assign users to variants, not requests. If users are sticky to their variant (same user always gets control or always gets treatment), you need to model this in your significance test — observations within a user are correlated, violating the independence assumption of a standard t-test. Use cluster-robust standard errors or a mixed-effects model, or accept that your effective sample size is users, not requests.

Tooling in 2025-2026

You can build this infrastructure yourself — a database table for experiment assignments, an async queue for evaluations, a dashboard — or use one of the platforms that have converged on this problem:

Tool	Strength	Watch out for
Braintrust	Native LLM eval + experiment tracking; strong statistical reporting	Adds a network hop for scoring; requires data to leave your infra
LangSmith	Deep integration with LangChain ecosystem; replay and tracing	Best features require LangChain; less useful if you're not in that stack
Statsig	Best-in-class statistical engine; feature flags + experiments unified	LLM scoring is newer and less mature than core A/B testing
Arize AI	Strong observability + experiment comparison; hallucination detection	Pricing at scale; evaluation is a feature, not the core product
Traceloop / OpenLLMetry	Open-source OTEL-based; no vendor lock-in	You assemble the pieces; no out-of-box statistical significance tooling
Roll your own	Full control; no data egress	Takes 2-4 weeks to build minimally viable; statistical correctness is on you

The prompt versioning and CI/CD article covers how experiments integrate with the deployment pipeline. The short answer: experiment infrastructure lives downstream of the CI evaluation gate. CI runs your offline eval suite; A/B testing validates on real traffic what CI approved for deployment.

Running the experiment: an operational checklist

Before you start canary traffic:

• Prompt variants are committed to version control with semantic IDs (not "new prompt v2 final FINAL")
• Experiment definition is written: hypothesis, primary metric, guardrail metrics, minimum N, stopping rule, end date
• Scorer is deployed and tested on a sample of historical data; judge calibration is verified
• Cost monitoring alert is set at 2× control cost
• Rollback plan is documented: who triggers it, how, target < 15 minutes to revert

During the canary:

• Check score distribution on day 1 to confirm scorer is working (not all 3s)
• Monitor for distributional shift in query types between control and treatment buckets
• Do not look at statistical significance until minimum N is reached
• Trigger rollback immediately if cost > 2× control OR safety pass rate drops > 2%

At experiment close:

• Run pre-specified significance test — do not switch tests after seeing the data
• Record full results including negative results; write a one-page experiment report
• If treatment wins, file the control for deprecation; if treatment loses, file why for institutional memory

The judgment call: when to run experiments and when not to

Not every LLM change needs a full A/B test. Shadow testing is cheap; a full canary experiment with 1,000 samples can take 3 weeks at low traffic. Match the rigor to the risk:

Run a full canary experiment when:

• Swapping the underlying model or model version
• Major prompt rewrite changing instructions or persona
• Changes to RAG retrieval strategy that affect what context the model sees
• Any change to a high-traffic, high-stakes flow (checkout, medical triage, financial advice)

Shadow test only when:

• Prompt change is moderate in scope and offline eval already shows clear improvement
• You're validating a regression fix, not exploring a new direction
• Traffic volume is too low for statistical significance within an acceptable window

Deploy directly (with eval-gated CI only) when:

• Bug fix to prompt (incorrect format instruction, typo in few-shot example)
• Change is confined to a feature with its own isolated eval coverage
• Rollback is instant and the change is easily reversible

The eval pipeline article covers the offline layer that gates changes before they reach experiment infrastructure. The system only works when both layers are in place: offline eval as the quality gate, A/B testing as the production validator. Running one without the other leaves a gap. Offline eval without production validation finds regressions before users do, but misses distribution shift. Production testing without offline eval wastes experiment budget on changes that would have failed a basic regression check.

The experiment infrastructure is not the bureaucratic overhead that slows down shipping. It's the thing that makes you confident you can ship fast — because when you have the measurement apparatus in place, you can move quickly and know immediately when something has gone wrong.