Synthetic Data Engineering: Pipelines, Quality, and Model Collapse Risks

The team had a fine-tuning budget of $50,000. Human annotation at $2 per sample would buy them 25,000 training pairs. Their domain expert estimated they needed at least 80,000 samples to cover the edge cases their support chatbot was hitting. The math didn't work.

So they generated 200,000 samples with GPT-4o, skipped the filtering step to save time, fine-tuned Llama-3-8B on all of them, and shipped a model that scored 78% on their internal eval set — a 15-point improvement over the base. Three weeks later, the support team noticed the model was confidently giving wrong answers about a policy change from six months ago. The model had absorbed the teacher's errors at scale. The 78% eval score was real; it just measured the wrong thing, on contaminated data, evaluated against a distribution the model had memorized.

This is where synthetic data engineering gets complicated.

Why synthetic data is now the default starting point

The economics settled in 2024. A frontier LLM generates one instruction-response pair for roughly $0.01. A human annotator with domain expertise costs $1–5 per pair, with longer review cycles and variance in quality. For an 80,000-sample dataset, that's $800 versus $80,000–$400,000. Even accounting for the filtering step that removes 40–70% of raw generated samples, synthetic data comes in at $0.025–$0.035 per usable sample.

But cost alone doesn't explain why SFT practitioners have converged on synthetic pipelines. The deeper reason is that frontier models generate diverse, well-formatted, coherent instruction-response pairs that are often higher quality than what crowdsource annotators produce. The AlpaGasus study demonstrated this in the starkest terms: filtering the 52,000-sample Alpaca dataset down to 9,000 high-quality samples — identified by asking GPT-4 to score each example — produced better downstream benchmark performance than training on all 52,000 raw samples. More data made the model worse because the bad data was teaching the model bad behaviors at scale.

The lesson: volume is not the goal. Filtered, diverse, uncontaminated data is the goal.

Frontier labs maintain human preference data as a competitive moat — the preference data behind RLHF, DPO, and SimPO alignment steps still benefits substantially from human judgment on subtle value questions. But for instruction-following, format consistency, and domain-specific knowledge, synthetic generation with quality filtering is now the standard engineering approach.

The two generation architectures

Mutation-based: Evol-Instruct

Evol-Instruct starts with a seed dataset — as small as a few hundred manually curated examples — and iteratively mutates each instruction using a set of rewrite operations. Two axes of mutation matter:

In-depth evolution rewrites a prompt to be harder for the model: add explicit reasoning steps, impose new constraints ("answer in under 100 words"), combine multiple concepts, increase specificity. The resulting prompts require more capability to answer correctly.

In-breadth evolution generates new instructions in adjacent domains, using the seed example as a starting-point rather than a template to copy. This is how you get from "explain database normalization" to "explain normalization in the context of a time-series schema with high write throughput."

WizardLM and WizardCoder both used Evol-Instruct to scale their training data, and the technique works: iterative mutation produces harder, more diverse examples than a static seed set. But it is bounded by the seed's distribution. If your seed covers five domains, Evol-Instruct will produce diverse coverage across those five domains. It will not spontaneously invent the sixth.

Seed-free: Magpie

Magpie (ICLR 2025) exploits a structural property of instruction-tuned models. When you feed an aligned chat model only the system prompt and the beginning of the user-turn in the chat template — just the formatting tokens that indicate "user is about to say something" — the model will autoregressively generate what a user would say. Then it generates the response. Both halves come from a single generation call with no seed data.

This only works if you control the chat template. A closed API like GPT-4o or Claude applies the template server-side: send a system message with no user turn and you get an ordinary assistant reply to an empty conversation, not a model-generated user query. Magpie teachers must be open-weight models served through a raw completions endpoint — the paper used Llama-3-Instruct for exactly this reason.

# Magpie-style generation (Llama-3 template; needs raw completion access)
# Serve the model yourself, e.g.:
#   vllm serve meta-llama/Meta-Llama-3-8B-Instruct
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# The raw prompt ends exactly where user content would begin —
# a dangling user header with nothing after it
prefix = (
    "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n"
    "You are a helpful assistant.<|eot_id|>"
    "<|start_header_id|>user<|end_header_id|>\n\n"
)

completion = client.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    prompt=prefix,
    max_tokens=1024,
)

# The model fills in the user query, closes the turn, then writes
# the assistant response — parse the turn boundary to split them
raw = completion.choices[0].text
query, _, rest = raw.partition("<|eot_id|>")
response = rest.split("<|end_header_id|>")[-1].strip()

Because the model is completing raw template tokens, the topic distribution is driven by what the model "wants to talk about" — which, for a well-aligned model, is broad and diverse.

The ICLR 2025 Magpie paper showed 300K filtered samples from Llama-3-70B-Instruct outperformed ShareGPT, Evol-Instruct, and UltraChat on Llama-3-8B instruction-following benchmarks. The breadth comes from the fact that no seed dataset introduces a distribution bottleneck.

flowchart LR
    subgraph "Evol-Instruct"
        SEED["Seed examples\n(200-500)"] -->|"mutate in-depth\nmutate in-breadth"| MUT["Mutated prompts\n(5K-50K)"] --> GEN1["Teacher response\ngeneration"]
    end
    subgraph "Magpie"
        PREFIX["Chat template prefix\n(no user content)"] --> GEN2["Model generates\nuser query + response\nfrom one call"]
    end
    GEN1 --> FILT["Quality\nfiltering"]
    GEN2 --> FILT
    style SEED fill:#15803d,color:#fff
    style PREFIX fill:#15803d,color:#fff
    style GEN1 fill:#a855f7,color:#fff
    style GEN2 fill:#a855f7,color:#fff
    style FILT fill:#00e5ff,color:#0a0a0f

Which to use? If you have a domain-specific seed set and want to scale it without losing the domain signal, Evol-Instruct is correct. If you want broad instruction-following coverage with no seed data, Magpie produces more diverse samples. For most production pipelines the answer is both: Magpie for the general backbone, Evol-Instruct for domain-specific augmentation.

One constraint that shapes teacher choice as much as quality: output licensing. OpenAI's terms of service prohibit using API outputs to develop models that compete with OpenAI, and Anthropic's terms carry a similar restriction — so "generate with GPT-4o, fine-tune Llama-3-8B" is a conversation to have with legal before you ship, not after. Open-weight teachers sidestep the API terms but bring their own conditions: the Llama license permits training on Llama outputs but requires derivative models to carry Llama naming and attribution, while Apache-2.0 models (most Qwen and Mistral releases, as of mid-2026) impose essentially none. This, not benchmark scores, is a large part of why production teams reach for open-weight teachers. If the fine-tuned model will be sold or exposed to customers, get the teacher's terms reviewed before generating 250K samples you may not be allowed to train on.

Quality filtering: the step most teams skip

Raw generation produces a lot of garbage. The teacher model repeats itself. It generates grammatically fine but semantically empty responses. It produces near-duplicate samples that teach the model the same thing fifty times and reduce effective dataset diversity. It occasionally hallucinates wildly.

Four filtering signals catch most failure modes:

Reward model scoring. Pass each (prompt, response) pair through a reward model trained on human preferences — Llama-3 70B trained on preference data, ArmoRM, or a similar open-weight reward model. Drop samples below a quality threshold. This catches responses that are correct but unhelpfully phrased, incomplete, or misaligned with what users actually prefer. Reward models have their own biases (verbosity bias is well-documented), so use this as one signal among several.

LLM-judge rubrics. Ask a second frontier model to score the response on dimensions you care about: factual accuracy, instruction-following, format compliance, concision. Prompt-based judges catch dimension-specific failures that a reward model's single scalar misses. This costs more per sample but catches systematic issues.

Perplexity filtering. High perplexity under a reference model indicates unusual or incoherent text. Very low perplexity indicates a near-memorized response — likely a contamination signal. Both extremes deserve filtering. Run your sample through a small reference LLM and drop samples in the top 5% and bottom 5% of perplexity.

Deduplication. MinHash on character n-grams (n=5 or 6) identifies near-duplicate samples across the whole dataset. Duplicates don't add training signal; they bias the model toward overconfidence on the duplicated distribution. A dataset with 1,000 variations of "explain gradient descent" teaches the model far more about gradient descent than about anything else. Deduplicate aggressively.

# Quality filtering sketch: reward model + dedup
from datasketch import MinHash, MinHashLSH
import numpy as np

def compute_minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for char_ngram in [text[i:i+5] for i in range(len(text) - 4)]:
        m.update(char_ngram.encode("utf8"))
    return m

def deduplicate_samples(
    samples: list[dict],
    threshold: float = 0.85,
) -> list[dict]:
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    kept = []
    for i, sample in enumerate(samples):
        text = sample["instruction"] + " " + sample["response"]
        m = compute_minhash(text)
        key = f"sample_{i}"
        if not lsh.query(m):  # no near-duplicate found
            lsh.insert(key, m)
            kept.append(sample)
    return kept

# After dedup, apply reward model scoring
# Drop samples below reward_score < 0.6 (threshold is dataset-dependent)
def filter_by_reward(
    samples: list[dict],
    reward_model,      # your reward model inference function
    threshold: float = 0.6,
) -> list[dict]:
    scored = [(s, reward_model(s["instruction"], s["response"])) for s in samples]
    return [s for s, score in scored if score >= threshold]

Together these filters typically remove 40–70% of raw generation. That removal is not waste — it is the engineering. The cost of generating 250K samples to get 100K filtered usable ones is still $2,500 in API fees, versus $150,000 for human annotation of comparable quality.

Model collapse: the real failure mode and how to avoid it

Model collapse is the name for what happens when you train a model on synthetic data, use that model to generate the next round of synthetic data, train again, and repeat — without injecting real data into the cycle. Each generation cycle amplifies the teacher model's biases and prunes low-frequency behaviors. After three to five cycles, the model's output distribution has measurably narrowed: it generates less variety, makes more confident errors on edge cases, and performs worse on tasks that require behaviors that happened to be underrepresented in the first synthetic generation.

The 2023 theoretical analysis (Shumailov et al.) demonstrated that even a small amount of model-generated data mixed into training causes detectable distribution shift, and that iterative mixing causes exponential tail collapse — rare but important behaviors disappear first.

The mechanisms are straightforward. A teacher model has its own systematic blind spots. Those blind spots appear in every sample it generates. Train the student on that data, and the student learns the blind spots as correct behavior. Use the student as the teacher for generation round two, and the blind spots get amplified. By round three, the behaviors are gone.

flowchart TD
    R1["Generation 1:\nFull distribution\nreal + synthetic"] -->|"train model M1"| M1["Model M1\nDistribution slightly narrowed"]
    M1 -->|"generate data without real mixing"| R2["Generation 2:\nSynthetic only\nM1's distribution"]
    R2 -->|"train model M2"| M2["Model M2\nDistribution further narrowed"]
    M2 -->|"generate"| R3["Generation 3:\nSynthetic only\nM2's compressed distribution"]
    R3 -->|"train model M3"| M3["Model M3\nCollapsed distribution\nfails on tail queries"]
    style M3 fill:#ef4444,color:#111
    style R3 fill:#ef4444,color:#111

Prevention requires three things simultaneously, not any one of them:

1. Real-data mixing at every training step. The 2025 consensus is 20–30% real human data in every fine-tuning run. This ratio maintains enough distribution coverage to prevent collapse while still allowing the cost savings of synthetic generation. Dropping the ratio below 10% over multiple iterations is where collapse risk becomes real.

2. Multiple teacher diversity. If GPT-4o generated your round-one data and you use your fine-tuned model to generate round-two data, you have one distribution compounding. Rotate teachers: use Claude for round two, Llama-3-70B for domain-specific samples, and Gemini 1.5 Pro for multilingual coverage. Different model families have different blind spots, and mixing them prevents any single blind spot from compounding.

3. Deduplication between generations. The same example appearing in generation one and generation three is not harmless — it is the model overweighting that example three times. Cross-generation deduplication prevents this.

A single training run on synthetic data — no iterative loop, real-data mixing at 25%, proper deduplication — does not cause model collapse. The risk is specific to the iterative self-consumption pattern. If your pipeline generates synthetic data once and fine-tunes once, the collapse concern is a red herring; focus your energy on contamination and quality filtering instead.

Benchmark contamination: the silent score inflator

This failure mode is subtler than collapse and far more common in practice.

GPT-4, GPT-4o, and Claude were trained on text that includes or substantially overlaps with MMLU, GSM8K, HumanEval, TriviaQA, ARC-Challenge, and most other standard benchmarks. When you use these models as teachers to generate your training data, they sometimes generate questions that are paraphrases or near-duplicates of benchmark questions — because those benchmarks were in their training data. Your student model trains on those paraphrases, effectively memorizing the answers to your evaluation set before it takes the exam.

The result is inflated benchmark scores that don't generalize. The model scores 82% on GSM8K and 61% on genuinely novel math problems. You shipped a "better" model that is actually worse on user queries.

Prevention requires cross-checking every synthetic prompt against the benchmark sets you use for evaluation:

def get_ngrams(text: str, n: int = 13) -> set[tuple[str, ...]]:
    """Extract word n-grams for contamination checking.

    Word-level, not character-level: 13 characters is barely two
    words, and char-level shingles (the right tool for MinHash
    dedup) would flag generic English phrasing as contamination.
    """
    words = text.lower().split()
    return {tuple(words[i:i+n]) for i in range(len(words) - n + 1)}

def contamination_ratio(
    synthetic_prompt: str,
    benchmark_prompts: list[str],
    n: int = 13,
) -> float:
    """
    Return fraction of synthetic prompt's n-grams that appear
    in any benchmark prompt. > 0.30 is typically flagged.
    """
    synthetic_ngrams = get_ngrams(synthetic_prompt, n)
    if not synthetic_ngrams:
        return 0.0

    benchmark_ngrams: set[tuple[str, ...]] = set()
    for bp in benchmark_prompts:
        benchmark_ngrams.update(get_ngrams(bp, n))

    overlap = synthetic_ngrams & benchmark_ngrams
    return len(overlap) / len(synthetic_ngrams)

def filter_contaminated(
    samples: list[dict],
    benchmark_prompts: list[str],
    threshold: float = 0.30,
) -> tuple[list[dict], int]:
    """Return (clean samples, count dropped)."""
    clean = []
    dropped = 0
    for sample in samples:
        ratio = contamination_ratio(sample["instruction"], benchmark_prompts)
        if ratio < threshold:
            clean.append(sample)
        else:
            dropped += 1
    return clean, dropped

The 13-gram convention comes from the GPT-3 paper (Brown et al. 2020, Appendix C), and subsequent contamination work inherited it. The Magpie paper published explicit contamination rates showing their pipeline achieves substantially lower overlap than Evol-Instruct against GSM8K and MMLU — one reason to prefer seed-free generation when contamination control matters.

Apply contamination checks to every benchmark you will use for evaluation before training, not after. After training it is too late.

Reasoning traces and the learnability gap

A specific failure mode appears in reasoning distillation — covered in depth in the reasoning distillation article, but worth flagging here because it surfaces at the data pipeline level.

DeepSeek-R1 trained on 671B parameters with GRPO and verifiable rewards, producing structured chain-of-thought traces that are long, multi-step, and require tracking complex intermediate state. When those traces are distilled into a 7B student via plain SFT, the student can fail to learn effectively — not because the data is wrong, but because the trace complexity exceeds what a 7B model can absorb without scaffolding.

Learnability gap estimation:
  DeepSeek-R1 671B: avg trace length ~2,000-3,000 tokens for competition math
  Llama-3-8B: context length sufficient, but reasoning depth limited
  
  Observable symptom: student model reproduces surface trace structure
  (numbered steps, "let me verify") but makes wrong reasoning transitions —
  it learned the format, not the logic.
  
  Fix: difficulty-aware curriculum
  - Phase 1: train on traces where final answer is correct AND trace length < 800 tokens
  - Phase 2: filter to < 1,500 tokens, correct final answer, no obvious logical jumps
  - Phase 3: full trace length, correct answers only
  
  This 3-phase curriculum vs flat SFT on all traces:
  ~+4-8 points on MATH-500 for 7B student (results vary by base model)

The intervention at the data level is curriculum filtering: sort traces by difficulty proxy (trace length, number of reasoning steps, or a separate difficulty classifier), and train on progressively harder examples. Flat SFT on all traces wastes compute on examples the model cannot yet learn from, and can actively degrade performance by teaching the model to pattern-match on structural features of hard traces without learning the underlying reasoning.

Rubric-based generation for non-verifiable domains

RLVR — Reinforcement Learning with Verifiable Rewards — works cleanly for math and code because you can check the answer programmatically. For science questions, medical reasoning, and factuality tasks, there is no automatic verifier.

The 2025 frontier response is rubric-based reward generation. For each prompt, a frontier model generates a set of atomic evaluation criteria specific to that prompt:

# Rubric generation for non-verifiable domains
from anthropic import Anthropic

client = Anthropic()

def generate_rubric(prompt: str, draft_response: str) -> list[str]:
    """Generate prompt-specific evaluation criteria."""
    rubric_prompt = f"""
Given this question and a draft answer, generate 5-8 atomic evaluation criteria.
Each criterion must be independently checkable (yes/no) without human expertise.

Question: {prompt}
Draft answer: {draft_response}

Output a JSON array of criteria strings. Each criterion should be specific,
falsifiable, and cover a distinct aspect of quality.
"""
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": rubric_prompt}],
    )
    import json
    return json.loads(response.content[0].text)

# Then score candidate responses against the rubric
def score_against_rubric(response: str, criteria: list[str]) -> float:
    # Ask LLM-judge to check each criterion independently
    # Return fraction of criteria met
    ...

This converts a non-verifiable domain task into a verifiable-at-inference-time scoring task. The rubric itself is generated by a frontier model, so it inherits the frontier model's judgment — which is imperfect — but it's substantially better than a generic reward model that has no prompt-specific knowledge. OpenThoughts 3 and similar 2025 open datasets use rubric-based scoring to extend reasoning training beyond math and code.

Rubric-scored synthetic pairs feed preference training, and the training method determines how much data you need and what hardware it takes:

{"type": "rlhf-dpo", "title": "RLHF vs DPO vs GRPO training pipelines", "method": "dpo"}

What breaks

Teacher model outages and rate limits

A synthetic data pipeline that generates 250K samples over three days hits API rate limits repeatedly. Build queue management and retry logic with exponential backoff from day one. Batching prompts (using batch API endpoints where available) reduces cost by ~50% and smooths rate-limit pressure. Budget for ~10–15% sample loss from timeouts and filtering of malformed outputs.

Teacher consistency drift

GPT-4o is not a fixed function. Model updates change generation style, verbosity, and occasionally factual claims. If you generate data across a model update boundary, your dataset will have a distribution shift between the early and late portions. Pin your teacher model version (using specific model IDs like gpt-4o-2024-11-20) and regenerate if an update introduces visible style changes.

Diversity collapse without explicit prompting

Left to generate freely, models gravitate toward common instruction types. A Magpie run without explicit diversity prompting produces too many "explain X," "write code for Y," and "translate Z" examples relative to the tail of genuinely complex multi-step tasks. Force diversity by prompting the teacher with topic seeds sampled from a topic taxonomy, or by tracking category distribution in real time and suppressing overrepresented categories.

Evaluation set leakage through LLM-judge

If you use the same frontier model as both teacher (data generation) and judge (quality filtering), the judge's scores are not independent. GPT-4o will rate GPT-4o-generated responses higher than Claude-generated responses for similar quality. Use a different model family for judging than for generation, or use a reward model trained on human preferences rather than model preferences.

The "correct but hallucinated" problem

A teacher LLM generates a response that is fluent, well-structured, follows the instruction format perfectly, and states a false fact confidently. Your reward model gives it a high score because it looks good. Your contamination checker passes it because the false fact doesn't appear in eval benchmarks. The student learns the false fact. This is the hardest failure mode to catch because no automated filter catches confidently wrong content without domain-specific factual verification.

For factuality-sensitive domains (medical, legal, financial), include a factual verification step using a separate retrieval system or knowledge base. For general-purpose instruction tuning, accept that a small fraction of synthetic samples will contain errors and design your evaluation pipeline to detect the downstream effect.

Agentic data pipelines: the 2025 frontier

Static instruction-response pairs are insufficient for training agents that use tools across multiple steps. NeMo Gym, RLFactory, and similar frameworks (mid-2026) generate training data by running agents in interactive simulated environments — a tool-calling loop where the model generates tool calls, a simulator executes them, and the trajectory is recorded.

This creates multi-turn preference data: complete trajectories with intermediate tool calls, observation, and final success/failure labels. The preference signal comes from whether the trajectory reached the goal, not from human rating of individual responses. For agent fine-tuning, this trajectory-level data is strictly better than static pairs because the student learns to handle the distribution of states it will encounter during deployment, not idealized single-turn responses.

The contamination and collapse risks apply here too, but the diversity concern is more acute: simulated environments have known topologies, and agents can overfit to the simulator's specific state space rather than generalizing to real environments. Varying environment parameters, randomizing task structure, and evaluating on held-out environments before deployment are the standard mitigation.

When to use what

Use Magpie-style seed-free generation when you want broad instruction-following coverage with no existing domain data, or when you need to quickly bootstrap a dataset for a new task and don't want to manually curate seed examples.

Use Evol-Instruct when you have 200–500 domain-specific examples and want to scale them to 50K+ without losing domain focus. The seed distribution guides the mutations toward your domain, which is the property you want.

Use rubric-based generation for preference data in non-verifiable domains (science, factuality, reasoning over text) where RLVR cannot apply. Budget for the extra inference call per sample to generate the rubric, and use a different model family to generate rubrics than to generate candidate responses.

Use trajectory-based generation when you are fine-tuning a model for multi-step tool use or agent behavior. Static pairs will not produce agents that can recover from tool errors or adapt to unexpected environment states.

The decision on whether to fine-tune at all comes before choosing a generation strategy. Synthetic data engineering is only worth the engineering investment when fine-tuning is the right choice for your task. If the problem is factual retrieval over a dynamic knowledge base, no amount of synthetic data engineering will substitute for RAG. If you do decide to run SFT, the synthetic data pipeline described here is the way to get training data cheaply without sacrificing the quality controls that determine whether the fine-tuned model is actually better.

The team in the opening story eventually rebuilt their pipeline with contamination checking, reward model filtering, and 25% real-data mixing. Their re-run on 90K filtered samples (from 230K generated) produced a model that scored 74% on their internal eval — four points lower than before. But on genuinely novel user queries, it outperformed the old model by 22 points. The real improvement was always there. The inflated score was the problem.