SFT and Instruction Tuning: Data Is the Job

The first time you look at a fine-tuning run, it looks reassuringly simple. Loss goes down. Evaluation scores go up. You ship the model and the support team starts getting a new category of complaint: the model confidently gives wrong answers in a way the base model never did. That is what happens when you SFT on data with a high hallucination rate — you did not teach the model to be better, you taught it to be wrong with more conviction.

This is the core SFT lesson, and teams re-learn it constantly: the training loop is three lines of PyTorch. The curation pipeline is the actual engineering job.

What SFT actually does

Supervised fine-tuning takes a pretrained base model and trains it further on a dataset of (instruction, response) pairs using the same cross-entropy objective used during pretraining — but with a critical difference. Loss is computed only on the response tokens, not the instruction tokens. The gradient flows backward from the response; the instruction is context, not target.

# Pseudocode: how loss masking works in a standard SFT trainer
# Using HuggingFace Trainer with TRL's DataCollatorForCompletionOnlyLM

from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
response_template = "<|start_header_id|>assistant<|end_header_id|>"

collator = DataCollatorForCompletionOnlyLM(
    response_template=response_template,
    tokenizer=tokenizer,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    data_collator=collator,   # masks instruction tokens in loss
    args=training_args,
)
trainer.train()

The gradient update adjusts the model's weights so it assigns higher probability to the tokens in the ideal response given the instruction context. Run this over enough high-quality examples and the model learns the format, tone, style, and knowledge patterns represented in your data.

The word "knowledge" is where teams get into trouble. SFT does not inject new factual knowledge the way people assume. It adjusts the probability distribution over tokens. If your training examples contain factual errors, the model learns to produce those errors confidently. If your training examples are correct but inconsistent, the model learns inconsistent behavior. The model is not reading and memorizing facts; it is learning a transformation from instruction-context to response-token-sequence.

flowchart TD
    BASE["Base model\n(pretrained on the internet)"] --> SFT_STEP["SFT on instruction data"]
    SFT_STEP --> INST["Instruction-following model\n(format, style, behavior)"]
    
    NOTE1["What SFT changes:\nformat compliance, response style,\nbehavior patterns, tone"] --> INST
    NOTE2["What SFT does NOT reliably inject:\nnew factual knowledge,\nmath/code capabilities beyond base"] -.-> INST

    style BASE fill:#0e7490,color:#fff
    style INST fill:#a855f7,color:#fff
    style NOTE1 fill:#15803d,color:#fff
    style NOTE2 fill:#ff2e88,color:#111

If the real problem is that your model hallucinates facts, SFT on factually incorrect training data makes it worse. The right fix for factual gaps is RAG or grounding, not more training. The decision ladder between fine-tuning, RAG, and prompting matters here before you spend a single GPU-hour.

The chat template problem

Before any data quality discussion, one mechanical issue kills more fine-tuning runs than any other: using the wrong chat template.

Every instruction-tuned model expects its instructions wrapped in a specific format. Llama-3 uses:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
4<|eot_id|>

Mistral-v0.3 uses [INST] ... [/INST]. Qwen2.5 uses <|im_start|> tokens. If you fine-tune with a different template than the one used during the model's instruction-tuning pretraining, you are fighting the model's priors. The base model may still converge, but you will see artifacts at inference time — partial completions, weird stops, incorrect turn separations — because the model learned to associate certain tokens with role boundaries, and you have replaced those tokens with something it has never seen in that context.

Use tokenizer.apply_chat_template() from Transformers. Set tokenize=False to inspect what the template actually produces before feeding it to training. If your base model is a raw pretrained model (not instruction-tuned), this matters less, but for fine-tuning on top of an existing instruction-tuned model, template mismatch is a guaranteed subtle failure.

Data quality: what the research actually shows

The most cited evidence in SFT data discussions is the AlpaGasus experiment. Alpaca-52K was one of the first large instruction datasets, generated by GPT-3.5 from seed instructions. It had widely documented problems: repeated instructions, responses that were wrong, responses that mixed answers to multiple questions, formatting inconsistencies. A follow-up paper called AlpaGasus used ChatGPT as a quality judge to score each example from 0 to 5 and filtered down to roughly 9,000 examples rated above 4.5.

The result: Llama-1 fine-tuned on 9,000 filtered examples outperformed the same model fine-tuned on all 52,000 on every downstream benchmark tested. Not by a small margin. The quality filter removed roughly 83% of the data and improved performance.

This is not an isolated finding. LIMA (Less Is More for Alignment, Meta 2023) showed that 1,000 carefully curated, high-diversity human-written examples could produce a model competitive with much larger fine-tuned datasets. The field's working model is now: you need enough examples to cover your behavioral space, and every example that does not meet that quality bar is negative signal.

What makes an example high quality?

• The instruction is unambiguous. A single, clear task — not two questions merged or a question that could be interpreted five ways.
• The response fully addresses the instruction. Not a partial answer, not a response to a slightly different question, not a refusal when the task is benign.
• The response is factually correct in the domains you are training on. "Factually correct" for style tasks means stylistically consistent.
• The response is at the target length. Do not include responses that are half a sentence if your use case requires paragraph-length answers; do not include essay-length responses if you want concise outputs.
• Coverage. High quality across 100 examples of the same type is less valuable than high quality across 100 diverse types. Diversity in the instruction distribution is the second axis after per-sample quality.

Building the data pipeline

A production SFT data pipeline has five stages. Most teams compress them into a single script; separating them makes each stage testable independently.

Stage 1: Collection. Gather instructions from your domain. Sources in rough order of quality ceiling: human-written expert examples, domain-specific documentation turned into instruction-response pairs, synthetic generation from a frontier model, repurposed existing data.

Stage 2: Deduplication. MinHash or SimHash across instruction text. Exact duplicates are common in scraped data and in synthetic pipelines that drift toward popular topics. Deduplicate at ~0.85 Jaccard similarity threshold. Near-duplicates teach the model a narrow slice of the space very well and everything else poorly.

Stage 3: Quality filtering. Three approaches, ordered by cost:

• Perplexity filtering: compute the base model's perplexity on each response; unusually high perplexity often indicates grammatical errors or factual noise. Set a threshold at roughly the 95th percentile.
• LLM-judge scoring: prompt a frontier model to rate each (instruction, response) pair on a rubric (instruction-following, factual accuracy, clarity, appropriate length). Filter below a threshold, typically 4/5 or equivalent. Cost at scale: ~$0.005 per example using GPT-4o mini.
• Reward model scoring: use an open reward model (Llama-3-based reward models exist on HuggingFace) to score examples. Faster than an API call, lower quality ceiling than a frontier judge.

Stage 4: Formatting. Apply the chat template, add the system prompt if one will be present at inference, compute attention masks, and set labels to -100 for instruction tokens (this is what tells the loss function to ignore them).

Stage 5: Train/eval split. The split must be on a different distribution from training data, not just a random hold-out. If you train on customer support tickets from Q1 and evaluate on Q1 tickets, you will not catch a regression on Q2 ticket types. Carve out a held-out set from a domain slice not represented in training, or a time window after training data ends, or a manually curated set of failure modes you actually care about. The eval pipeline for LLM systems is its own topic, but the principle here is simple: your eval set must be capable of detecting things going wrong that your training set cannot show you.

Multi-turn conversations

Everything above frames an example as one (instruction, response) pair, but most real data — including the support tickets in the worked example below — arrives as multi-turn threads. The masking rule generalizes: set labels to -100 for every token the assistant did not produce (system prompt, all user turns, role markers) and compute loss on every assistant turn, not just the last one. Training only on the final turn discards most of your signal and teaches the model that mid-conversation replies are unimportant. If one turn in an otherwise good conversation is low quality, truncate the conversation just before it rather than trying to mask it selectively.

DataCollatorForCompletionOnlyLM handles the multi-turn case if you pass instruction_template (the user-turn marker, e.g. <|start_header_id|>user<|end_header_id|>) alongside response_template — it then alternates the mask at each role boundary instead of unmasking everything after the first assistant header. If your tooling only supports single-turn masking, split each conversation into per-turn examples where the prompt is the full prior history. You pay duplicated context tokens across the split examples, but the masking is correct, which matters more.

Synthetic data with Magpie

The practical problem with the pipeline above is that Stage 1 (collection) is expensive. Human-written expert examples cost $1–5 each. For 10,000 examples, that is $10,000–50,000 before filtering.

Magpie (accepted at ICLR 2025) sidesteps the seed data problem. An already-aligned model has internalized the distribution of human requests during its own instruction training. Prompt it with only the opening tokens of the user turn — literally just the beginning of the chat template, before any user text — and it generates a realistic user query autoregressively. Then generate the assistant response. One API call yields one complete (instruction, response) pair with no seed instructions required.

The ICLR 2025 Magpie paper filtered 300,000 generated pairs using a reward model and found the resulting dataset outperformed ShareGPT, Evol-Instruct, and UltraChat when used to fine-tune Llama-3-8B. The mechanism is direct: because the generating model was trained on real human queries, its query distribution is broader and more realistic than any static seed instruction set.

One constraint people miss: Magpie needs raw-completion access, because you are sending a bare chat-template prefix and letting the model complete the user turn. Chat-completions APIs (OpenAI's included) will not accept that, so GPT-4o cannot be your Magpie generator. You need an open-weight instruct model on a completions-capable host — Llama-3.1-70B-Instruct on Together or Fireworks, or self-hosted vLLM. The upside is that open-weight token pricing makes generation nearly free:

10,000 (instruction, response) pairs via Magpie pipeline
(pricing illustrative as of mid-2026):
  Generation (Llama-3.1-70B-Instruct on a completions API,
  ~$0.90/M output tokens, avg 400 tokens/pair):            ~$0.0004/pair → ~$4
  Quality filtering (reward model scoring, on-device):      ~$0/pair after model download
  Quality filtering (LLM-judge alternative):                ~$0.005/pair → $50
  Total:                                                    ~$5–55

vs. human annotation at $2/sample:                          $20,000

See the synthetic data engineering article for the full pipeline with working code, model collapse prevention, and contamination checks against standard eval sets — all of which apply to any Magpie-based collection effort.

Training recipe

Once the data is clean, training is the easy part. Ranges that work across 7B–13B models on LoRA:

Learning rate:     2e-4 (cosine decay, 3-5% linear warmup)
Epochs:            2–3
Effective batch:   16–64 (use gradient accumulation if per-device batch is small)
Precision:         bf16 (not fp16 — bf16 is more numerically stable at this learning rate)
Sequence length:   Match your longest training example; pad shorter ones
LoRA rank:         r=32 for general SFT; r=16 for narrow style tasks; r=64 for code/reasoning
LoRA alpha:        2× rank (alpha=64 for r=32)
LoRA targets:      q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

The last bullet is important. The default in many tutorials targets only q_proj and v_proj. Missing the MLP projections (gate_proj, up_proj, down_proj) freezes most of the model's "knowledge" layers. For reasoning or coding tasks, this caps quality gains significantly. For pure style tasks (response format, tone), it is sometimes acceptable. Default to all seven projections unless you have a specific memory constraint.

{ "type": "lora", "model": "7b", "rank": 32, "title": "LoRA rank vs. trainable parameters and adapter VRAM (7B base)" }

General capabilities decay is real with narrow datasets. If you fine-tune a 7B model on 5,000 medical Q&A examples and nothing else, it will get better at medical Q&A and worse at everything else. The practical fix is a general instruction data mix: add 5–10% of a general instruction dataset (open datasets like OpenHermes, Orca, or a Magpie-generated general set) to act as an anchor. This is sometimes called rehearsal or continual learning mixing.

sequenceDiagram
    participant DATA as Training data
    participant TRAINER as SFT Trainer
    participant MODEL as Model weights
    participant EVAL as Held-out eval

    Note over DATA: Domain examples (90%)\n+ General mix (10%)
    DATA->>TRAINER: Batches with response-only loss mask
    TRAINER->>MODEL: Gradient update (bf16, cosine LR)
    MODEL-->>EVAL: Checkpoint at end of each epoch
    EVAL-->>TRAINER: Domain task score + general capability score
    Note over EVAL: If general capability regresses,\nincrease general mix %

The two numbers to watch during training: domain task performance and a general capability proxy. If domain performance improves but general performance drops below your threshold, the general mix is insufficient or the learning rate is too high.

What breaks

Training on validation distribution. The most common way to ship a model that looks good but regresses in production. If your held-out eval set is a random 10% split from the same data as training, you are measuring memorization, not generalization. Hold out a distinct slice, or use a curated human evaluation set for the tasks you actually care about.

Wrong chat template at inference. You formatted training data correctly but call the inference endpoint without applying the chat template. The model receives raw text instead of the formatted turn structure it learned. Output quality degrades in exactly the way you saw in training on wrong-template data, just at inference time. Always use tokenizer.apply_chat_template() at inference too.

Catastrophic forgetting. The model fine-tuned on domain data forgets how to follow general instructions. Symptom: great on domain queries, bizarre on anything else. Fix: general data mixing at training time, or a LoRA approach that avoids changing the base model's general parameters.

DPO without an SFT warm-up. If you plan to follow SFT with preference optimization via DPO or SimPO, the SFT step is not optional. DPO applied to a base model that is not instruction-tuned tends to diverge — the log-ratio objective assumes the model already knows what reasonable responses look like. SFT first, then DPO.

Instruction noise inflation. Your quality filter grades responses, but not instructions. An instruction that is grammatically correct but semantically confused ("explain what makes blue the color that it is as a metaphor for database normalization") will still pass response quality filters and produce a training example where the model learns to respond to nonsense. Grade the instruction side too, or at minimum review a random sample manually.

Benchmark contamination. If GPT-4 generated your training data and you evaluate on MMLU or GSM8K, there is a contamination risk: GPT-4's training data likely included those benchmarks. Your scores are comparing contaminated training to contaminated eval. This does not invalidate synthetic data pipelines, but it does invalidate leaderboard comparisons. For internal quality tracking, use a held-out domain eval that GPT-4 could not have memorized.

Worked example: a customer support SFT dataset

To make this concrete, here is how I would build a 5,000-example SFT dataset for a B2B SaaS support assistant.

Step 1: Seed data collection
  - Export 2 years of closed support tickets (instruction = user query, response = agent resolution)
  - ~15,000 raw examples, roughly tagged by product area

Step 2: Deduplication
  - MinHash at 0.85 similarity → removes ~30% near-duplicates
  - Remaining: ~10,500 examples

Step 3: Quality filtering
  - Perplexity filter: remove top 5% highest-perplexity responses → ~10,000 remain
  - LLM judge (GPT-4o mini, scoring 1-5 on: instruction clarity, response correctness, completeness, tone):
    Filter below 3.5 → ~6,000 remain
  - Cost: 10,000 × $0.005 = $50

Step 4: Coverage check
  - Group by product area tag
  - Top 3 categories represent 70% of dataset → resample: cap at 20% per category
  - Augment thin categories with Magpie-generated synthetic examples (same quality filter applied)
  - Final dataset: 5,000 examples, balanced across categories

Step 5: Held-out eval
  - 200 examples manually selected from Q4 tickets (after training data cutoff)
  - Include 50 adversarial examples where the old system failed

Training cost (QLoRA on a single A100 80GB, 5K examples, 3 epochs):
  ~2 hours at $2-3/hour on RunPod or Modal → ~$5–8 total

At $50 for filtering plus $8 for training, this is genuinely accessible. The expensive part is the human time auditing the quality filter output and manually reviewing the held-out examples — not the compute.

{ "type": "rlhf-dpo", "method": "dpo", "title": "RLHF vs DPO vs GRPO: what each stage removes" }

When SFT is the wrong tool

SFT is the right tool when you need behavioral change that is stable across inputs: format compliance, tone, response structure, domain-specific answer style, task specialization. It is the wrong tool for:

Injecting new factual knowledge. SFT can reinforce facts the model already has weak associations with, but training on domain facts with a small dataset mostly teaches the model to generate plausible-sounding domain text. For factual accuracy, RAG is more reliable.

Fixing hallucination by adding correct answers. The model learns to associate the pattern of the instruction with the pattern of your correct response. It does not develop a fact-checking mechanism. On slightly rephrased versions of the same question, it may still hallucinate.

Preference alignment. SFT teaches the model to produce outputs that look like your examples. If you want to steer the model toward preferred outputs compared to rejected ones — less sycophancy, less refusal, better reasoning quality — that requires preference optimization, which adds a second training stage after SFT.

Complex reasoning. Adding reasoning traces to your SFT data helps, but for deep reasoning improvements, reasoning distillation from a larger model or GRPO-based training with verifiable rewards is more effective than instruction SFT alone.

The judgment call

The playbook for 2026 SFT is clear enough to state as a decision sequence:

1. Define the behavior you want with five concrete examples before touching training code. If you cannot write five unambiguous (instruction, response) pairs that represent what you want, your data collection will be fuzzy.

2. Decide your data source. Human-annotated examples if you have domain experts and budget. Synthetic Magpie-style generation if you want breadth. Real user traffic filtered for quality if you have a deployed product. Usually some combination.

3. Filter aggressively. Use an LLM judge on every example. Set the quality bar high enough that you lose 40–60% of examples. What remains should represent what you actually want the model to do.

4. Check coverage. The model will learn the distribution of your training set. If 80% of examples are one task type, 80% of what it learns will be that task type. Resample to balance coverage across the behavioral space you care about.

5. Add a general mix. 5–10% of a general instruction dataset prevents catastrophic forgetting. Do not skip this unless the model will exclusively be used for one narrow task in a controlled environment.

6. Evaluate on a held-out set that can catch regressions. The eval set that shipped your model is not useful for catching failure modes — it already passed. A separate adversarial set and a general capability probe are both worth maintaining.

SFT is not the endpoint. For alignment and quality, DPO or SimPO runs after SFT on preference-paired data. For reasoning, distillation from a larger model's chain-of-thought augments what SFT alone can achieve. And if you are reaching for fine-tuning because the model cannot answer domain questions accurately, check the decision ladder first — RAG often solves the factual problem at a fraction of the cost, and the two approaches compose well anyway.

The model that works in production is almost never the one with the most training data. It is the one built from the data someone cared enough to curate.