Fine-Tuning: The Decision Framework

A team spends two weeks collecting examples, fine-tunes a 13B model, and ships. Support tickets spike the next day — not because the model got worse, but because it now answers questions about the 2024 pricing page with complete conviction. The new pricing page is six months old and nowhere in the training set. Two weeks of training to bake in a bug that a nightly re-index would have prevented.

This is the most common fine-tuning mistake, and it has nothing to do with hyperparameters or learning rates. The team reached for fine-tuning when they had a knowledge problem, not a behavior problem. The fix was a RAG pipeline that takes ten hours to build, not a two-week model training job.

Fine-tuning is not "making the model better at your task." It is a specific surgical tool for a specific class of problems. Know what it cuts and what it doesn't.

The escalation ladder

Before writing a single training example, you should have exhausted two cheaper options in sequence. Each one has a ceiling. Fine-tuning starts making sense only when you've hit that ceiling. Each arrow below is an escalation trigger — the reason you leave the rung you're on.

flowchart TD
    A[Prompting] --> |"knowledge is dynamic, external, or too large for context"| C[RAG]
    C --> |"behavior/format/latency goal not achievable by retrieval"| D[Fine-Tuning]
    D --> |"narrow task, high volume, need <200ms"| E["Small Model + Fine-Tuning"]
    style A fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style E fill:#a855f7,stroke:#a855f7,color:#fff

Prompting gets you 60–70% of the way on most tasks in hours, with zero infrastructure. The ceiling is reproducibility — you can't enforce output format, tone, or behavior with prompts alone at any scale. When you've written a 2,000-token system prompt and it still breaks on 15% of inputs, something else is needed.

RAG solves the knowledge problem. Facts, documents, policy pages, product data — anything that changes or exceeds your context window lives in a retrieval index. The cost is a retrieval hop (50–150 ms depending on index size), plus index maintenance. That's fine for most applications. The ceiling is behavioral: RAG does nothing for consistent output format, tone matching, or latency requirements that exclude network round-trips.

Fine-tuning solves the behavior problem. It does not solve the knowledge problem. If you can't say clearly which behavior you're trying to change — and write 50 test cases that demonstrate the current failure and the desired output — you're not ready to fine-tune.

The question is not "should I fine-tune?" The question is: "Am I trying to change FORM or inject KNOWLEDGE?" FORM is fair game. KNOWLEDGE is not.

SFT: data is the whole job

Supervised fine-tuning (SFT) is conceptually simple: you provide (instruction, response) pairs, run cross-entropy loss over the response tokens, and update the weights. The model learns to mimic the distribution of your examples.

The entire quality ceiling is set by your data. Not your learning rate. Not your epochs. Not which base model you start from. The data.

The clearest empirical proof: the AlpaGasus study filtered the Alpaca 52K dataset down to ~9K high-quality samples by scoring each example with a strong language model judge. The 9K subset outperformed the full 52K set on every downstream benchmark. More data, lower quality — strictly worse.

What makes data high quality for SFT?

• Diversity across the task space, not redundant rephrasing of the same request
• Correctness — wrong answers train confident wrong behavior
• Format consistency — if you want JSON output, every training example should produce JSON
• Difficulty calibration — too easy and the model learns nothing new; too hard and it memorizes noise

The Magpie pipeline (ICLR 2025) generates both the user query and the response by prompting an already-aligned model with only the chat template preamble, letting it autoregressively produce the instruction. No seed data required. 300K filtered Magpie instances on Llama-3-8B outperformed ShareGPT and Evol-Instruct on AlpacaEval 2. The technique is cheap (~$0.01/sample via API) and surprisingly effective for broad instruction-following tuning.

For domain-specific tasks, human-curated examples from subject matter experts are still worth the cost. Budget $1–5 per example for annotation, target 500–2,000 examples before your first training run, and spend the first 20% of your budget on the eval set.

That last point deserves its own line: build your eval set first, before you collect a single training example. An eval set of 100–200 labeled examples lets you know whether the fine-tune improved anything. Without it, you are flying blind.

Standard training recipe for SFT with a small dataset:

# Using Hugging Face TRL (up to date as of mid-2026)
import torch
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",              # QLoRA's NF4 — the default is fp4
        bnb_4bit_compute_dtype=torch.bfloat16,  # otherwise compute falls back to fp32
    ),
    device_map="auto",
)

lora_config = LoraConfig(
    r=32,                    # rank — see below for how to choose
    lora_alpha=64,           # alpha = 2 * r is a reliable default
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",   # all 7, not just q+v
    ],
    lora_dropout=0.05,
    bias="none",
)

trainer = SFTTrainer(
    model=model,
    train_dataset=train_ds,
    args=SFTConfig(
        output_dir="./checkpoints",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,   # effective batch = 16
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        bf16=True,
    ),
    peft_config=lora_config,
)
trainer.train()

Notice target_modules includes all seven projection matrices. A common mistake is targeting only q_proj and v_proj — the two most commonly cited in tutorials. That leaves the MLP projections (gate_proj, up_proj, down_proj) frozen, which significantly caps quality on any task requiring reasoning.

LoRA and QLoRA: the VRAM math

Full fine-tuning a 7B model in float16 requires storing the base weights (14 GB), optimizer states (AdamW stores two moments per parameter in float32 — another ~56 GB), gradients (~14 GB), and activation memory — comfortably over 100 GB total. That needs a node with 4–8 A100 80GB GPUs.

LoRA changes the math fundamentally. Instead of updating all parameters, it inserts low-rank trainable matrices next to the frozen base weights. For a weight matrix W of shape (d_in, d_out), LoRA adds two matrices A (d_in × r) and B (r × d_out) where r is the rank, typically 8–64. At inference time, W + BA gives you the adapted weight. Only A and B are trained.

Trainable parameters (7B model, r=32, 7 target modules):
  Per module: r × (d_in + d_out)
  Attention (q, k, v, o): 4096×4096 → 32 × 8,192 = 262,144 each → ~1.05M
  MLP (gate, up, down):   4096×11008 → 32 × 15,104 = 483,328 each → ~1.45M
  Per layer ≈ 2.5M × 32 layers ≈ ~80M trainable params
  vs ~6,700M total → ~1.2% of weights move

QLoRA (from Dettmers et al.) stacks another reduction on top: it quantizes the frozen base weights to 4-bit NF4 format, cutting base weight memory from 14 GB to ~4 GB for a 7B model. The LoRA adapters remain in bfloat16. With paged optimizers, a 7B QLoRA fine-tune fits comfortably on a single RTX 4090 (24 GB).

Explore the rank-vs-memory tradeoff interactively:

{
  "type": "lora",
  "title": "LoRA rank vs trainable parameters and VRAM",
  "rank": 32,
  "model": "7b"
}

How to pick rank:

• r=16: style tasks, format alignment, light behavioral nudges
• r=32: general SFT, instruction following, most production use cases
• r=64: complex coding tasks, multi-turn reasoning, heavy domain adaptation

Set lora_alpha = 2 × r. Always target all seven projection matrices. Merging the adapter back into the base weights at inference time eliminates the adapter's latency overhead entirely — the final deployed model runs at base model speed.

One gotcha that trips up a lot of people: if you quantize the base to 4-bit and then try to merge a bfloat16 adapter, the weights corrupt. Always dequantize the base to bfloat16 or float16 before merging.

DPO: alignment without the RL circus

SFT teaches the model what you want. It doesn't teach the model what you don't want, or how to rank outputs against each other. That's where preference optimization enters.

Classic RLHF (Reinforcement Learning from Human Feedback) requires:

1. A fine-tuned SFT model
2. A separately trained reward model
3. A PPO training loop with actor and critic networks

That's three models running simultaneously, with PPO's notoriously sensitive hyperparameters and the reward model's own data collection problem. Expensive, fragile, and hard to reproduce.

DPO (Direct Preference Optimization) replaces the reward model with a mathematical equivalence: given (prompt, chosen_response, rejected_response) triples, the optimal policy update is equivalent to maximizing the log-ratio of the policy's probability for the chosen response versus the rejected response, relative to a reference model. One model, one loss function, no RL loop.

The result is comparable alignment quality with dramatically lower complexity. DPO is now the default first alignment step in most post-training pipelines.

{
  "type": "rlhf-dpo",
  "title": "RLHF vs DPO vs GRPO: what each method removes",
  "method": "dpo"
}

SimPO (2024) improves on DPO by dropping the reference model entirely. It uses average log-probability as an implicit reward rather than a log-ratio against a frozen reference. The published paper showed clear gains over DPO on both AlpacaEval 2 and Arena-Hard (the specific deltas depend on the base model and setup). Less infrastructure, better numbers.

ORPO (Odds Ratio Preference Optimization) merges SFT and preference tuning into a single training stage by adding an odds-ratio penalty on rejected responses alongside the standard language modeling loss. No reference model, no separate SFT pass, faster iteration — useful when you want to go from raw base model to aligned model in one shot.

For data collection, you need (chosen, rejected) pairs. The cheapest source is synthetic: generate N completions for each prompt, rank them with a strong judge model or your own reward function, and take the best and worst as your pair. No human annotators required for a first pass.

Distillation: teaching a 4B model to reason like a 70B model

The DeepSeek-R1 release in January 2025 changed what small models can do. DeepSeek trained a 671B MoE model with GRPO and verifiable rewards until it produced structured chain-of-thought traces. Then they ran plain SFT on 800K of those traces to train six dense student models ranging from 1.5B to 70B parameters.

The result: DeepSeek-R1-Distill-Qwen-7B outperformed QwQ-32B-Preview on math benchmarks — a 7B model, trained on teacher traces via standard cross-entropy loss, beating a 32B model on AIME and MATH-500 (it still trailed on coding; the 14B distill is the one that wins across the board).

The mechanism is offline distillation: the teacher generates samples ahead of training, the student learns to reproduce them via SFT. Simple. The tricky part is the learnability gap — a 7B model can't always learn from a 671B model's hardest traces. Too large a gap between student and teacher causes the student to memorize surface features without learning the underlying reasoning pattern. Curriculum filtering (sorting examples by difficulty and starting with easier ones) and difficulty-aware data sampling both help.

sequenceDiagram
    participant Teacher as "Teacher (671B)"
    participant Filter as "Quality + Difficulty Filter"
    participant Student as "Student (7B)"
    participant Eval as "Held-out Eval Set"

    Teacher->>Filter: 1M reasoning traces
    Filter->>Student: 800K filtered traces
    Student->>Eval: Training run (SFT)
    Eval-->>Student: Regression check per epoch
    Note over Student,Eval: Eval set built before training

For teams without H100 clusters: the OpenThoughts 3 dataset (2025) offers 1M+ open synthetic reasoning samples across math, code, and science. You can fine-tune a 7B model on a subset for a few hundred dollars on cloud GPUs and get a task-specific reasoner competitive with much larger base models on your domain.

When the small fine-tuned model wins

A fine-tuned 4B model beats a frontier 70B model under three conditions that co-occur in a lot of production systems:

1. Narrow task: the task has a clear, bounded input-output structure that can be expressed in 2,000 training examples
2. High volume: enough traffic that per-token cost is a real budget line
3. Latency constraint: the application needs responses under 200 ms, which excludes large frontier models on shared infrastructure

Back-of-envelope on a real use case — structured data extraction from invoices:

Frontier model (e.g., Claude Sonnet-class, mid-2026 illustrative pricing):
  Input:  ~800 tokens × $3/M  = $0.0024/request
  Output: ~200 tokens × $15/M = $0.0030/request
  Total:  ~$0.0054/request
  At 10M requests/month: $54,000/month
  Latency: 1,500–3,000ms (shared API)

Fine-tuned 4B model (self-hosted, 1× A10G at $1.20/hr):
  Throughput: ~200 requests/min (continuous batching)
  Cost: $1.20/hr ÷ 200 req/min ÷ 60 min = $0.0001/request
  At 10M requests/month: ~$1,000/month
  Latency: 80–150ms

Break-even on GPU cost alone:
  ~$876/month fixed (A10G always-on) ÷ ~$0.0053 saved/request
  ≈ 165K requests/month
Add amortized eng/ops time (someone owns that GPU) and the
practical bar lands at a few hundred thousand requests/month

Those numbers are illustrative and move with GPU spot prices and API pricing, but the structure holds. At low volume, the frontier API wins on total cost (no infra overhead). At high volume on a narrow task, a small fine-tuned model running on a single GPU wins by a wide margin.

The two-column comparison hides the option most teams actually pick first: managed fine-tuning. OpenAI's and Google's fine-tuning APIs, and hosted LoRA serving from providers like Together and Fireworks, train and serve your adapter without you owning a GPU. The math changes shape: fixed cost drops to roughly zero, but you pay a per-token premium — fine-tuned endpoints typically run 1.5–4x the provider's base-model rate, though hosted LoRA adapters on open models often serve at or near base rates (illustrative, mid-2026). That makes managed the default at mid volume — tens of thousands to a few hundred thousand requests a month, below the self-host break-even — or whenever nobody on the team wants to carry a GPU pager. The trade-offs are concrete: your training data leaves your VPC, and the resulting adapter is frequently locked to that provider's serving stack rather than a portable artifact you can move. Self-host when volume clears the break-even and someone owns the infrastructure; go managed when it doesn't, or they don't — data governance permitting.

The model quality argument also holds on narrow tasks. A 4B model that has seen 2,000 examples of your exact invoice format will extract fields more accurately than a 70B model that has never seen your format and gets a one-shot example in the prompt. The smaller model has memorized the structure; the larger model is generalizing.

This stops being true when the task requires open-ended reasoning, multi-step planning, or broad world knowledge. Don't fine-tune a 4B model to be a general assistant.

What breaks in practice

The most expensive prompt fix. The first thing teams try is solving a quality problem with more training data. If the prompt is broken — ambiguous instructions, wrong persona, missing examples — fine-tuning bakes that brokenness into the weights permanently with higher confidence. Fix the prompt first. Then evaluate whether the problem is behavioral (fine-tuning material) or knowledge (retrieval material).

Skipping the eval set. Without a held-out eval set, you cannot know whether your fine-tune improved the target task or silently regressed on adjacent tasks. The eval set should exist before a single training example is collected. 100–200 labeled examples is enough to detect real movements. Teams who skip this step regularly ship a model that scores better on training loss and worse on everything that matters.

Benchmark contamination in synthetic data. If you generate training data using GPT-4o and then evaluate on GSM8K, your results are meaningless: GSM8K is likely in GPT-4o's training set. Cross-check your synthetic prompts against standard benchmarks before training. The AlpacaEval and Arena-Hard leaderboards have been contaminated by exactly this pattern.

DPO without SFT warm-up. DPO on a raw base model that hasn't been instruction-tuned first tends to diverge or degrade instruction-following quality. The SFT step is load-bearing — it gives the model enough instruction-following ability for DPO's log-ratio objective to push meaningfully in the right direction.

GRPO without a length penalty. GRPO samples groups of responses per prompt and normalizes rewards within the group. Without a length penalty, models learn quickly that longer responses maximize group-relative advantage scores without actually improving answer quality. Every GRPO run should include a length regularization term.

The knowledge trap. Worth saying twice because it keeps happening: a model that confidently serves stale facts needs RAG, not fine-tuning. Fine-tuning on a knowledge base produces a model that has memorized those facts as of training time, which is strictly worse than retrieval for anything with a shelf life. RAG in One Pass is the right module to read before making this mistake.

flowchart LR
    P[Problem type?] --> K{Is the failure\nknowledge-based?}
    K -->|Yes: wrong facts,\nstale data| R[RAG]
    K -->|No: wrong format,\nwrong tone, wrong style| B{Does prompting\nfix it?}
    B -->|Yes| PR[Better Prompt]
    B -->|No| FT[Fine-Tuning]
    style P fill:#00e5ff,color:#0a0a0f
    style R fill:#0e7490,color:#fff
    style FT fill:#a855f7,color:#fff

The post-training stack in 2025

The production post-training recipe solidified in 2025 and has become fairly standardized:

1. SFT on 500–5,000 curated examples (or 50K+ synthetic via Magpie-style generation) to establish task-specific behavior
2. DPO or SimPO on preference pairs to align outputs with quality preferences, reduce refusals, and improve helpfulness
3. GRPO or RLVR (optional) for tasks with verifiable rewards — math, code execution, structured output validation — to push reasoning quality further without human annotators

Steps 2 and 3 are optional for most application fine-tuning. If you're building an invoice extractor, step 1 is often sufficient. Steps 2 and 3 matter most when you're training a general-purpose model or a model that needs to reason through multi-step problems.

The full preference optimization article covers DPO, SimPO, ORPO, KTO, and GRPO with implementation details and their exact loss functions. The SFT data quality article covers the data pipeline in depth, including Magpie and Evol-Instruct.

Where to go next

The Decision Ladder: Fine-Tuning vs RAG vs Prompting — a more detailed decision tree with worked cost models for each option, good if you're trying to make the call for a specific production use case.

LoRA, QLoRA, and the PEFT Landscape — goes deep on rank selection, DoRA, rsLoRA, and the practical details of adapter merging and quantization-aware training.

SFT and Instruction Tuning: Data Is the Job — data collection pipelines, annotation quality, Magpie, Evol-Instruct, and how to filter your way to a better dataset.

Preference Optimization: DPO, SimPO, ORPO, and the Post-RLHF Stack — the complete preference optimization family, with loss functions, data formats, and when each method outperforms the others.

Reasoning Distillation: Teaching Small Models to Think — the DeepSeek-R1 distillation recipe in depth, including on-policy distillation, learnability gaps, and the OpenThoughts 3 dataset.

Eval First: Building the Safety Net Before You Need It — because you cannot know whether your fine-tune worked without this, and most teams build it last.