Preference Optimization: DPO, SimPO, ORPO, and the Post-RLHF Stack

In late 2022, when RLHF-tuned models had shipped from exactly one lab, the standard joke inside every other ML team was that PPO for language models was less "reinforcement learning" and more "reinforcement anxiety" — you kept four models in sync, tuned KL penalties to stop the policy from collapsing, trained a reward model on annotation data that you hoped wasn't biased, and crossed your fingers that the critic network converged before you ran out of compute budget. The Anthropic Constitutional AI paper and InstructGPT paper made it look tractable at the lab scale. At the startup scale, it mostly looked like a six-week project that ended in "we went back to SFT."

DPO changed that in 2023. Then SimPO, ORPO, and GRPO reshaped what post-training looks like in 2024–2025. This article covers how each works mechanically, what each one removes from the stack, and the decisions you actually face when picking one for a real training job.

What RLHF-PPO actually required

Before you can appreciate what was removed, it helps to be specific about what the classic setup contained.

Stage 1: SFT. Train a base model on curated instruction-following data. This is the warm-up; without it, the policy has no prior for responding usefully to prompts.

Stage 2: Reward model training. Given human-annotated preference pairs (prompt, chosen response, rejected response), train a separate model with a regression head to score how good a response is. This model is typically a smaller version of the policy, stripped of the language model head and fitted with a scalar output.

Stage 3: PPO. The actor (policy) generates responses. The reward model scores them. A critic (a value head or separate network) estimates the expected future reward. PPO computes advantages, clips the policy gradient to prevent updates that move the policy too far from the reference distribution, and updates both the actor and critic. All four models — policy, reference, reward model, critic — need to be active at the same time.

The technical challenges were real: reward model calibration was fragile (a reward model that saturates at high scores kills the signal), KL coefficients needed tuning per task, the critic convergence often lagged behind the actor causing instability, and any GPU memory pressure produced cascading failures. Labs with 1,000 H100s could absorb these costs. Everyone else was doing SFT and calling it a day.

DPO: collapsing the reward model

Direct Preference Optimization (Rafailov et al., 2023) made a key mathematical observation: the optimal policy under the RLHF objective has a closed-form relationship to the reward function and the reference policy. If you rearrange that relationship, you can express the reward in terms of policy log-probabilities — no separate reward network needed.

The DPO loss for a preference pair (prompt $x$, chosen response $y_w$, rejected response $y_l$) is:

L_DPO = -log σ(β * (log π_θ(y_w|x) / π_ref(y_w|x)) - β * (log π_θ(y_l|x) / π_ref(y_l|x)))

Where π_θ is the trainable policy, π_ref is the frozen reference model (the SFT checkpoint), β is a temperature controlling how far the policy can drift from the reference, and σ is the sigmoid function.

In plain terms: the loss pushes the policy to assign more probability to chosen responses relative to the reference, and less to rejected responses relative to the reference. The reference model acts as a baseline so the policy doesn't just learn to always output high-probability tokens regardless of preference.

# Minimal DPO training loop (TRL / Hugging Face)
from datasets import load_dataset
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
ref_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Rows must have: {"prompt": ..., "chosen": ..., "rejected": ...}
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
training_args = DPOConfig(
    beta=0.1,                    # KL penalty; lower = more deviation from reference allowed
    max_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,          # Much smaller than SFT; you are fine-tuning a fine-tune
    num_train_epochs=1,
    bf16=True,
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,         # Frozen — no gradient through this
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,  # TRL renamed the old tokenizer= kwarg
)
trainer.train()

What DPO gives you: the reward model disappears. No separate training run, no regression head, no annotation pipeline that has to match the reward model's architecture. What it keeps: you still need the frozen reference model in VRAM. For a 7B model in bfloat16, that is 14 GB sitting there doing nothing except providing a KL baseline. Not catastrophic, but it shapes the hardware requirements.

The practical catch: DPO requires that SFT came first. A base model (not instruction-tuned) does not have the prior for instruction following that DPO's log-ratio objective assumes. Running DPO directly on a base model produces garbage. This is the most common failure mode teams hit when first implementing it.

{ "type": "rlhf-dpo", "method": "dpo", "title": "RLHF vs DPO vs GRPO: which boxes disappear?" }

SimPO: removing the reference model

SimPO (Meng et al., 2024) asked what the reference model actually does, and found the answer: it normalizes the log-probabilities so that the policy does not trivially favor short responses (which have higher per-token probability by default). The reference model is a normalization device.

SimPO replaces this normalization with the average log-probability of a response — divide the sequence log-prob by sequence length. This is a self-contained signal that does not require a second model. The SimPO loss is:

L_SimPO = -log σ((β/|y_w|) * log π_θ(y_w|x) - (β/|y_l|) * log π_θ(y_l|x) - γ)

Where |y| is the sequence length and γ is a margin term that encourages a minimum gap between chosen and rejected scores. The margin is important: without it, SimPO can still collapse toward very short responses.

The empirical results from the SimPO paper were sharp. On AlpacaEval 2 (a leaderboard for instruction-following quality), SimPO outperformed DPO by 6.4 points. On Arena-Hard (a harder benchmark closer to real user preferences), it was +7.5 points. The mechanism seems to be that the average log-prob implicit reward better captures response quality without the drift that comes from training against a static reference checkpoint.

The memory saving is the side benefit. Without a reference model, a 7B SimPO training run fits on a single 40 GB A100, versus DPO's 80 GB requirement. With QLoRA, the policy drops to ~5 GB total VRAM, putting 7B SimPO training within reach of a high-end consumer GPU.

One nuance: SimPO is more sensitive to the SFT warm-up quality than DPO. Because average log-prob can be gamed by shorter or more confident outputs — which is exactly the failure the margin γ guards against — if your SFT model has uneven calibration across response lengths, SimPO training can amplify that variance. The fix is not complicated — higher-quality SFT data — but it means you cannot skip the SFT step to save time and then expect SimPO to compensate.

ORPO: one training stage instead of two

Both DPO and SimPO still assume a two-stage pipeline: SFT first to establish instruction following, then preference optimization to align style and safety. If you are starting from a raw pre-trained base model, that is two full training jobs.

ORPO (Hong et al., 2024) merges these stages by appending an odds-ratio penalty term directly to the standard SFT cross-entropy loss:

L_ORPO = L_SFT + λ * L_OR

L_SFT = -log π_θ(y_w|x)           # standard cross-entropy on chosen response
L_OR  = -log σ(log(odds_θ(y_w|x) / odds_θ(y_l|x)))

where odds_θ(y|x) = π_θ(y|x) / (1 - π_θ(y|x))

The SFT loss teaches the model to produce the chosen response. The odds-ratio loss penalizes the model for assigning similar or higher probability to the rejected response. Both signals update the weights in one pass, so you do not need a separate SFT checkpoint to initialize DPO from.

The practical benefit is speed: for a team iterating on alignment, eliminating a training stage cuts iteration time roughly in half. The cost is that ORPO requires careful balancing of λ — the weight on the odds-ratio term. Too high and the model overfits to rejecting specific patterns rather than learning to follow instructions well; too low and the preference signal is swamped by SFT loss.

ORPO also does not require a reference model. Like SimPO, everything is self-contained in the single policy. The result is that ORPO's VRAM footprint matches SimPO — only the trainable policy plus optimizer states.

Method	Reference model	Separate SFT stage	Min VRAM (7B, full fine-tune)
PPO (classic RLHF)	Yes	Yes	~58–69 GB
DPO	Yes (frozen)	Yes	~42 GB
SimPO	No	Yes	~28 GB
ORPO	No	No (merged)	~28 GB
KTO	Yes (frozen)	Yes	~42 GB

KTO (Kahneman-Tversky Optimization) deserves a brief mention: it works from binary like/dislike signals on individual responses rather than paired chosen/rejected comparisons. If your annotation budget allows only thumbs-up/thumbs-down per response rather than relative preferences between pairs, KTO is your option. It gives up some quality compared to DPO/SimPO on benchmarks but dramatically reduces annotation cost.

GRPO: preference optimization without human preferences

The methods above all require a dataset of human-labeled preference pairs. That annotation cost — typically $1–5 per comparison — is manageable for general alignment but scales badly when you want to train on reasoning tasks. A math problem's correct answer is either right or wrong; you do not need a human to tell you which of two chain-of-thought traces is "preferred."

Group Relative Policy Optimization (DeepSeek, 2024) exploits this. For each prompt, GRPO samples a group of G responses from the current policy, scores each with a verifiable reward function (an external math grader, a Python execution engine, a test suite), and computes advantages as the normalized reward within the group:

advantage_i = (r_i - mean(r_1...r_G)) / std(r_1...r_G)

No critic network. No value head. The group mean acts as the baseline that a critic would estimate. This sidesteps the main instability of PPO for language models: the value head trying to predict future rewards across sequences of unpredictable length.

The GRPO loss is a clipped policy gradient (same clip mechanism as PPO) on these group-relative advantages, plus a KL penalty against a reference policy to prevent the model from drifting too far:

# Conceptual GRPO training step
for prompt in batch:
    responses = [policy.sample(prompt) for _ in range(G)]   # G = group size, typically 8–16
    rewards = [verifier.score(prompt, r) for r in responses]
    
    advantages = normalize(rewards)  # subtract mean, divide by std
    
    # Standard clipped PPO gradient, but advantages come from the group, not a critic
    loss = clipped_policy_gradient(responses, advantages, old_log_probs, clip_epsilon=0.2)
    loss += beta * kl_penalty(policy, reference_policy, responses)
    
    loss.backward()

GRPO is what DeepSeek used to train DeepSeek-R1. The verifiable reward was a math solver checking final answers — binary, programmatic, no human annotators needed at training time. The result was a model matching OpenAI o1 on reasoning benchmarks. Qwen3 used the same recipe. This pattern — RLVR (Reinforcement Learning with Verifiable Rewards) — is now the standard approach for reasoning training outside of research labs.

sequenceDiagram
    participant P as Policy (training)
    participant V as Verifier (math/code)
    participant G as GRPO loss

    Note over P: For each prompt
    P->>P: Sample G=8 responses
    P->>V: Score all 8 responses
    V-->>P: rewards [0, 1, 0, 1, 1, 0, 1, 0]
    P->>G: Compute group-normalized advantages
    G-->>P: Gradient update (no critic needed)
    Note over P: No reward model, no value head

One failure mode specific to GRPO: length reward hacking. Vanilla GRPO averages the loss per sample, so a 2,000-token wrong answer and a 50-token wrong answer contribute the same total penalty — which means each token of the long one is penalized far more weakly. Long incorrect chains of thought are systematically under-punished, and response length drifts upward without accuracy improving (this is the bias the DAPO and Dr. GRPO analyses pinned down, and why DAPO switches to token-level loss). The model learns to be verbose rather than accurate. The fix, now standard, is to add a per-token length penalty to the reward: r_effective = r_verifiable - α * len(response). Getting α right takes a few runs, but omitting it entirely produces models that output thousand-token chains of thought for arithmetic that should take two lines.

{ "type": "rlhf-dpo", "method": "grpo", "title": "Training complexity: from PPO to GRPO" }

What breaks

DPO on an un-instructed base model. DPO's log-ratio objective assumes the policy already generates plausible instruction-following responses that can be meaningfully ranked. A raw base model's chosen/rejected log-probabilities are near-random, and the loss gradient points in useless directions. Always SFT first. Even 1,000 high-quality instruction examples are enough — you do not need the full SFT dataset before running DPO.

DPO on off-policy pairs. DPO's gradient only teaches the policy about responses it might plausibly generate. Train on a preference dataset sampled from a different model — the typical off-the-shelf dataset — and the chosen/rejected pairs sit in a region of output space your policy rarely visits, so wins on the training loss stop translating into wins on evals. The symptom is a run that looks healthy (loss falls, reward margins widen) while AlpacaEval-style scores plateau after the first fraction of an epoch. On-policy pairs — responses sampled from your own SFT checkpoint, then labeled — reliably beat larger volumes of borrowed data.

SimPO length collapse. If your SFT checkpoint has calibration issues — different output lengths for different prompt types — SimPO's average log-prob reward can collapse toward very short responses for some topic domains. Monitor response length distribution by category during training. The margin parameter γ helps, but does not fully compensate for an uncalibrated SFT model.

GRPO without a length penalty. Already discussed above. The DAPO paper (2025) extended GRPO with clip-higher and token-level loss policies specifically to address reward hacking in long chain-of-thought training. For reasoning tasks, consider DAPO as a drop-in improvement over vanilla GRPO.

Over-rejecting with ORPO. If λ (the odds-ratio weight) is too high, the model learns to avoid the rejected response patterns too aggressively and degrades instruction following on topics not covered in the preference data. Keep λ in the 0.1–1.0 range and monitor instruction-following benchmarks alongside preference metrics.

KL collapse in all methods. Every preference method has a temperature or KL penalty parameter (β in DPO/SimPO, β KL term in GRPO). If this is set too low, the policy drifts far from the SFT distribution and catastrophically forgets instruction-following behavior. If it is set too high, the preference signal is too small to move the policy at all. The typical symptom of a KL penalty that is too low: the model starts output length increasing, outputs degenerate toward high-probability filler tokens, and eval scores plateau or drop after epoch 1. Treat β as the primary hyperparameter to tune before anything else.

flowchart TD
    FAIL["Alignment degradation"] --> Q1{"DPO only?"}
    Q1 -->|Yes| A1["SFT warm-up missing?\nBase model input to DPO"]
    Q1 -->|No| Q2{"Length increasing / verbose outputs?"}
    Q2 -->|Yes| A2["KL penalty too low (β)\nOR GRPO missing length penalty"]
    Q2 -->|No| Q3{"Poor on out-of-distribution prompts?"}
    Q3 -->|Yes| A3["Preference data coverage narrow\nOR ORPO λ too high → over-rejection"]
    Q3 -->|No| A4["Eval distribution mismatch\nCheck if benchmark is in training data"]

The decision in practice

The choice between methods comes down to three variables: what hardware you have, where your data comes from, and whether your task has a verifiable reward signal.

Start with DPO if you are using a popular SFT checkpoint (Llama 3.1, Qwen 2.5, Mistral), have standard human preference pairs, and want the widest ecosystem support. TRL's DPOTrainer is production-grade. The reference model fits on most multi-GPU setups, and DPO is well-understood enough that you can debug it without expert help.

Switch to SimPO if you are constrained to a single 40 GB A100 (or want to avoid the reference model VRAM cost), and your evaluation is instruction-following or open-ended chat quality. SimPO's benchmark numbers are consistently better than DPO on these dimensions. The SFT quality requirement is higher, but if your SFT data is reasonable, SimPO is a clear upgrade. TRL added CPOTrainer (which implements SimPO as a contrastive variant) in 2024.

Use ORPO when you are starting from a raw base model and want to combine SFT and alignment in one run. It is also a good default for rapid iteration when you do not yet know whether your preference data is high-quality — the merged loss is somewhat more forgiving of noisy annotations than the standalone DPO loss, because the SFT signal keeps the model anchored.

Use GRPO/RLVR for math, code, or any task where you can programmatically verify correctness. This is not optional for reasoning tasks — preference-based methods cannot efficiently teach chain-of-thought reasoning because they require humans to label which trace is "better," and humans are poor judges of intermediate reasoning steps. A math grader is not. You need a slightly larger GPU budget (the group sampling requires generating 8–16 responses per prompt instead of 1), but the annotation cost drops to near zero.

flowchart TD
    START["New alignment job"] --> Q1{"Task has verifiable\nground truth?"}
    Q1 -->|"Yes (math, code, logic)"| GRPO["GRPO + RLVR\nno human pairs needed"]
    Q1 -->|No| Q2{"Starting from raw\nbase model?"}
    Q2 -->|Yes| ORPO["ORPO\none-stage SFT + alignment"]
    Q2 -->|No, SFT done| Q3{"VRAM constrained\n(≤40 GB A100)?"}
    Q3 -->|Yes| SIMPO["SimPO\nno reference model"]
    Q3 -->|No| DPO["DPO\nwidest ecosystem support"]

Whatever method you pick, plan for iterative rounds rather than one pass over a static dataset. The production recipes that work — Llama 3's post-training is the documented example — run DPO in multiple rounds: sample responses from the current policy, label the pairs (human annotators or an LLM judge), train, then repeat from the new checkpoint. Two or three rounds of this online loop routinely beat a single pass over ten times as much off-policy data, because each round's pairs reflect what the model actually says now, not what some other model said a year ago. Budget for the sampling and labeling infrastructure up front; it is where most of the real cost of preference optimization lives.

If you are combining these methods — which the research literature and the DeepSeek recipe both suggest is better than any single method alone — the sequence is:

1. SFT on high-quality instruction data (see SFT and instruction tuning for data quality guidance).
2. DPO or SimPO for general alignment, helpfulness, and safety preferences.
3. GRPO/RLVR for domain-specific reasoning capability if your task warrants it.

For the fastest iteration on a resource-constrained setup, ORPO replaces steps 1–2 with a single training run, then you layer GRPO on top for reasoning.

One thing to be clear-eyed about: preference optimization does not fix factual hallucination. A model trained on DPO with human preference pairs learns what humans prefer in responses — and humans cannot always distinguish fluent incorrect answers from fluent correct ones. If your core problem is factual accuracy in a specific domain, preference optimization will make outputs more polished and less repetitive, but it will not eliminate hallucination. That requires retrieval-augmented generation or SFT on verified domain data. Preference methods shape style and behavior; grounding shapes facts.

The ecosystem here — TRL, Axolotl, LLaMA-Factory, unsloth — has caught up with the research papers fast enough that you can implement any of these methods with 50-100 lines of config today. The choice of method matters less than it did in 2023. What still matters: the quality of your preference data, whether your SFT warm-up was any good, and whether you are measuring the right things. Those three are not solved by the choice of loss function. For the data quality angle, the synthetic data engineering article covers how to generate high-quality preference pairs when human annotation is not an option. For the reasoning training angle, the RLHF, reward models, and constitutional AI article covers the reward model design and the preference data collection side. And for the training efficiency question — whether you should be fine-tuning all weights or just adapters — see the LoRA and PEFT article, since every method here runs on top of either full fine-tuning or LoRA, and the adapter choice affects memory math significantly.