# Preference Optimization: DPO, SimPO, ORPO, and the Post-RLHF Stack > How DPO, SimPO, ORPO, and GRPO replaced the classic RLHF pipeline — what each method removes, when to use which, and the 2025 reasoning-alignment frontier. Canonical: https://ironclad.academy/ai-engineering/articles/preference-optimization-dpo-simpo-orpo-grpo | Difficulty: medium | Topics: Fine-Tuning, Post-Training, LLM Internals, Reasoning Models Covers: Meta, DeepSeek, Hugging Face ## Key takeaways - DPO collapses the RLHF reward model into the policy update via a log-ratio loss, cutting four-model PPO complexity to a two-model fine-tune with comparable quality. - SimPO drops the reference model entirely and outperforms DPO by +6.4 on AlpacaEval 2 and +7.5 on Arena-Hard, at lower VRAM cost. - ORPO merges SFT and preference tuning into one training pass via an odds-ratio penalty, eliminating the sequential SFT-then-DPO pipeline. - GRPO + verifiable rewards (RLVR) is how DeepSeek-R1 and Qwen3 achieved o1-level reasoning without a critic network or human preference labels at training time. - Skipping SFT before DPO is a reliable way to break alignment: the policy needs a warm instruction-following prior before the preference loss can shape it. - The practical 2025 post-training stack is SFT → DPO/SimPO for alignment → GRPO/RLVR for reasoning, with ORPO as a shortcut when starting from a raw base model. > **In one line:** DPO, SimPO, ORPO, and GRPO are a family of methods that progressively stripped components out of the classic RLHF-PPO pipeline — cutting from four models to two to one — while matching or exceeding the alignment quality of the full stack. **The idea.** Classic RLHF kept four models active simultaneously: a policy (the model you are training), a frozen reference copy (the KL anchor), a reward model (trained on human preference pairs), and a critic (a value head that estimates future reward for PPO). Getting this to train stably took significant engineering effort and compute. The post-2023 insight was that you can derive the reward model's signal directly from the policy log-probabilities — no separate reward network needed. DPO made this move in 2023 and became the default alignment step. SimPO (2024) went further and dropped the reference model. ORPO merged the SFT and alignment passes. GRPO replaced the PPO critic with group-relative advantages, enabling stable RL on verifiable rewards without a value head. **Key points.** - Classic RLHF (PPO): SFT model → reward model (trained separately) → PPO actor-critic loop. Four models in VRAM (policy, reference, reward model, critic), complex to stabilize. - DPO: one policy + one frozen reference model. Log-ratio loss maximizes the gap between chosen and rejected log-probs. No reward model, no critic. - SimPO: one policy, no reference model. Average per-token log-prob as implicit reward. Simpler memory footprint, stronger empirical results on open-ended benchmarks. - ORPO: single-stage training. SFT cross-entropy + odds-ratio penalty on rejected responses. No separate SFT pass, no reference model. - GRPO: group sampling + normalized rewards from a verifiable signal (code executor, math grader). No critic network. The backbone of DeepSeek-R1 and Qwen3 reasoning training. **Back-of-the-envelope: GPU footprint per method for a 7B model in bfloat16 (illustrative; assumes 8-bit Adam for optimizer states).** ``` Full RLHF (PPO): Policy (7B × 2 bytes) = 14 GB Policy optimizer (8-bit Adam) = ~14 GB (2 bytes/param) Reference model (frozen) = 14 GB Reward model (~1–3B) = 2–6 GB Critic + its optimizer = ~14–21 GB Total active ≈ 58–69 GB → 2× 40 GB A100s minimum DPO: Policy weights (bf16) = 14 GB 8-bit Adam optimizer states = ~14 GB (2 bytes/param) Reference model (frozen, bf16) = 14 GB Total active ≈ 42 GB → fits 1× 80 GB A100 (Standard fp32 Adam adds ~56 GB for optimizer states alone → multi-GPU) SimPO / ORPO: Policy weights + 8-bit optimizer = ~28 GB No reference model needed Total active ≈ 28 GB → fits 1× 40 GB A100 (With QLoRA: 7B base at 4-bit ≈ 3.5 GB + adapters ≈ 1 GB → single RTX 4090) ``` **The pipeline in one diagram.** ```mermaid flowchart LR HD["Human preference\npairs (chosen/rejected)"] --> DPO["DPO / SimPO / ORPO\npreference loss"] SFT["SFT checkpoint"] --> DPO DPO --> ALIGNED["Aligned model"] VR["Verifiable rewards\n(math grader / code runner)"] --> GRPO["GRPO / RLVR\ngroup-relative RL"] ALIGNED --> GRPO GRPO --> REASONING["Reasoning model\n(e.g. DeepSeek-R1)"] ``` **Key design decisions.** | Decision | Options | When to pick which | | --- | --- | --- | | Need reference model? | DPO (yes) vs SimPO/ORPO (no) | Skip reference model if VRAM is tight or you want simpler setup | | SFT before alignment? | Separate SFT pass (DPO/SimPO) vs merged (ORPO) | Use ORPO when starting from a raw base model to save a training stage | | Reward signal | Human pairs (DPO/SimPO/ORPO) vs verifiable (GRPO/RLVR) | GRPO only when you have ground-truth verifiable answers (math, code) | | RL algorithm | PPO (classic) vs GRPO (no critic) | GRPO for reasoning; PPO rarely justified anymore given stability/cost | | Data format | Paired (chosen + rejected) vs unpaired (KTO) | KTO when you only have binary feedback, no paired comparisons | **If you have 60 seconds, say this.** "Classic RLHF needs four models and a PPO loop, which is expensive and fragile to tune. DPO fixed this by reformulating the reward as a log-ratio between policy and a frozen reference model — you get alignment in one fine-tuning pass. SimPO went further and dropped the reference model, using average log-prob as an implicit reward, and empirically beats DPO on instruction-following benchmarks. ORPO merges the SFT pass into the alignment loss, so one training job handles both. For reasoning tasks where you have a verifiable reward — a math grader, a code executor — GRPO samples groups of responses, computes normalized advantages without a critic network, and does RL without human preference pairs at all. In 2025, the default production recipe is SFT first, then DPO or SimPO for alignment, then optionally GRPO for reasoning tasks." In late 2022, when RLHF-tuned models had shipped from exactly one lab, the standard joke inside every other ML team was that PPO for language models was less "reinforcement learning" and more "reinforcement anxiety" — you kept four models in sync, tuned KL penalties to stop the policy from collapsing, trained a reward model on annotation data that you hoped wasn't biased, and crossed your fingers that the critic network converged before you ran out of compute budget. The Anthropic Constitutional AI paper and InstructGPT paper made it look tractable at the lab scale. At the startup scale, it mostly looked like a six-week project that ended in "we went back to SFT." DPO changed that in 2023. Then SimPO, ORPO, and GRPO reshaped what post-training looks like in 2024–2025. This article covers how each works mechanically, what each one removes from the stack, and the decisions you actually face when picking one for a real training job. ## What RLHF-PPO actually required Before you can appreciate what was removed, it helps to be specific about what the classic setup contained. **Stage 1: SFT.** Train a base model on curated instruction-following data. This is the warm-up; without it, the policy has no prior for responding usefully to prompts. **Stage 2: Reward model training.** Given human-annotated preference pairs (prompt, chosen response, rejected response), train a separate model with a regression head to score how good a response is. This model is typically a smaller version of the policy, stripped of the language model head and fitted with a scalar output. **Stage 3: PPO.** The actor (policy) generates responses. The reward model scores them. A critic (a value head or separate network) estimates the expected future reward. PPO computes advantages, clips the policy gradient to prevent updates that move the policy too far from the reference distribution, and updates both the actor and critic. All four models — policy, reference, reward model, critic — need to be active at the same time. The technical challenges were real: reward model calibration was fragile (a reward model that saturates at high scores kills the signal), KL coefficients needed tuning per task, the critic convergence often lagged behind the actor causing instability, and any GPU memory pressure produced cascading failures. Labs with 1,000 H100s could absorb these costs. Everyone else was doing SFT and calling it a day. ## DPO: collapsing the reward model **Direct Preference Optimization** (Rafailov et al., 2023) made a key mathematical observation: the optimal policy under the RLHF objective has a closed-form relationship to the reward function and the reference policy. If you rearrange that relationship, you can express the reward in terms of policy log-probabilities — no separate reward network needed. The DPO loss for a preference pair (prompt $x$, chosen response $y_w$, rejected response $y_l$) is: ``` L_DPO = -log σ(β * (log π_θ(y_w|x) / π_ref(y_w|x)) - β * (log π_θ(y_l|x) / π_ref(y_l|x))) ``` Where `π_θ` is the trainable policy, `π_ref` is the frozen reference model (the SFT checkpoint), `β` is a temperature controlling how far the policy can drift from the reference, and `σ` is the sigmoid function. In plain terms: the loss pushes the policy to assign more probability to chosen responses relative to the reference, and less to rejected responses relative to the reference. The reference model acts as a baseline so the policy doesn't just learn to always output high-probability tokens regardless of preference. ```python # Minimal DPO training loop (TRL / Hugging Face) from datasets import load_dataset from trl import DPOTrainer, DPOConfig from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct") ref_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct") tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct") # Rows must have: {"prompt": ..., "chosen": ..., "rejected": ...} dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train") training_args = DPOConfig( beta=0.1, # KL penalty; lower = more deviation from reference allowed max_length=2048, per_device_train_batch_size=4, gradient_accumulation_steps=8, learning_rate=5e-7, # Much smaller than SFT; you are fine-tuning a fine-tune num_train_epochs=1, bf16=True, ) trainer = DPOTrainer( model=model, ref_model=ref_model, # Frozen — no gradient through this args=training_args, train_dataset=dataset, processing_class=tokenizer, # TRL renamed the old tokenizer= kwarg ) trainer.train() ``` What DPO gives you: the reward model disappears. No separate training run, no regression head, no annotation pipeline that has to match the reward model's architecture. What it keeps: you still need the frozen reference model in VRAM. For a 7B model in bfloat16, that is 14 GB sitting there doing nothing except providing a KL baseline. Not catastrophic, but it shapes the hardware requirements. The practical catch: DPO requires that SFT came first. A base model (not instruction-tuned) does not have the prior for instruction following that DPO's log-ratio objective assumes. Running DPO directly on a base model produces garbage. This is the most common failure mode teams hit when first implementing it. [Interactive visualizer on the original page: rlhf-dpo — RLHF vs DPO vs GRPO: which boxes disappear?] ## SimPO: removing the reference model SimPO (Meng et al., 2024) asked what the reference model actually does, and found the answer: it normalizes the log-probabilities so that the policy does not trivially favor short responses (which have higher per-token probability by default). The reference model is a normalization device. SimPO replaces this normalization with the **average log-probability** of a response — divide the sequence log-prob by sequence length. This is a self-contained signal that does not require a second model. The SimPO loss is: ``` L_SimPO = -log σ((β/|y_w|) * log π_θ(y_w|x) - (β/|y_l|) * log π_θ(y_l|x) - γ) ``` Where `|y|` is the sequence length and `γ` is a margin term that encourages a minimum gap between chosen and rejected scores. The margin is important: without it, SimPO can still collapse toward very short responses. The empirical results from the SimPO paper were sharp. On AlpacaEval 2 (a leaderboard for instruction-following quality), SimPO outperformed DPO by 6.4 points. On Arena-Hard (a harder benchmark closer to real user preferences), it was +7.5 points. The mechanism seems to be that the average log-prob implicit reward better captures response quality without the drift that comes from training against a static reference checkpoint. The memory saving is the side benefit. Without a reference model, a 7B SimPO training run fits on a single 40 GB A100, versus DPO's 80 GB requirement. With QLoRA, the policy drops to ~5 GB total VRAM, putting 7B SimPO training within reach of a high-end consumer GPU. One nuance: SimPO is more sensitive to the SFT warm-up quality than DPO. Because average log-prob can be gamed by shorter or more confident outputs — which is exactly the failure the margin `γ` guards against — if your SFT model has uneven calibration across response lengths, SimPO training can amplify that variance. The fix is not complicated — higher-quality SFT data — but it means you cannot skip the SFT step to save time and then expect SimPO to compensate. ## ORPO: one training stage instead of two Both DPO and SimPO still assume a two-stage pipeline: SFT first to establish instruction following, then preference optimization to align style and safety. If you are starting from a raw pre-trained base model, that is two full training jobs. **ORPO** (Hong et al., 2024) merges these stages by appending an odds-ratio penalty term directly to the standard SFT cross-entropy loss: ``` L_ORPO = L_SFT + λ * L_OR L_SFT = -log π_θ(y_w|x) # standard cross-entropy on chosen response L_OR = -log σ(log(odds_θ(y_w|x) / odds_θ(y_l|x))) where odds_θ(y|x) = π_θ(y|x) / (1 - π_θ(y|x)) ``` The SFT loss teaches the model to produce the chosen response. The odds-ratio loss penalizes the model for assigning similar or higher probability to the rejected response. Both signals update the weights in one pass, so you do not need a separate SFT checkpoint to initialize DPO from. The practical benefit is speed: for a team iterating on alignment, eliminating a training stage cuts iteration time roughly in half. The cost is that ORPO requires careful balancing of `λ` — the weight on the odds-ratio term. Too high and the model overfits to rejecting specific patterns rather than learning to follow instructions well; too low and the preference signal is swamped by SFT loss. ORPO also does not require a reference model. Like SimPO, everything is self-contained in the single policy. The result is that ORPO's VRAM footprint matches SimPO — only the trainable policy plus optimizer states. | Method | Reference model | Separate SFT stage | Min VRAM (7B, full fine-tune) | | --- | --- | --- | --- | | PPO (classic RLHF) | Yes | Yes | ~58–69 GB | | DPO | Yes (frozen) | Yes | ~42 GB | | SimPO | No | Yes | ~28 GB | | ORPO | No | No (merged) | ~28 GB | | KTO | Yes (frozen) | Yes | ~42 GB | KTO (Kahneman-Tversky Optimization) deserves a brief mention: it works from binary like/dislike signals on individual responses rather than paired chosen/rejected comparisons. If your annotation budget allows only thumbs-up/thumbs-down per response rather than relative preferences between pairs, KTO is your option. It gives up some quality compared to DPO/SimPO on benchmarks but dramatically reduces annotation cost. ## GRPO: preference optimization without human preferences The methods above all require a dataset of human-labeled preference pairs. That annotation cost — typically $1–5 per comparison — is manageable for general alignment but scales badly when you want to train on reasoning tasks. A math problem's correct answer is either right or wrong; you do not need a human to tell you which of two chain-of-thought traces is "preferred." **Group Relative Policy Optimization** (DeepSeek, 2024) exploits this. For each prompt, GRPO samples a group of G responses from the current policy, scores each with a **verifiable reward function** (an external math grader, a Python execution engine, a test suite), and computes advantages as the normalized reward within the group: ``` advantage_i = (r_i - mean(r_1...r_G)) / std(r_1...r_G) ``` No critic network. No value head. The group mean acts as the baseline that a critic would estimate. This sidesteps the main instability of PPO for language models: the value head trying to predict future rewards across sequences of unpredictable length. The GRPO loss is a clipped policy gradient (same clip mechanism as PPO) on these group-relative advantages, plus a KL penalty against a reference policy to prevent the model from drifting too far: ```python # Conceptual GRPO training step for prompt in batch: responses = [policy.sample(prompt) for _ in range(G)] # G = group size, typically 8–16 rewards = [verifier.score(prompt, r) for r in responses] advantages = normalize(rewards) # subtract mean, divide by std # Standard clipped PPO gradient, but advantages come from the group, not a critic loss = clipped_policy_gradient(responses, advantages, old_log_probs, clip_epsilon=0.2) loss += beta * kl_penalty(policy, reference_policy, responses) loss.backward() ``` GRPO is what DeepSeek used to train DeepSeek-R1. The verifiable reward was a math solver checking final answers — binary, programmatic, no human annotators needed at training time. The result was a model matching OpenAI o1 on reasoning benchmarks. Qwen3 used the same recipe. This pattern — **RLVR (Reinforcement Learning with Verifiable Rewards)** — is now the standard approach for reasoning training outside of research labs. ```mermaid sequenceDiagram participant P as Policy (training) participant V as Verifier (math/code) participant G as GRPO loss Note over P: For each prompt P->>P: Sample G=8 responses P->>V: Score all 8 responses V-->>P: rewards [0, 1, 0, 1, 1, 0, 1, 0] P->>G: Compute group-normalized advantages G-->>P: Gradient update (no critic needed) Note over P: No reward model, no value head ``` One failure mode specific to GRPO: **length reward hacking**. Vanilla GRPO averages the loss per sample, so a 2,000-token wrong answer and a 50-token wrong answer contribute the same total penalty — which means each token of the long one is penalized far more weakly. Long incorrect chains of thought are systematically under-punished, and response length drifts upward without accuracy improving (this is the bias the DAPO and Dr. GRPO analyses pinned down, and why DAPO switches to token-level loss). The model learns to be verbose rather than accurate. The fix, now standard, is to add a per-token length penalty to the reward: `r_effective = r_verifiable - α * len(response)`. Getting `α` right takes a few runs, but omitting it entirely produces models that output thousand-token chains of thought for arithmetic that should take two lines. [Interactive visualizer on the original page: rlhf-dpo — Training complexity: from PPO to GRPO] ## What breaks **DPO on an un-instructed base model.** DPO's log-ratio objective assumes the policy already generates plausible instruction-following responses that can be meaningfully ranked. A raw base model's chosen/rejected log-probabilities are near-random, and the loss gradient points in useless directions. Always SFT first. Even 1,000 high-quality instruction examples are enough — you do not need the full SFT dataset before running DPO. **DPO on off-policy pairs.** DPO's gradient only teaches the policy about responses it might plausibly generate. Train on a preference dataset sampled from a different model — the typical off-the-shelf dataset — and the chosen/rejected pairs sit in a region of output space your policy rarely visits, so wins on the training loss stop translating into wins on evals. The symptom is a run that looks healthy (loss falls, reward margins widen) while AlpacaEval-style scores plateau after the first fraction of an epoch. On-policy pairs — responses sampled from your own SFT checkpoint, then labeled — reliably beat larger volumes of borrowed data. **SimPO length collapse.** If your SFT checkpoint has calibration issues — different output lengths for different prompt types — SimPO's average log-prob reward can collapse toward very short responses for some topic domains. Monitor response length distribution by category during training. The margin parameter `γ` helps, but does not fully compensate for an uncalibrated SFT model. **GRPO without a length penalty.** Already discussed above. The DAPO paper (2025) extended GRPO with clip-higher and token-level loss policies specifically to address reward hacking in long chain-of-thought training. For reasoning tasks, consider DAPO as a drop-in improvement over vanilla GRPO. **Over-rejecting with ORPO.** If `λ` (the odds-ratio weight) is too high, the model learns to avoid the rejected response patterns too aggressively and degrades instruction following on topics not covered in the preference data. Keep `λ` in the 0.1–1.0 range and monitor instruction-following benchmarks alongside preference metrics. **KL collapse in all methods.** Every preference method has a temperature or KL penalty parameter (`β` in DPO/SimPO, `β` KL term in GRPO). If this is set too low, the policy drifts far from the SFT distribution and catastrophically forgets instruction-following behavior. If it is set too high, the preference signal is too small to move the policy at all. The typical symptom of a KL penalty that is too low: the model starts output length increasing, outputs degenerate toward high-probability filler tokens, and eval scores plateau or drop after epoch 1. Treat `β` as the primary hyperparameter to tune before anything else. ```mermaid flowchart TD FAIL["Alignment degradation"] --> Q1{"DPO only?"} Q1 -->|Yes| A1["SFT warm-up missing?\nBase model input to DPO"] Q1 -->|No| Q2{"Length increasing / verbose outputs?"} Q2 -->|Yes| A2["KL penalty too low (β)\nOR GRPO missing length penalty"] Q2 -->|No| Q3{"Poor on out-of-distribution prompts?"} Q3 -->|Yes| A3["Preference data coverage narrow\nOR ORPO λ too high → over-rejection"] Q3 -->|No| A4["Eval distribution mismatch\nCheck if benchmark is in training data"] ``` ## The decision in practice The choice between methods comes down to three variables: what hardware you have, where your data comes from, and whether your task has a verifiable reward signal. **Start with DPO** if you are using a popular SFT checkpoint (Llama 3.1, Qwen 2.5, Mistral), have standard human preference pairs, and want the widest ecosystem support. TRL's `DPOTrainer` is production-grade. The reference model fits on most multi-GPU setups, and DPO is well-understood enough that you can debug it without expert help. **Switch to SimPO** if you are constrained to a single 40 GB A100 (or want to avoid the reference model VRAM cost), and your evaluation is instruction-following or open-ended chat quality. SimPO's benchmark numbers are consistently better than DPO on these dimensions. The SFT quality requirement is higher, but if your SFT data is reasonable, SimPO is a clear upgrade. TRL added `CPOTrainer` (which implements SimPO as a contrastive variant) in 2024. **Use ORPO** when you are starting from a raw base model and want to combine SFT and alignment in one run. It is also a good default for rapid iteration when you do not yet know whether your preference data is high-quality — the merged loss is somewhat more forgiving of noisy annotations than the standalone DPO loss, because the SFT signal keeps the model anchored. **Use GRPO/RLVR** for math, code, or any task where you can programmatically verify correctness. This is not optional for reasoning tasks — preference-based methods cannot efficiently teach chain-of-thought reasoning because they require humans to label which trace is "better," and humans are poor judges of intermediate reasoning steps. A math grader is not. You need a slightly larger GPU budget (the group sampling requires generating 8–16 responses per prompt instead of 1), but the annotation cost drops to near zero. ```mermaid flowchart TD START["New alignment job"] --> Q1{"Task has verifiable\nground truth?"} Q1 -->|"Yes (math, code, logic)"| GRPO["GRPO + RLVR\nno human pairs needed"] Q1 -->|No| Q2{"Starting from raw\nbase model?"} Q2 -->|Yes| ORPO["ORPO\none-stage SFT + alignment"] Q2 -->|No, SFT done| Q3{"VRAM constrained\n(≤40 GB A100)?"} Q3 -->|Yes| SIMPO["SimPO\nno reference model"] Q3 -->|No| DPO["DPO\nwidest ecosystem support"] ``` Whatever method you pick, plan for iterative rounds rather than one pass over a static dataset. The production recipes that work — Llama 3's post-training is the documented example — run DPO in multiple rounds: sample responses from the current policy, label the pairs (human annotators or an LLM judge), train, then repeat from the new checkpoint. Two or three rounds of this online loop routinely beat a single pass over ten times as much off-policy data, because each round's pairs reflect what the model actually says now, not what some other model said a year ago. Budget for the sampling and labeling infrastructure up front; it is where most of the real cost of preference optimization lives. If you are combining these methods — which the research literature and the DeepSeek recipe both suggest is better than any single method alone — the sequence is: 1. SFT on high-quality instruction data (see [SFT and instruction tuning](/ai-engineering/articles/sft-instruction-tuning-data-quality) for data quality guidance). 2. DPO or SimPO for general alignment, helpfulness, and safety preferences. 3. GRPO/RLVR for domain-specific reasoning capability if your task warrants it. For the fastest iteration on a resource-constrained setup, ORPO replaces steps 1–2 with a single training run, then you layer GRPO on top for reasoning. One thing to be clear-eyed about: preference optimization does not fix factual hallucination. A model trained on DPO with human preference pairs learns what humans prefer in responses — and humans cannot always distinguish fluent incorrect answers from fluent correct ones. If your core problem is factual accuracy in a specific domain, preference optimization will make outputs more polished and less repetitive, but it will not eliminate hallucination. That requires [retrieval-augmented generation](/ai-engineering/articles/rag-from-scratch-ingestion-retrieval-generation) or [SFT on verified domain data](/ai-engineering/articles/sft-instruction-tuning-data-quality). Preference methods shape _style and behavior_; grounding shapes _facts_. The ecosystem here — TRL, Axolotl, LLaMA-Factory, unsloth — has caught up with the research papers fast enough that you can implement any of these methods with 50-100 lines of config today. The choice of method matters less than it did in 2023. What still matters: the quality of your preference data, whether your SFT warm-up was any good, and whether you are measuring the right things. Those three are not solved by the choice of loss function. For the data quality angle, the [synthetic data engineering article](/ai-engineering/articles/synthetic-data-engineering-llm-fine-tuning) covers how to generate high-quality preference pairs when human annotation is not an option. For the reasoning training angle, the [RLHF, reward models, and constitutional AI article](/ai-engineering/articles/rlhf-reward-models-constitutional-ai) covers the reward model design and the preference data collection side. And for the training efficiency question — whether you should be fine-tuning all weights or just adapters — see the [LoRA and PEFT article](/ai-engineering/articles/lora-qlora-peft-parameter-efficient-fine-tuning), since every method here runs on top of either full fine-tuning or LoRA, and the adapter choice affects memory math significantly. ## Frequently asked questions Q: What is DPO and how does it differ from RLHF? A: Direct Preference Optimization (DPO) reformulates the RLHF objective so that the reward model is implicit in the policy itself. Instead of training a separate reward model and running PPO, DPO computes a log-ratio loss between how likely the policy is to produce a chosen response versus a rejected one, using a frozen reference model as a baseline. This eliminates two of the four models PPO keeps active (reward model and critic), makes training more stable, and produces alignment quality comparable to PPO on most benchmarks, at a fraction of the compute cost. Q: When should I use SimPO instead of DPO? A: SimPO (Simple Preference Optimization) drops the reference model entirely and uses the average per-token log-probability as an implicit reward. It has outperformed DPO on AlpacaEval 2 (+6.4 points) and Arena-Hard (+7.5 points) in published evaluations. Use SimPO when you want a simpler training setup without a frozen reference model in VRAM, and when your evaluation favors open-ended instruction following. The main caveat is that SimPO requires a stronger SFT warm-up to avoid length collapse, since the average log-prob reward can reward brevity. Q: What is ORPO and when does it make sense over DPO? A: ORPO (Odds Ratio Preference Optimization) merges the supervised fine-tuning loss and preference loss into a single training stage by adding an odds-ratio penalty on rejected responses on top of the standard cross-entropy SFT loss. This eliminates the need for a separate SFT pass before preference tuning, which saves both compute and the operational overhead of staging two sequential training jobs. ORPO is a good default when you are starting from a raw pre-trained base model (not yet instruction-tuned) and want the fastest path to a deployable aligned model. Q: What is GRPO and why did DeepSeek use it instead of PPO? A: Group Relative Policy Optimization (GRPO) samples a group of responses per prompt, scores them with a verifiable reward function (math grader, code executor), and computes advantages within the group rather than using a separately trained critic network. Removing the critic cuts GPU memory requirements roughly in half compared to PPO and sidesteps the instability that comes from training a value head jointly with a language model policy. DeepSeek used GRPO with verifiable rewards (RLVR) to train DeepSeek-R1, which matched o1-level reasoning without relying on human preference labels at training time. Q: Do I need a reference model for DPO training? A: Classic DPO requires a frozen reference model (typically the SFT checkpoint) to compute the KL-regularization baseline. You keep it in VRAM but do not update it. With 8-bit or 4-bit quantization you can run reference + policy on a single 80 GB A100 for a 7B model. SimPO and ORPO eliminate the reference model requirement entirely, freeing roughly 14–28 GB for a 7B model in bfloat16, which matters if you are training on consumer-grade hardware. Q: What is the right order of post-training steps in 2025? A: The dominant production recipe is: (1) supervised fine-tuning (SFT) on high-quality instruction data to establish instruction-following behavior; (2) DPO or SimPO for general alignment, helpfulness, and safety; (3) GRPO or RLVR for tasks with verifiable ground truth, such as math reasoning or code generation. ORPO can replace steps 1+2 if you are base-model training. Skipping SFT before DPO is a known failure mode: DPO diverges on base models that have not been instruction-tuned first.