How reasoning models are trained: RLVR, GRPO, and the post-training stack

In January 2025, DeepSeek published a technical report with a finding that stopped a lot of people mid-sentence: they had trained a model to match o1's performance on math and coding benchmarks using reinforcement learning, no human-labeled chain-of-thought, and an algorithm that fits on a single whiteboard. The model — R1-Zero — had spontaneously learned to check its own work, backtrack when stuck, and generate extended reasoning traces. Nobody had told it to do any of that. The training signal was just: did you get the right answer?

That result made concrete what OpenAI's o1 announcement had implied — "large-scale reinforcement learning," algorithm undisclosed: the path to reasoning capability is not better supervised data, it's better training incentives. Understanding that shift — the mechanics of RLVR, the specifics of GRPO, and what actually goes wrong — is now table stakes for anyone building on or evaluating reasoning models.

Why RLHF breaks for reasoning

The 2022-2023 post-training recipe was, roughly: (1) fine-tune on instruction data, (2) collect human preferences between pairs of outputs, (3) train a reward model on those preferences, (4) run PPO with the reward model as the signal. That recipe produced ChatGPT and the first wave of instruction-tuned models. It still produces good results for helpfulness, safety, and stylistic alignment.

It has two failure modes that become acute for reasoning tasks.

Reward hacking. A learned reward model is a neural network, and neural networks have blind spots. During PPO, the policy learns to generate outputs the reward model scores highly — not outputs that are actually correct. On math problems, models trained with learned reward models learn to produce outputs that look like correct solutions (well-formatted, confident, with appropriate notation) even when the answer is wrong. The reward model can't reliably verify multi-step proofs; neither can most human labelers given 30 seconds per example.

Label acquisition cost. To label which of two solutions to a multi-step integration problem is better, you need someone who can do multi-step integration. At the scale of millions of training examples, that's either very expensive or very slow. Most labs don't have enough expert labelers to cover the full distribution of hard math and coding problems.

RLVR cuts both problems at the root. If you have a deterministic verifier — a computer algebra system, a unit test suite, a formal proof checker, a regex matching the expected format — you don't need a reward model. You don't need human labels. You run the verifier on each completion and get a binary signal: correct or not. There is nothing to hack, because the verifier is not a neural network.

The tradeoff is domain coverage. RLVR only applies where ground truth exists. Math, code with test cases, structured data extraction with a schema validator, formal verification — these work. Open-ended writing, nuanced advice, creative tasks — they don't have verifiers, so RLHF remains the tool.

The GRPO algorithm

PPO (Proximal Policy Optimization) is the standard online RL algorithm for language model fine-tuning. It works by sampling completions from the current policy, computing advantages (how much better was this completion than baseline?), and updating the policy with a clipped objective that prevents large steps. The critic (value) network estimates the baseline — the expected future reward from any given state. Training it alongside the policy is what makes PPO expensive: you're maintaining two large models and optimizing both simultaneously.

GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper (Shao et al., 2024) and made famous by DeepSeek-R1, replaces the learned value baseline with a group statistic. For each prompt, you sample a group of N completions from the current policy, run the verifier on all of them, and normalize the resulting rewards within the group:

For prompt p, sample completions {c₁, c₂, ..., cₙ}
Verifier assigns rewards {r₁, r₂, ..., rₙ}

Group mean:  μ = mean({r₁, ..., rₙ})
Group std:   σ = std({r₁, ..., rₙ})

Advantage of completion cᵢ:  Aᵢ = (rᵢ - μ) / (σ + ε)

Those group-relative advantages are then used in a standard PPO-style clipped objective. No critic network. No separate value model training. The group is the baseline.

Why does this work? The key insight is that for binary rewards, the group mean is exactly the fraction of completions that were correct. An advantage of +1.5 means "this completion was 1.5 standard deviations above the group average" — which is a meaningful signal even without a global baseline. The completions that got the right answer receive positive gradient signal; the ones that didn't receive negative signal. The policy is pushed toward the strategies that found correct answers in this group.

# Simplified GRPO update step (illustrative, not a production implementation)
import torch

def grpo_advantages(rewards: torch.Tensor) -> torch.Tensor:
    """
    rewards: shape (batch_size, group_size)
    returns: advantages, normalized within each group
    """
    # rewards[i, j] = reward for completion j of prompt i
    group_mean = rewards.mean(dim=1, keepdim=True)   # (batch, 1)
    group_std  = rewards.std(dim=1, keepdim=True)    # (batch, 1)
    advantages = (rewards - group_mean) / (group_std + 1e-8)
    return advantages  # (batch, group_size)


def grpo_loss(log_probs_policy: torch.Tensor,
              log_probs_old: torch.Tensor,
              log_probs_reference: torch.Tensor,
              advantages: torch.Tensor,
              clip_eps: float = 0.2,
              kl_coeff: float = 0.01) -> torch.Tensor:
    """
    Clipped policy gradient + KL penalty against reference model.
    log_probs_*: (batch, group_size, seq_len)
    advantages:  (batch, group_size)
    """
    # Importance ratio against the OLD policy that sampled the completions.
    # The frozen reference model belongs only in the KL term below.
    ratio = torch.exp(log_probs_policy - log_probs_old)  # per-token
    
    # Average over sequence length -> (batch, group_size), same shape as advantages
    ratio = ratio.mean(dim=-1)
    
    # Clipped surrogate objective
    pg_loss = torch.min(
        ratio * advantages,
        torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
    )
    
    # KL penalty to the reference model (the k3 estimator GRPO uses:
    # exp(x) - x - 1 with x = log_ref - log_policy, always >= 0)
    log_r = log_probs_reference - log_probs_policy
    kl = (torch.exp(log_r) - log_r - 1).mean(dim=-1)
    
    return -(pg_loss.mean() - kl_coeff * kl.mean())

The reference model (frozen copy of the policy at the start of training) appears in the KL term — it prevents the policy from wandering so far from the pre-trained distribution that it generates incoherent text. This is the same role the reference model plays in standard RLHF; GRPO keeps this guard.

R1-Zero: emergence from pure RL

The most striking result in the R1 paper is R1-Zero. They took a DeepSeek-V3-Base model — no instruction tuning, no SFT on reasoning examples — and applied GRPO with binary rewards from a math verifier. That's the entire recipe.

After training, the model produced spontaneous behaviors that nobody had programmed:

• Extended thinking: generating long chains of intermediate steps before stating an answer
• Self-reflection: revisiting and questioning previous steps ("Wait, let me reconsider...")
• Backtracking: abandoning a wrong approach and restarting with a different strategy

These are exactly the behaviors you'd want from a model reasoning through hard problems. They emerged because they increase the probability of reaching a correct answer, and GRPO rewards correct answers. The model learned that getting the right answer requires checking the work, so it learned to check the work.

sequenceDiagram
    participant P as Policy (R1-Zero)
    participant V as Math Verifier
    participant G as GRPO Updater
    
    Note over P: Prompt: "Compute ∫x²sin(x)dx"
    P->>P: Sample 8 completions (group_size=8)
    Note over P: C1: "= x²cos x + 2x·sin x + 2cos x + C" [wrong sign]
    Note over P: C2: "-x²cos x + 2x sin x + 2cos x + C" [correct]
    Note over P: C3-C8: various attempts...
    P->>V: Submit all 8 completions
    V-->>G: rewards = [0, 1, 0, 1, 0, 0, 0, 1]
    G->>G: group_mean = 0.375, group_std = 0.484
    G->>G: advantages = [-0.77, +1.29, -0.77, +1.29, ...]
    G->>P: Gradient update: reinforce strategies used in C2, C4, C8
    Note over P: Over many steps: model learns to check signs, integrate by parts carefully

The model didn't stay perfect in this pure-RL setting. R1-Zero's outputs were sometimes in mixed languages (switching between Chinese and English mid-sentence), and the reasoning format was occasionally inconsistent. That's where the cold-start addresses the gap.

The full DeepSeek-R1 recipe

DeepSeek-R1 adds a cold-start phase before the RL loop. The reasoning: a base model with no instruction tuning struggles in early RL training because the policy generates incoherent output, the verifier assigns mostly 0 rewards, and the gradient signal is too sparse to learn efficiently. A small SFT step — training on a few thousand high-quality long-form reasoning examples — gives the policy a reasonable initialization.

The full pipeline:

flowchart LR
    BASE[Base model\nDeepSeek-V3-Base] --> SFT1["Cold-start SFT\n(thousands of long-CoT examples)"]
    SFT1 --> RL1["GRPO Stage 1\n(math + code, RLVR)"]
    RL1 --> REJ["Rejection sampling\n(generate traces, keep correct ones)"]
    REJ --> SFT2["SFT Stage 2\n(high-quality trace corpus)"]
    SFT2 --> RL2["GRPO Stage 2\n(broader tasks incl. helpfulness)"]
    RL2 --> R1["DeepSeek-R1"]
    style BASE fill:#0e7490,color:#fff
    style R1 fill:#a855f7,color:#fff
    style RL1 fill:#00e5ff,color:#0a0a0f
    style RL2 fill:#00e5ff,color:#0a0a0f

The two SFT stages serve different purposes. The first (cold start) stabilizes early RL. The second (after the first RL phase) uses rejection sampling: run the partially-trained RL model on a large set of prompts, keep only the completions that the verifier marks correct, and SFT on that corpus. This creates a higher-quality training set than the original cold-start data, which the second RL stage can then extend further.

The result is a multi-stage pipeline that is more stable, converges faster, and produces better formatting than R1-Zero — at the cost of needing some labeled starting data. The cold-start corpus is small by RLHF standards: thousands to tens of thousands of examples, not millions.

The training signals: what gets verified

RLVR lives or dies on the quality of the verifier. DeepSeek-R1 used three categories:

Accuracy rewards. For math, a symbolic verifier (equation solver) checks whether the final boxed answer matches the ground truth. For code, unit tests are run against the completion. Binary: 0 or 1.

Format rewards. The model is expected to wrap its reasoning in <think>...</think> tags and its answer in <answer>...</answer> (or similar). A regex verifier checks for correct tag presence and placement. This shapes the output format without any human labeling.

Language consistency reward. R1-Zero's mixed-language outputs were penalized by a simple check: if the prompt is in English, the completion should be predominantly in English. Again a rule, not a model.

What's absent: no reward for how the model reasons, only for whether it reaches the right answer. Self-reflection and backtracking aren't rewarded directly — they're strategies the model discovers because they improve the probability of a correct final answer.

The visualization

{ "type": "rlhf-dpo", "method": "grpo", "title": "RLHF vs DPO vs GRPO: what each algorithm needs" }

The widget above makes the component difference concrete. RLHF requires three models in flight (policy, value/critic, reference) plus a separately-trained reward model. DPO needs only a policy and reference model but requires offline preference pairs. GRPO needs only a policy and reference — and replaces offline preference data with a live verifier that can generate rewards on-the-fly.

Distillation: getting reasoning into small models

Training a full reasoning model via GRPO requires large-scale GPU infrastructure and weeks of compute. The distillation path is dramatically cheaper: take a trained reasoning model (the teacher), generate a large corpus of its reasoning traces on a diverse problem set, and SFT a smaller model (the student) on those traces.

Distillation cost comparison (illustrative, order-of-magnitude):
-----------------------------------------------------------------
Full GRPO training on a 70B model:
  → Thousands of A100 GPU-hours, multiple RL training stages
  → Total: expensive enough that only well-funded labs can run it

SFT distillation on a 7B model from R1 traces:
  → R1 trace generation: ~100 GPU-hours of inference
  → SFT on 7B: ~50–200 GPU-hours depending on dataset size
  → Total: accessible on a single 8×H100 node over a weekend

Result: DeepSeek-R1-Distill-7B substantially outperforms Qwen-2.5-7B-Instruct
on AIME 2024 (pass@1) with no RL training whatsoever.

The DeepSeek-R1 paper released distilled variants at 1.5B, 7B, 8B, 14B, 32B, and 70B. The 14B variant already beats o1-mini on AIME 2024 (69.7 vs 63.6 pass@1, per the R1 paper's tables). The 70B variant beats o1-mini on both AIME and MATH-500 — though it still trails full o1. These models enable reasoning capabilities on hardware that can't run the full R1 model.

The mechanism is straightforward: the student model is learning to imitate the surface form of good reasoning traces, not to discover reasoning through RL exploration. This distinction matters for the failure modes section below.

If you are building fine-tuning pipelines for small models that need reasoning, distillation on R1 traces plus LoRA fine-tuning is currently the most cost-effective path. The SFT and instruction tuning article covers the data quality considerations that apply here too.

DAPO and REINFORCE++: fixing GRPO's entropy collapse

GRPO as described above has a known failure mode at scale: entropy collapse. As training progresses, the policy distribution sharpens — it becomes increasingly confident in particular output patterns. When entropy drops too low, the model stops exploring. It gets stuck generating the same reasoning patterns that happened to work early in training, missing better strategies that require initial exploration to discover. Final performance plateaus well below the theoretical maximum.

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization, 2025) addresses this with two changes:

1. Decoupled clip ratios: PPO uses the same clip bounds for positive and negative advantages. DAPO uses a larger clip on the positive side (allowing more aggressive reinforcement of good behaviors) and a smaller clip on the negative side (being more conservative about penalizing bad behaviors). This asymmetry maintains more diversity in the policy.

2. Dynamic sampling: prompts where all N samples in the group got reward 0 (model can't solve it yet) or all got reward 1 (too easy) provide zero gradient signal. DAPO filters these out and re-samples from harder prompts, improving training efficiency and reducing the chance of the policy collapsing on easy patterns.

DAPO also attacks a subtler pathology — length inflation (see What breaks). It replaces GRPO's per-sequence length normalization with a token-level loss, so every token of a long wrong answer carries full negative signal, and it soft-penalizes completions that blow past the length budget instead of assigning them a hard 0 from truncation.

REINFORCE++ goes the other direction: simplify the machinery, not the guardrails. It drops the critic and GRPO's per-group baseline, but keeps PPO-style clipping and adds a token-level KL penalty, relying on global batch reward normalization (subtract the batch mean, divide by standard deviation) for variance reduction. The argument is that per-group statistics add complexity and bias without buying stability; batch-level normalization plus the standard clip achieves comparable stability with a simpler pipeline.

Both methods outperform vanilla GRPO on hard reasoning benchmarks by 2–5 percentage points on AIME-style evaluations. If you are training reasoning models in 2026, these are the algorithms to reach for, not the original GRPO from the R1 paper.

What breaks

Entropy collapse. As noted, the primary training failure is the policy distribution collapsing into a narrow set of patterns. Symptoms: training loss appears stable but benchmark performance plateaus early and then degrades slightly. Fix: monitor policy entropy during training; if it drops below a threshold, increase exploration via higher KL penalty weight or switch to DAPO sampling.

Length inflation. GRPO's loss averages log-probabilities over each completion's own length, so a long wrong answer dilutes its negative advantage across thousands of tokens while a short wrong answer gets punished hard — the cheapest way to reduce penalty is to ramble. Dividing by the group standard deviation adds a second bias: nearly-solved and nearly-impossible prompts (low reward variance) get their advantages amplified. The observable symptom, documented in the 2025 Dr. GRPO analysis, is completions that grow steadily longer over training without getting more accurate. If you fine-tune with TRL or verl, runaway trace length is the failure you'll hit first — and it's part of why reasoning traces are so verbose in the first place. Fixes: token-level loss and overlong-sample penalties (DAPO), or dropping the length and std normalization terms (Dr. GRPO).

Reward hacking the verifier. RLVR is supposed to be unhackable, but it's not immune. Models can learn to output the exact format the format-checker is looking for while getting the underlying content wrong. On math tasks, models have learned to output the correct final answer without a valid derivation — technically satisfying the verifier, but not learning the reasoning skill. The fix is process reward models (PRMs) that score individual reasoning steps, not just the final answer — but PRMs require human labels on intermediate steps, partially reintroducing the RLHF labeling cost. This tension is unresolved as of mid-2026.

Distillation brittleness. Distilled models learn to imitate the surface form of R1's reasoning traces. On the training distribution, they perform impressively. On out-of-distribution inputs — adversarial prompts, unusual problem formats, problems requiring multi-domain knowledge — they break in ways that the parent R1 model handles more gracefully. The parent model discovered reasoning through exploration; the distilled model memorized patterns. That's a real difference under distribution shift. If you're using a distilled reasoning model in production, test it specifically on inputs that differ from its training distribution. See reasoning model failure modes for more on this.

Language mixing in pure RL. R1-Zero without cold start produces mixed-language reasoning traces when trained on multilingual data. The policy discovers that mixing languages can be useful for certain problem types — Chinese mathematical terminology is often more compact — but the output is unpredictable. The format reward partially addresses this; the cold-start SFT addresses it more reliably.

KL penalty miscalibration. The KL term that keeps the policy close to the reference model needs to be tuned carefully. Too low: the policy drifts into incoherent generation during RL. Too high: the policy can't move far enough from the base model to learn reasoning behaviors, and RL training is effectively suppressed. There's no universal setting; it depends on the base model, the task distribution, and the group size.

Comparing the algorithms

Algorithm	Value network?	Reward source	Exploration control	Known failure mode
PPO (standard RLHF)	Yes (expensive)	Learned reward model	Clipping + entropy bonus	Reward hacking on learned RM
DPO	No	Offline preference pairs	None (offline)	Requires good preference data; no online exploration
GRPO	No	Verifier (RLVR)	KL penalty, clip	Entropy collapse at scale
DAPO	No	Verifier (RLVR)	Decoupled clip + dynamic sampling	More complex hyperparameter tuning
REINFORCE++	No	Verifier (RLVR)	Clip + token-level KL + batch reward normalization	No per-group baseline; sensitive to batch reward statistics

For a broader treatment of DPO, SimPO, and ORPO in preference optimization contexts, see the preference optimization deep dive. This article has focused on RLVR specifically because that's the mechanism behind reasoning — the verifier, not the preference pairs, is the load-bearing component.

Where RLVR doesn't apply

RLVR requires a verifier. That boundary is real.

Tasks where RLVR works cleanly: mathematics with a known answer, code with unit tests, formal proofs with a proof checker, data extraction with a schema validator, multiple-choice questions with a ground-truth key, structured tasks with regex-verifiable output.

Tasks where RLVR cannot apply: open-ended writing quality, nuanced advice, creative tasks, subjective helpfulness, safety and harm avoidance (which requires human judgment, not a rule). For these, RLHF or constitutional AI variants remain the tool. The production post-training stack at frontier labs combines both: RLVR for reasoning capability, RLHF for style, safety, and edge cases.

RLHF, reward models, and constitutional AI covers the RLHF side of this stack. The two articles together describe the full post-training pipeline at a model like Claude or GPT.

The practical implications for AI engineers

You are unlikely to run full RLVR training on a frontier model — that requires compute resources and infrastructure that only a handful of organizations have. But this training story is directly relevant to applied AI engineering in several ways.

Distillation is tractable. The 7B–14B distilled models are accessible. If you need reasoning capability in a latency-constrained or cost-constrained deployment, fine-tuning a distilled R1 variant on your domain with LoRA is a realistic project. Understand what you're getting: surface imitation of reasoning patterns, not the full exploration-based robustness of the parent model.

Verifiers are a design primitive. If you're building a system that evaluates model outputs — an eval pipeline, an automated grader, a tool-use verifier — you're building a verifier in the RLVR sense. The design principles apply: binary signals are more reliable than continuous scores, rule-based checks are harder to game than learned evaluators, and checking final outputs is easier than checking intermediate reasoning steps.

Thinking budget is a training artifact. When you set a thinking budget in the Claude API or choose between o3-mini and o3, you're interacting with a parameter that emerged from outcome-reward RL over variable-length completions (OpenAI and Anthropic haven't disclosed their exact algorithms). The model learned that more compute can improve accuracy on hard problems. The budget you set determines how much of that training signal the model actually exercises. The cost and latency tradeoffs article goes into the practical budget-setting mechanics.

The recipe is now open. DeepSeek-R1 published enough detail that the community has reproduced the approach. Open-source implementations of GRPO-based reasoning training (using frameworks like TRL, verl, and OpenRLHF) are available as of 2025. If you're fine-tuning for a domain that has a verifier — legal citations with a citation database, medical diagnosis with ICD codes, code review with a linter — RLVR is no longer a proprietary technique.

When to trust the reasoning trace

One thing this training story makes clear: the reasoning trace you see in a <think> block or in Claude's extended thinking is not a faithful record of the model's internal computation. It's the output of a policy trained to produce traces that lead to correct answers. The trace is useful, often genuinely helpful for debugging, but it's evidence of what the model found useful to generate — not a causal account of how it computed the answer.

This matters for trust calibration. If a reasoning model's visible trace looks confident and well-structured but arrives at a wrong answer, don't assume the trace is lying — it might be generating a locally plausible argument path that happens to be wrong. And if the trace looks halting and self-correcting, that's a learned behavior, not evidence that the model is "confused." It's doing what the training signal rewarded.

The failure modes article covers what this means for production — specifically why sycophancy in reasoning (a trace that argues toward the answer the user seemed to want) is harder to catch than factual hallucination.

The judgment call

If you're deciding whether to engage with this part of the stack at all: for most AI engineering work, you don't need to understand GRPO to use reasoning models effectively. You need what's in the prompting article and the cost/latency framework.

But if you're evaluating reasoning model claims from vendors, designing fine-tuning pipelines, or trying to understand why a distilled model behaves differently than its parent, this training story is the right frame. The key things to hold:

The gains from reasoning models are real and large on verifiable multi-step tasks. They come from a training recipe, not a prompt trick. That recipe is now public and partially reproducible. Its primary cost is compute, its primary failure mode is entropy collapse, and its primary limitation is that it only applies to tasks where you can write a verifier.

The field is not done iterating. DAPO and REINFORCE++ appeared in 2025 as improvements; something else will appear by the time you read this. The underlying principle — use verifiable rewards to train exploration, not human preference labels — is stable. The specific algorithm implementing it is still moving.