Reasoning Distillation: Teaching Small Models to Think

In the week DeepSeek-R1 released its model weights in January 2025, something quietly explosive happened: six smaller models trained exclusively via SFT on the teacher's traces appeared alongside the 671B flagship. The 7B version beat QwQ-32B-Preview on AIME 2024 and hit 92.8% on MATH-500. No RL. No reward model. No group sampling. Just a standard supervised fine-tuning run on a filtered dataset of reasoning traces. Teams running 32B models for math and code workflows suddenly had a cheaper path — and the recipe was open-source.

This article covers how that recipe works, where it breaks, and when to reach for it versus running RL directly on a small model.

What reasoning distillation actually is

Knowledge distillation in the classical sense (Hinton et al., 2015) trains a small student to match the output distribution of a large teacher — soft labels over the full vocabulary let the student absorb uncertainty and "dark knowledge" the hard target throws away. Reasoning distillation is a specialization: the thing being transferred is not a token distribution but a behavioral policy — the structured, multi-step reasoning process the teacher uses to arrive at answers.

The distinction matters. A standard LLM trained to predict tokens will output fluent text. A reasoning model trained with GRPO on verifiable rewards learns to work through problems — backtrack when stuck, verify intermediate steps, try alternate paths. That behavior is encoded in the chain-of-thought traces the model produces. Distillation copies those traces into the student via SFT, so the student learns to mimic the process, not just the answer.

Why not just prompt the student to think step-by-step? You can, and chain-of-thought prompting does help. But a model that was never trained to use reasoning traces cannot maintain them reliably under distribution shift — it collapses to direct answers when it gets confused, which is exactly when reasoning matters most. The distilled model has internalized the habit of structured thought.

sequenceDiagram
    participant T as Teacher (671B, GRPO-trained)
    participant D as Trace Dataset
    participant S as Student (7B, SFT)

    Note over T: Already learned reasoning<br/>through expensive RL
    T->>D: Generate 800K traces offline<br/>(think + answer blocks)
    Note over D: Filter by difficulty,<br/>deduplicate, format
    D->>S: SFT with cross-entropy<br/>on full trace + answer
    Note over S: Learns behavioral policy<br/>without running RL

The DeepSeek-R1 recipe in detail

The DeepSeek-R1 paper is the clearest public blueprint for this at scale. The teacher — a 671B Mixture-of-Experts model — was trained, roughly, in two stages: first a cold-start SFT stage on curated long chain-of-thought data — the fix for the unreadable, language-mixing traces of the pure-RL R1-Zero — then GRPO on math and code problems that have verifiable ground-truth answers (a symbolic math solver or a code executor provides the reward signal). The result was a model that produces structured reasoning traces with a consistent <think>...</think> format, backtracking behavior, and self-verification.

The distillation dataset: 800K samples total, with 600K reasoning traces from math/code domains and 200K general instruction-following samples. That general mix is important — a student trained only on math reasoning traces loses general instruction-following ability, a phenomenon sometimes called alignment tax erasure. The 200K non-reasoning samples preserve the base behavior.

Six student models were trained: Qwen-2.5 at 1.5B, 7B, 14B, and 32B, and Llama-3 at 8B and 70B. All trained via standard SFT — no GRPO, no reward model, no reference model. The result:

Benchmark comparison (from DeepSeek-R1 technical report):
─────────────────────────────────────────────────────────────
                        AIME 2024   MATH-500   LiveCodeBench
─────────────────────────────────────────────────────────────
QwQ-32B-Preview         50.0%       90.6%      41.9%
R1-Distill-Qwen-7B      55.5%       92.8%      37.6%
R1-Distill-Qwen-14B     69.7%       93.9%      53.1%
R1-Distill-Qwen-32B     72.6%       94.3%      57.2%
─────────────────────────────────────────────────────────────
7B beats 32B on AIME because the teacher's traces transfer well
to smaller models when the task has verifiable structure.

A 7B model beating a 32B on a hard math benchmark is what makes this technique worth understanding deeply. And QwQ-32B-Preview is no strawman — it is itself an RL-trained reasoning model with heavy self-verification. The punchline is not that the 32B lacked reasoning behavior; it is that plain SFT on a stronger teacher's traces let a model 4.5× smaller match or beat a dedicated 32B reasoner.

The traces only exist because of the teacher's RL loop — the visualizer below shows the GRPO group-sampling mechanism that produced them.

{ "type": "rlhf-dpo", "method": "grpo", "title": "How the Teacher Was Trained: GRPO Group Sampling" }

Offline vs on-policy distillation

The DeepSeek recipe is offline distillation: the teacher generates all traces before training begins, the student trains on a static dataset. This is the simplest and cheapest path. But it has a structural weakness: the student is only exposed to states the teacher visits, which means the student distribution diverges from the training distribution as the student generates text.

Think of it as a student reading solved exam papers. They learn from the solutions as written, but they never practice the decision of which path to choose when the first approach hits a wall — because the teacher's trace already made that decision. At test time, the student's own intermediate steps may not match the states the teacher's traces were conditioned on.

On-policy distillation (OPD) addresses this by having the student sample its own reasoning traces during training, then having the teacher score each student token via per-token KL divergence against the teacher's conditional distribution. The student learns not just from the teacher's traces, but from correcting its own deviations. The quality gain is real — roughly 10–15% on hard reasoning benchmarks in 2025 survey results — but the compute cost is 2–3× higher because the teacher model must run a forward pass at every training step.

flowchart TD
    subgraph OFFLINE["Offline (DeepSeek recipe)"]
        OT[Teacher] -->|"pre-generate traces"| OD[(Static dataset)]
        OD -->|"cross-entropy loss"| OS[Student]
    end

    subgraph ONPOL["On-Policy Distillation"]
        PT[Teacher running live]
        PS[Student samples tokens]
        KL["Per-token KL:\nD_KL(P_teacher || P_student)"]
        PS -->|"student draft tokens"| KL
        PT -->|"teacher conditional distribution"| KL
        KL -->|"gradient"| PS
    end

    style OFFLINE fill:#0e7490,color:#fff
    style ONPOL fill:#a855f7,color:#fff
    style OT fill:#a855f7,color:#fff
    style OS fill:#00e5ff,color:#0a0a0f

In practice: use offline distillation first. If you have a clear quality ceiling and the budget to go further, add on-policy distillation or GRPO on top of the distilled checkpoint. Running GRPO from scratch on a small model without the distillation head start is inefficient — you pay RL training costs to discover reasoning patterns the teacher has already figured out.

The learnability gap

Here is the pitfall most teams hit: they take an open reproduction of the DeepSeek trace set (OpenThoughts 3, Open-R1 — the original 800K traces were never released) or traces they generated against the R1 API, fine-tune a 7B model, and the reasoning behavior is either absent or regresses halfway through training. The problem is not the recipe — it's trace difficulty mismatch.

A 671B MoE teacher can hold extraordinarily deep context and draw on breadth that a 7B dense model simply does not have. When the teacher's trace references a theorem application or multi-step inference pattern that is effectively out-of-distribution for the student, the student's gradient on that token sequence is noise. Worse, a difficult trace may teach the student to start reasoning chains it cannot complete, which is more confusing than no reasoning at all.

Fixing the learnability gap:

Difficulty-aware filtering. Score traces by the student's per-token perplexity on the first 20 tokens of the reasoning trace. Traces where the student is wildly uncertain (PPL > threshold, typically set empirically around the 90th percentile) should be discarded or deferred. The student must be able to enter the reasoning state before it can learn from the path through it.

Curriculum ordering. Sort the training dataset from simpler to harder traces and train on them in order. The student builds scaffolding from easier examples before attempting the difficult ones. This is the same intuition as curriculum learning in vision models — trivial to implement (sort by trace length is a rough proxy for difficulty), meaningful quality lift.

Multi-teacher diversity. If you have access to multiple teacher models of different sizes (say, R1-Distill-32B and R1 671B), mixing traces from both can reduce the gap — the student can benefit from the smaller teacher's traces early in training and graduate to harder examples from the larger teacher.

The trace format is load-bearing

The <think>...</think> format in DeepSeek traces is not a cosmetic choice. It creates a clear boundary between the reasoning process (which the student is also trained on) and the final answer (where correctness is verified). Training on the full trace including the thinking tokens is what instills the backtracking habit.

What happens if you strip the thinking tokens and train only on the final answers? You get a faster model that has lost the reasoning behavior — it is now just a smaller SFT model. You also get a worse model than you would have gotten from training the same student on a standard instruction-following dataset, because you have wasted your fine-tuning budget on an impoverished signal.

The correct SFT loss is cross-entropy on the entire sequence: [<think> ... </think> <answer> ... </answer>]. Some teams mask the <think> tokens from the loss (train only to predict the answer), which sounds appealing — why train on the thinking tokens if you only care about the answer? Because the thinking tokens are how the model develops the internal state it needs to produce the answer correctly. Masking them from the loss is masking the training signal on the behavior you actually care about.

# Correct: train on the entire trace including thinking tokens
from datasets import Dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

# A sample from the distillation dataset
example = {
    "messages": [
        {"role": "user", "content": "Solve: find all x where x^2 - 5x + 6 = 0"},
        {
            "role": "assistant",
            "content": (
                "<think>\n"
                "Let me factor this quadratic. I need two numbers that multiply to 6 "
                "and add to -5. That's -2 and -3. So (x-2)(x-3) = 0.\n"
                "Check: if x=2, 4-10+6=0 ✓. If x=3, 9-15+6=0 ✓.\n"
                "</think>\n"
                "x = 2 and x = 3"
            )
        }
    ]
}
dataset = Dataset.from_list([example])  # in practice: your full filtered trace set

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

# SFTTrainer applies the chat template and puts loss on the assistant turn
# (including the <think> block)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=LoraConfig(r=32, lora_alpha=64, target_modules="all-linear"),
    args=SFTConfig(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,  # LoRA-scale LR; drop to ~1e-5 for a full fine-tune
        lr_scheduler_type="cosine",
        num_train_epochs=2,
        bf16=True,
        max_length=8192,
    ),
)
trainer.train()

What breaks

Reasoning inflation from GRPO without a length penalty

If you run GRPO on the distilled student to push quality further, models learn quickly that longer chains score better within the group-relative reward normalization — because more tokens give more opportunity to eventually land on a correct step. Without an explicit length penalty in the reward function, you will see reasoning chains grow from ~500 tokens to 3,000–5,000 tokens with no accuracy improvement. The reward signal is being gamed.

The fix is a length penalty term in the reward: R_total = R_correctness - λ × log(len(trace) / target_len) where λ controls the trade-off. The DAPO variant of GRPO (ByteDance Seed, 2025) introduced an overlong-response penalty alongside other GRPO improvements. If you ship a distilled model and then apply GRPO, budgeting for this is not optional.

Teacher traces from CoT-prompted models transfer poorly

There is a class of what I will call imitation reasoning traces: outputs from GPT-4 or Claude prompted with "think step by step" rather than outputs from models that learned to reason through RL. These traces look similar on the surface but are fundamentally different. A CoT-prompted model is doing pattern-matching to produce reasoning-shaped text — it has no RL-trained habit of backtracking or self-correction. Students trained on these traces pick up the superficial format without the substance, producing confident-looking chain-of-thought that does not catch errors.

If you are building a distillation pipeline, the teacher must be a model trained with GRPO or equivalent RL, and its raw chain of thought must actually be visible to you — DeepSeek-R1, QwQ-32B-Preview, or Qwen3's thinking-mode checkpoints. OpenAI's o-series and Gemini's thinking models qualify mechanistically, but their APIs hide or summarize the raw reasoning tokens, so you cannot harvest usable traces from them. OpenThoughts 3 provides 1M+ open traces from such models, which is the practical entry point for teams without API access to these teachers.

Catastrophic forgetting of general capabilities

Training on 800K reasoning traces, even with the 200K general samples mixed in, can degrade general instruction-following on tasks far from the training distribution. This shows up on benchmarks like MT-Bench or on conversational tasks that don't require multi-step reasoning. The mitigation is maintaining the 25–30% general-sample ratio in the training mix and evaluating on both reasoning benchmarks and general capability benchmarks before deployment.

Merging LoRA adapters at wrong precision

When distilling with QLoRA (the memory-efficient path for consumer hardware), you train bf16 LoRA adapters on a 4-bit quantized base. At the end, many pipelines merge the adapter back into the base weights to eliminate inference latency overhead. Merging bf16 adapters into an int4 quantized base corrupts the weights — the arithmetic does not commute across quantization levels. Always dequantize the base model to bf16 before merging, then re-quantize if needed for deployment. This is a silent failure: the merge succeeds without an error, but the model quality drops several points.

Distillation without a frontier-lab teacher

Most teams cannot run the 671B model. A few practical paths:

API-based trace generation. Call the DeepSeek-R1 API in batch mode — or another teacher whose raw <think> traces the API actually returns, like Qwen3 in thinking mode. OpenAI's o-series does not qualify: the API has hidden the raw chain of thought since o1, returning at most a summary, and the terms of service prohibit training competing models on outputs. At DeepSeek API pricing (approximately $2.19/1M tokens for R1 as of mid-2026 — verify current pricing), 800K traces averaging 2K tokens is roughly $3,500. Expensive but far cheaper than H100 cluster time for self-hosted reasoning RL.

Use OpenThoughts 3. The dataset (1M+ traces, ~10B tokens, from multiple reasoning teachers) is openly available. For many applications this is enough — you pick the subset relevant to your domain, apply difficulty filtering, and train. No API costs.

Go small on trace count, not just on models. Two early-2025 results reset the budget math: s1 fine-tuned Qwen2.5-32B-Instruct on ~1,000 hand-curated long reasoning traces, and LIMO used ~800, and both recovered most of the reasoning gains of trace sets hundreds of times larger — s1-32B was competitive on AIME and MATH-500 after a training run measured in tens of GPU-minutes. The caveat: both were demonstrated on strong 32B bases, so do not assume a thousand traces survives the learnability gap on a 7B student. But the direction matches everything else in this article: selection quality dominates volume. A few thousand aggressively filtered traces is a legitimate starting point, which makes the ~$3,500 API figure above an upper bound, not the entry fee.

Distill from smaller open-weight reasoning models. DeepSeek-R1-Distill-Qwen-32B is itself a capable reasoning teacher. Distilling from it into a 7B or 3B student is more tractable on available hardware and narrows the learnability gap because the teacher-student capability distance is smaller.

{ "type": "lora", "rank": 32, "model": "7b", "title": "QLoRA Memory Budget for 7B Distillation" }

Comparison: distillation vs direct RL vs prompting

Method	When it wins	Cost	Quality ceiling
CoT prompting	Base model already decent on the task; no training	Near-zero	Teacher's prompted quality
SFT on reasoning traces (distillation)	Small model, have a trained teacher or open traces	$3K–$10K total	Teacher's RL-trained distribution
GRPO on small model from scratch	Task has verifiable rewards; no suitable teacher	$15K–$40K+	Novel reasoning beyond teacher
GRPO on top of distilled checkpoint	Need to push past distillation ceiling	+50% over distillation cost	Best of both
On-policy distillation	High-quality bar; traces cover student failure modes	2–3× offline cost	Higher than offline at same budget

The decision depends on whether you have a trained reasoning teacher (not just a CoT-prompted model), whether your task has verifiable rewards, and whether you are trying to push past the teacher's capability distribution. See the reasoning model training article for the full GRPO/RLVR stack, and the preference optimization article for how DPO and SimPO fit into the broader post-training sequence.

The production judgment call

Distillation is a compression and transfer operation, not a capability amplifier. The student cannot discover reasoning patterns the teacher never produced. If your task requires a qualitatively different type of reasoning from what the teacher was trained on — say, multi-hop legal inference when the teacher was trained on math — you will get reasoning-shaped outputs that fail to generalize.

The practical threshold: distillation is the right move when your teacher can already solve your target tasks reliably and you need the capability at 4–8× lower inference cost. A distilled 7B that behaves like a 32B reasoning model is a real and substantial win for production latency and serving economics. See the inference cost analysis for the math on how reasoning model serving costs scale with output token length, which is the dominant variable.

Where distillation fails is when you need the student to generalize beyond the teacher's traces — handling task variants the teacher never saw, or developing reasoning strategies the teacher's RL training did not reward. In those cases, running GRPO directly on the student (starting from the distilled checkpoint) is the path forward, not more distillation data.

One honest limitation: the published DeepSeek-R1 distillation numbers are for math and code tasks with clear ground truth. Replicating those gains on messier real-world tasks — customer support, document analysis, scientific reasoning — is harder, requires careful trace quality filtering, and the current evidence is thinner. The synthetic data engineering article covers the quality pipeline considerations in depth; the distillation trace filter is an instance of the same problem.

The technique is not experimental. Teams running 7B models at inference costs that would make a 32B deployment untenable are shipping this in 2026. The recipe is open, the datasets exist, and the compute budget is within reach of any team with a modest cloud bill. The bottleneck is not the training run — it is trace quality, difficulty filtering, and honest evaluation against a held-out task distribution. Spend the time there, not on the hyperparameters.

For teams earlier in the fine-tuning stack, the decision ladder article covers when to reach for fine-tuning at all — reasoning distillation is near the top of that ladder, warranted only after you have confirmed that prompting and RAG cannot meet your quality bar.