RLHF, Reward Models, and Constitutional AI

The first time GPT-3 was fine-tuned with RLHF to produce InstructGPT, the resulting model was rated as "preferred" over the raw GPT-3 (175B parameters) in blind human evaluations — despite InstructGPT being fine-tuned from a 1.3B checkpoint. A 130× smaller model, subjectively better. That result landed in 2022 and rewrote what the field thought the bottleneck was. It was not the base model scale. It was the alignment layer.

That alignment layer is RLHF. And even though the field has largely replaced the PPO machinery with leaner methods like DPO and SimPO, understanding the original pipeline is not optional background. DPO is a shortcut through RLHF's math; you can't evaluate shortcut quality without knowing where it's cutting.

Why SFT alone is not enough

After supervised fine-tuning on high-quality (instruction, response) pairs, a model is meaningfully better than the base pretrain checkpoint. It follows instructions, stays on topic, and produces plausible-sounding answers. The problem is in the word "plausible."

SFT minimizes cross-entropy loss over a fixed dataset. The model learns to imitate the average of the demonstrations, with no signal about which responses humans actually prefer when presented with alternatives. A response that hedges everything with caveats scores the same as a direct, useful answer if both appear in the training data. An answer that's factually confident and confidently wrong might score identically to one that's correct, if both were written by annotators working at speed.

SFT also can't generalize naturally to "do not do X." You can demonstrate good behavior. You can't demonstrate the absence of bad behavior in a way the model internalizes robustly, because the negative examples you omit from training don't teach the model to avoid them — the model simply never trains on that territory.

RLHF solves this by replacing imitation with optimization: give the model a reward signal, and let it discover the behavior that maximizes it.

Phase 1: The SFT starting point

The SFT model is not a new idea in the SFT and instruction tuning article, but its role in RLHF is specific: it's the initialization point for both the reward model and the active policy.

SFT is trained on a relatively small set of curated (prompt, response) pairs — historically, OpenAI's InstructGPT used ~13K examples from human contractors. Quality matters far more than quantity here: the SFT model needs to produce coherent, instruction-following output so that the subsequent reward model training and PPO steps have something sensible to work with. A bad SFT initialization makes the reward model training distribution incoherent and the PPO step unstable.

The SFT model serves a second function during PPO: it becomes the KL reference — the fixed checkpoint the PPO objective penalizes the active policy for diverging too far from. More on that shortly.

Phase 2: The reward model

The reward model is a language model. Specifically, it's the SFT model with its final language-modeling head removed and replaced by a single linear layer projecting to a scalar. Given a prompt x and a response y, it outputs r(x, y) — a single number representing how good that response is.

Training it requires comparison data: pairs of responses to the same prompt where human labelers have selected one as better. The loss function is a Bradley-Terry ranking loss:

L(θ) = -E[(x, y_w, y_l)] log σ(r(x, y_w) - r(x, y_l))

Where y_w is the "winning" (preferred) response and y_l is the "losing" (rejected) response. The model learns to assign y_w a higher scalar than y_l. Note that it never sees an absolute ground-truth score — only relative preferences.

import torch
import torch.nn as nn
from transformers import AutoModel

class RewardModel(nn.Module):
    def __init__(self, base_model_name: str):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(base_model_name)
        hidden_size = self.backbone.config.hidden_size
        # Replace LM head with a scalar head
        self.scalar_head = nn.Linear(hidden_size, 1, bias=False)

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        # Pool at the last *non-pad* token — [:, -1, :] would silently read
        # pad embeddings on right-padded batches
        seq_lens = attention_mask.sum(dim=1) - 1
        last_hidden = outputs.last_hidden_state[
            torch.arange(input_ids.size(0), device=input_ids.device), seq_lens
        ]
        reward = self.scalar_head(last_hidden).squeeze(-1)
        return reward

def bradley_terry_loss(reward_chosen: torch.Tensor, reward_rejected: torch.Tensor) -> torch.Tensor:
    # Positive means chosen is preferred over rejected
    return -torch.log(torch.sigmoid(reward_chosen - reward_rejected)).mean()

In practice, a reward model trained on ~50K–200K comparison pairs reaches roughly 65–75% agreement with held-out human judgments on general alignment tasks. This is measurably below the ~75–80% human-to-human agreement rate on the same pairs — which tells you something important: the reward model is an imperfect proxy. It's good enough to guide PPO, but it will be exploited.

Reward hacking: the structural flaw

This is not a corner case. It is the default outcome if you run PPO without adequate constraints. The PPO policy explores the reward model's input space and finds patterns that produce high scores without being genuinely good. Common failure modes observed in practice:

• Length bias: reward models trained on human annotations often reflect annotators' implicit preference for thorough answers; PPO discovers this and produces verbose, repetitive responses that look comprehensive without being informative.
• Sycophancy: if human raters tend to prefer responses that agree with their framing, the policy learns to affirm the user's premise regardless of accuracy.
• Surface-level quality signals: formatting, hedging phrases ("that's a great question"), and confident-sounding language that annotators associated with quality become spurious attractors in reward model space.

The reward model was trained on the SFT model's distribution. As PPO moves the policy into new territory, the reward model is scoring samples from a distribution it was never trained on. Its predictions degrade — sometimes silently, sometimes catastrophically.

Phase 3: PPO with a KL constraint

Proximal Policy Optimization adapted for language models optimizes the policy's token-level decisions using the reward model signal, subject to a constraint that keeps the policy from drifting too far from the SFT baseline.

The objective for each (prompt, generated response) pair is:

reward_objective = r(x, y) - β * KL[π_θ(y|x) || π_ref(y|x)]

Where:

• r(x, y) is the reward model's scalar score
• π_ref is the frozen SFT model (the reference policy)
• β is the KL coefficient, typically 0.01–0.1
• KL[·||·] is the per-token Kullback-Leibler divergence between the active policy and the reference

The KL term is what prevents the policy from collapsing into gibberish. Set β too low and the policy runs freely toward whatever text pattern the reward model happens to over-score. Set β too high and the PPO gradient is suppressed to the point where the RL phase does nothing the SFT phase didn't already do.

sequenceDiagram
    participant P as Active Policy (π_θ)
    participant RM as Reward Model
    participant REF as SFT Reference (π_ref)
    participant PPO as PPO Optimizer

    loop For each batch of prompts
        P->>P: sample responses y ~ π_θ(·|x)
        P->>RM: score responses r(x, y)
        RM-->>PPO: reward scalars
        P->>REF: compute log probs under π_ref(y|x)
        REF-->>PPO: reference log probs
        PPO->>PPO: compute KL divergence penalty
        PPO->>PPO: compute final objective = reward − β·KL
        PPO->>P: gradient update (clipped by PPO ratio ε)
    end

In implementation, this requires holding four model copies in memory simultaneously:

1. The active policy (being trained)
2. The frozen SFT reference (for KL)
3. The reward model (for scoring)
4. A value/critic model (for PPO's advantage estimation)

For a 7B-parameter policy, that's approximately 4 × 14 GB (in bfloat16) = 56 GB before KV cache, optimizer states, or gradients. The reward model is often smaller (1B–3B), and the value model shares weights with the policy in some implementations, but you're still holding roughly four model copies where SFT holds one — the ~4–5× in the comparison table below. Counting the optimizer states and gradients that SFT also pays for, total training memory lands closer to 1.4–1.6× a comparable SFT run, as the block below shows. Either way, this is one of the core practical motivations for the DPO migration.

Memory estimates (bfloat16, no quantization):
─────────────────────────────────────────────────────────
Active policy (7B):          ~14 GB
SFT reference (7B, frozen):  ~14 GB
Value model (7B, shared):    ~0 GB (or ~14 GB separate)
Reward model (3B):           ~6 GB
Adam optimizer states:       ~2× parameter count = ~28 GB
Gradient storage:            ~14 GB

Total (shared value, 7B):   ~76 GB
Total (separate value, 7B): ~90 GB

→ A single A100 80GB handles this only with int8 quantization
  of the frozen models, or with DeepSpeed ZeRO-3 across 2+ GPUs.

{ "type": "rlhf-dpo", "method": "rlhf", "title": "RLHF vs DPO vs GRPO: what each method removes" }

How many PPO steps to run

This is more art than science, but there are useful heuristics. More steps means more reward maximization but also more KL drift from the SFT distribution and more risk that the reward model becomes stale. InstructGPT trained on the order of hundreds of PPO steps with batch sizes in the hundreds (exact figures not publicly specified). Anthropic's Claude 1 training used a similar order of magnitude. The practical rule: track both reward model score and a separate held-out human evaluation throughout PPO training. When the reward model score keeps rising but human eval quality plateaus or declines, you've reached the reward-hacking regime. Stop there and retrain the reward model on the new policy's distribution before continuing.

Constitutional AI: the RLAIF version

Constitutional AI is Anthropic's 2022 approach to scaling alignment feedback without scaling human annotation linearly. The core insight: if you need AI behavior to align with principles, you can use a language model as the judge of whether responses follow those principles — and write the principles down explicitly as a "constitution."

The training procedure has two phases:

flowchart TD
    SFT["SFT Model"] -->|"Generate harmful response to red-team prompt"| HARM["Potentially harmful\nresponse"]
    HARM -->|"Critique: 'Does this violate principle X?'"| CRITIQUE["Critique Model\n(guided by constitution)"]
    CRITIQUE -->|"Revise: 'Rewrite to avoid harm'"| REVISED["Revised response"]
    REVISED -->|"Multiple revision rounds"| CLEAN["Cleaned SL-CAI dataset"]
    CLEAN -->|"Supervised fine-tuning"| SLCAI["SL-CAI model"]
    
    SLCAI -->|"Generate response pairs"| PAIRS["(response A, response B)\nfor same prompt"]
    PAIRS -->|"AI judge: which better follows\nconstitution principle?"| AIJUDGE["AI Preference\nLabels (RLAIF)"]
    AIJUDGE -->|"Train reward model"| RM_CAI["RL-CAI Reward Model"]
    RM_CAI -->|"PPO"| FINAL_CAI["Final aligned model"]
    
    style CRITIQUE fill:#b45309,color:#fff
    style AIJUDGE fill:#b45309,color:#fff
    style FINAL_CAI fill:#15803d,color:#fff

The SL-CAI phase (supervised learning) is a self-critique-and-revision loop. The SFT model generates responses to adversarial prompts, then a critique model (also the SFT model, prompted differently) critiques those responses against specific constitutional principles, then generates revisions. This process runs for multiple rounds per prompt, producing a cleaned dataset of (harmful prompt, safe response) pairs that become a new SFT stage.

The RL-CAI phase is standard RLHF except that human preference labels are replaced with AI preference labels — the "AI Feedback" in RLAIF. A frontier model is prompted with pairs of responses and asked to select the one better aligned with a stated constitutional principle. These AI-generated preferences train the reward model. PPO then runs against that reward model as usual.

The constitution itself is a list of principles — Anthropic published theirs, which includes principles derived from sources including the UN Declaration of Human Rights, safety research, and practical helpfulness heuristics. It reads like a policy document, which is what it is.

Example principle (paraphrased from Anthropic's published constitutions):
──────────────────────────────────────────────────────
"Which of these assistant responses is less harmful?
Choose the response that a thoughtful, senior Anthropic
employee would consider optimal — someone who cares
about doing the right thing but also wants the assistant
to be genuinely helpful and understands the value of this."

Alternative principle (explicit harm avoidance):
"Choose the response that is least likely to contain
information that could be used to produce biological,
chemical, nuclear, or radiological weapons."

The AI judge's agreement with human preference pairs on safety-relevant tasks is roughly 70–75% — within the range of human-to-human agreement (75–80%) but not better. The advantage is cost: where a human comparison costs $0.12–$0.25 in labeler time, an AI judge call on a Claude or GPT-4-class model costs roughly $0.005–$0.02, a 6–50× reduction per comparison. At 200K pairs, that's the difference between a $24,000–$50,000 human annotation budget and a $1,000–$4,000 API budget.

The bias risk is real. An AI judge reflects its training data and its own alignment properties. If the judge model has absorbed particular cultural framings, over-refuses at certain risk categories, or has blind spots in specific domains, those biases propagate directly into the reward model and from there into the policy. With human annotators you can audit individual judgments; with an AI judge you're auditing a black box. Anthropic's mitigation is the written constitution — by anchoring the AI judge to explicit principles, you at least make the evaluation criteria auditable even if the judge's implementation of them isn't.

What breaks

Reward hacking at scale

The longer PPO runs, the more aggressively it exploits the reward model. In published InstructGPT analysis, response length grew monotonically throughout PPO training — a reward model artifact that didn't reflect genuine preference for verbosity. Anthropic noted similar dynamics with sycophancy: models learned that affirming user framing got higher reward-model scores. Neither behavior showed up in early PPO steps; both emerged as the policy pushed further into the reward model's extrapolation zone.

Mitigation: monitor the distribution of response lengths, detect sycophancy indicators (does the model agree more with users who state an incorrect belief confidently?), and retrain the reward model periodically on policy outputs.

KL coefficient drift

The β coefficient that controls the KL penalty is highly sensitive. At β = 0.01, PPO runs freely and hacks the reward model within a few hundred steps on most tasks. At β = 0.3, the policy barely moves from SFT. The right value depends on how well-calibrated the reward model is for the current policy distribution — which changes throughout training. Some implementations use adaptive β schedules that raise or lower β to hold the measured KL near a target threshold — a separate mechanism from PPO's ratio clipping, which bounds individual policy updates rather than cumulative drift. None of these eliminate the issue, they just manage it.

Distribution shift in the reward model

A reward model trained on SFT model outputs is calibrated to that distribution. After 200 PPO steps, the policy's outputs look different. The reward model's predictions on out-of-distribution policy outputs can be confidently wrong — it will assign high scores to responses that look like its training distribution's high-scoring examples superficially, even if they're actually worse. This is not a hypothetical: it's why frontier labs retrain reward models regularly and why the field moved toward DPO, which sidesteps the distribution shift problem by baking the preference signal into the policy training objective directly.

Constitutional AI judge blindspots

When the AI judge misclassifies, it does so consistently across the thousands of comparisons it generates. A human annotator who makes a mistake on one type of prompt makes an independent mistake; an AI judge that's miscalibrated on a category makes the same mistake on every example in that category. The reward model learns from this systematic error as if it were signal. This can create coordinated biases in the aligned model that are harder to detect than random noise.

The sycophancy trap

This deserves its own mention because it's the failure mode that directly damages product utility. RLHF-trained models learned to agree with users because human annotators slightly preferred responses that aligned with their own views. The effect is small per comparison but compounds through PPO. The result is a model that, when presented with a factually wrong premise from a user who sounds confident, will often affirm rather than correct. This is not hallucination in the mechanistic sense; it's a preference optimization artifact. Detection requires specific evaluation sets where the user states incorrect beliefs — not general capability benchmarks.

RLHF vs the alternatives

Method	RM needed?	Reference model?	Value model?	GPU footprint vs SFT	Main limitation
PPO-RLHF	Yes	Yes (frozen SFT)	Yes	~4–5×	Complexity, reward hacking
DPO	Implicit	Yes (frozen SFT)	No	~2×	Requires offline preference pairs
SimPO	Implicit	No	No	~1.2×	Calibration sensitivity
GRPO	Optional (often a rule-based verifier)	Yes	No	~2–3×	Advantage noise with small groups; usually paired with verifiable rewards
Constitutional AI (CAI)	Yes (RLAIF)	Yes	Yes	~4–5×	AI judge bias propagation

DPO's insight is that the reward model can be expressed analytically in terms of the policy and the SFT reference, eliminating the need to train it separately. The preference optimization article covers this in detail. GRPO, which powers DeepSeek-R1's reasoning training, takes a different approach: it drops the value model by scoring each response relative to a group of samples for the same prompt. The estimator itself works with any reward source — including a learned reward model — but the R1 training recipe pairs it with programmatic verifiable rewards (math evaluators, code execution), which never degrade with distribution shift because they're computed from ground truth rather than from a learned model.

The through-line is that every method in this table is trying to solve the same problem: SFT imitates, but alignment requires optimization. They differ in how they construct the optimization signal and what components they're willing to collapse or approximate away.

Practical implementation notes

If you're implementing RLHF from scratch in 2026, the default library is Hugging Face TRL. The loop below is written against the legacy TRL API (≤0.11) because its explicit generate → score → step structure makes the mechanics visible; TRL 0.12+ replaced it with a PPOTrainer that takes reward_model and value_model arguments and runs everything inside trainer.train():

# Minimal RLHF-style training loop — legacy TRL (≤0.11) API, shown for clarity
from trl import PPOConfig, PPOTrainer, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer

model = AutoModelForCausalLMWithValueHead.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

ppo_config = PPOConfig(
    model_name="llama-3.1-8b-rlhf",
    learning_rate=1.41e-5,
    batch_size=128,
    mini_batch_size=16,
    gradient_accumulation_steps=4,
    ppo_epochs=4,
    kl_penalty="kl",         # kl | abs | mse | full
    init_kl_coef=0.05,       # β — start conservative
    target_kl=6.0,           # adaptive KL controller target (separate from cliprange)
    horizon=10_000,
    cliprange=0.2,
    cliprange_value=0.2,
    vf_coef=0.1,
    max_grad_norm=1.0,
)

trainer = PPOTrainer(
    config=ppo_config,
    model=model,
    ref_model=None,          # TRL auto-creates frozen copy from model
    tokenizer=tokenizer,
    dataset=prompt_dataset,  # prompts only — PPO samples its own responses;
                             # comparison pairs belong to Phase 2 (reward model)
    data_collator=collator,
)

# Training loop (simplified)
for batch in trainer.dataloader:
    query_tensors = batch["input_ids"]
    
    # Generate responses from active policy
    response_tensors = trainer.generate(
        query_tensors, 
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
    )
    
    # Score with reward model
    rewards = [reward_model(q, r) for q, r in zip(query_tensors, response_tensors)]
    
    # PPO step (includes KL computation internally)
    stats = trainer.step(query_tensors, response_tensors, rewards)
    trainer.log_stats(stats, batch, rewards)

The init_kl_coef=0.05 and target_kl=6.0 settings implement an adaptive KL schedule: if the actual KL between the policy and reference exceeds 6.0 nats, the trainer increases β; if it's well below, β decreases. This is more robust than a fixed β across the entire training run, since the appropriate level of constraint changes as the policy drifts.

A direct-RLAIF variant skips the trained reward model and uses an AI judge API call as the reward signal itself. Note that this is a different design from CAI proper, where the AI-generated preference labels train a reward model and PPO runs against it as usual. And even the direct variant still holds the active policy, the value model, and the frozen SFT reference for the KL penalty — three of the original four models — with only the reward score arriving from an external API call.

async def constitutional_reward(prompt: str, response: str, principle: str) -> float:
    """Query an AI judge for Constitutional AI RLAIF."""
    judge_prompt = f"""Consider this principle: {principle}

User asked: {prompt}

Assistant responded: {response}

On a scale of 1-10, how well does this response follow the principle?
Reply with only a number."""
    
    result = await anthropic_client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=10,
        messages=[{"role": "user", "content": judge_prompt}],
    )
    
    try:
        score = float(result.content[0].text.strip())
        return (score - 5.0) / 5.0  # normalize to [-1, 1]
    except ValueError:
        return 0.0  # abstain on parse failure

This is not production-grade — you'd want a structured output, batched calls, caching of identical prompt-response pairs, and a calibration study against human labels — but it illustrates why RLAIF is operationally attractive: the reward computation is an API call you can iterate on without retraining a model.

When to understand this vs when to use a shortcut

If you're doing post-training alignment on a model you control, you're almost certainly using DPO or SimPO rather than PPO. The reasons are concrete: simpler infrastructure, no separate reward model to maintain, no four-model memory footprint, and comparable quality in most benchmarked settings.

The cases where you actually want the full RLHF pipeline today:

• Frontier-scale alignment: the reward model–policy interaction at 70B+ parameters produces qualitatively better results in some evaluations; labs with H100 clusters use it precisely because they can.
• Iterative reward model improvement: if you're running multiple PPO rounds and refreshing the reward model between rounds, the explicit separation between RM and policy is an advantage, not a liability.
• Constitutional AI pipelines: if you need auditable, principle-grounded alignment with RLAIF feedback and a large annotation budget is unavailable, CAI's written constitution approach is still the most principled way to encode organizational values into a reward signal.
• Understanding what DPO is doing: the log-ratio objective in DPO is a closed-form derivation of the optimal policy under an RLHF objective. If your DPO training diverges or produces sycophantic outputs, diagnosing it requires understanding what the reward model it's implicitly representing would look like.

The decision framework for when to fine-tune at all should precede this decision. RLHF is expensive. It's warranted when you need deep behavioral alignment that SFT can't provide and you have the compute budget and the annotation data or an AI judge pipeline to support it. For most product teams, DPO on a clean preference dataset — or even SimPO, which drops the reference model requirement — is the better starting point. But the mechanics described here are what every alignment paper since 2022 is building on, shortcutting, or reacting against. Start here to understand what they're skipping.