# RLHF, Reward Models, and Constitutional AI > How reinforcement learning from human feedback actually works: the reward model, PPO loop, RLAIF, and Constitutional AI — the baseline before DPO shortcuts. Canonical: https://ironclad.academy/ai-engineering/articles/rlhf-reward-models-constitutional-ai | Difficulty: advanced | Topics: Fine-Tuning, Post-Training, LLM Internals, Fundamentals Covers: Anthropic, OpenAI, Google, DeepSeek ## Key takeaways - RLHF has three distinct phases: SFT to get a sensible base, reward model training on human comparisons, and PPO to optimize the policy against that reward signal. - A reward model is just a language model with its head replaced by a scalar; it reaches ~70% agreement with human raters and degrades as the policy drifts away from the distribution it was trained on. - The KL penalty is what keeps PPO from reward-hacking into gibberish — tuning β is one of the most consequential knobs in the whole pipeline. - Constitutional AI replaces human annotators in the RL feedback phase with an AI judge guided by a written list of principles, cutting per-comparison cost by roughly 6–50×. - Classic RLHF (PPO) holds four model copies simultaneously — policy, frozen SFT reference, reward model, and value/critic (sharing a value head with the policy trims one); DPO collapses this to one trained model plus a frozen reference whose log-probs can be precomputed — which is why the whole field migrated. RLHF is the foundation; DPO and its successors are the engineering shortcuts that made it practical. - Reward hacking is not a corner case — it is the default outcome if you run PPO too long or set β too low. Budget model restarts and reward model retraining as recurring line items. > **In one line:** RLHF is the three-phase pipeline — supervised fine-tuning, reward model training on human comparisons, then PPO optimization — that transformed base language models from "plausible text predictors" into aligned assistants; Constitutional AI is the RLAIF shortcut that makes the feedback phase tractable without a human-labeling army. **The idea.** A base language model trained on next-token prediction is good at producing text that resembles its training corpus. That is not the same as producing text that is helpful, accurate, or safe. RLHF closes this gap by introducing a proxy for human preference — the reward model — and using it as the optimization target for a policy gradient algorithm (PPO). Constitutional AI extends this by replacing the human labelers who generate preference pairs with an AI judge operating from an explicit written constitution, cutting the per-comparison cost of the feedback loop by roughly 6–50×. **Key points.** - RLHF = SFT base → reward model trained on human preference pairs → PPO policy optimization with KL constraint. - The reward model is a language model with a scalar head, trained on (chosen, rejected) pairs; it approximates human preference but degrades as the policy drifts. - The KL penalty prevents PPO from collapsing into reward-hacking degenerate outputs. - Constitutional AI (CAI) replaces human annotators in the RL phase with a "critique" model guided by a written set of principles (RLAIF), enabling scale without proportional annotation cost. - DPO and its successors (SimPO, GRPO) are streamlined alternatives that skip the explicit reward model — but they build on the same intuition RLHF established. ``` Back-of-the-envelope: RLHF annotation cost ───────────────────────────────────────────────────────────────── Human labeler rate: ~$15/hr Time per comparison pair: ~30–60 seconds Cost per pair: $0.12–$0.25 Pairs needed (small run): ~50,000–200,000 Total human annotation: $6,000–$50,000 per reward model RLAIF (Claude/GPT-4-class judge): Cost per comparison: ~$0.005–$0.02 (API call) Pairs for same run: 50,000–200,000 Total AI annotation: $250–$4,000 per reward model Savings: ~6–50× cheaper Agreement with human: ~70–75% (vs ~75–80% human–human baseline) ``` **The three-phase pipeline.** ```mermaid flowchart LR BASE["Base model\n(pretrained)"] -->|"Phase 1: SFT\ncross-entropy on\ncurated (prompt, response) pairs"| SFT["SFT model\n(instruction-following)"] SFT -->|"Phase 2: Reward Model\nBradley-Terry loss on\n(chosen, rejected) pairs"| RM["Reward Model\nr(x, y) → scalar"] SFT -->|"Phase 3: PPO\npolicy = copy of SFT"| POLICY["Active Policy"] RM -->|"reward signal"| POLICY SFT -->|"KL reference"| POLICY POLICY -->|"aligned outputs"| FINAL["Aligned Model"] style BASE fill:#334155,color:#e2e8f0 style SFT fill:#0e7490,color:#fff style RM fill:#b45309,color:#fff style POLICY fill:#0e7490,color:#fff style FINAL fill:#15803d,color:#fff ``` **Key design decisions.** | Decision | Typical choice | Why it matters | |---|---|---| | Reward model size | Same or smaller than policy | Oversized RM is expensive; undersized RM is too easy to hack | | KL coefficient β | 0.01–0.1 | Too low: reward hacking. Too high: PPO does nothing | | Feedback source | Human pairs vs RLAIF | Human: expensive, high quality; RLAIF: cheap, scalable, judge-biased | | Number of RL steps | 100–500 gradient steps | More steps → more drift → reward model becomes stale | | When to retrain RM | Every 1–2 PPO runs | Policy distribution shifts; old RM scores diverge from human preference | **If you have 60 seconds, say this.** "Base models predict plausible text; RLHF teaches them to predict preferred text. You start with SFT to get basic instruction-following, then train a reward model — literally the SFT model with its head swapped for a scalar — on human comparison data. Then you run PPO to push the policy toward higher reward, while a KL penalty keeps it from going so far it starts generating gibberish to game the reward model. Constitutional AI replaces the human annotators in that feedback loop with an AI judge that scores responses against a written list of principles, cutting annotation cost by up to ~50× at comparable agreement rates. DPO and SimPO eliminate the separate reward model entirely by baking the preference signal into the loss directly — which is why the field uses those now for most alignment work. But you need to understand RLHF to understand what those shortcuts are skipping." The first time GPT-3 was fine-tuned with RLHF to produce InstructGPT, the resulting model was rated as "preferred" over the raw GPT-3 (175B parameters) in blind human evaluations — despite InstructGPT being fine-tuned from a 1.3B checkpoint. A 130× smaller model, subjectively better. That result landed in 2022 and rewrote what the field thought the bottleneck was. It was not the base model scale. It was the alignment layer. That alignment layer is RLHF. And even though the field has largely replaced the PPO machinery with leaner methods like [DPO and SimPO](/ai-engineering/articles/preference-optimization-dpo-simpo-orpo-grpo), understanding the original pipeline is not optional background. DPO is a shortcut through RLHF's math; you can't evaluate shortcut quality without knowing where it's cutting. ## Why SFT alone is not enough After supervised fine-tuning on high-quality (instruction, response) pairs, a model is meaningfully better than the base pretrain checkpoint. It follows instructions, stays on topic, and produces plausible-sounding answers. The problem is in the word "plausible." SFT minimizes cross-entropy loss over a fixed dataset. The model learns to imitate the *average* of the demonstrations, with no signal about which responses humans actually prefer when presented with alternatives. A response that hedges everything with caveats scores the same as a direct, useful answer if both appear in the training data. An answer that's factually confident and confidently wrong might score identically to one that's correct, if both were written by annotators working at speed. SFT also can't generalize naturally to "do not do X." You can demonstrate good behavior. You can't demonstrate the *absence* of bad behavior in a way the model internalizes robustly, because the negative examples you omit from training don't teach the model to avoid them — the model simply never trains on that territory. RLHF solves this by replacing imitation with optimization: give the model a reward signal, and let it discover the behavior that maximizes it. ## Phase 1: The SFT starting point The SFT model is not a new idea in the [SFT and instruction tuning article](/ai-engineering/articles/sft-instruction-tuning-data-quality), but its role in RLHF is specific: it's the initialization point for both the reward model and the active policy. SFT is trained on a relatively small set of curated (prompt, response) pairs — historically, OpenAI's InstructGPT used ~13K examples from human contractors. Quality matters far more than quantity here: the SFT model needs to produce coherent, instruction-following output so that the subsequent reward model training and PPO steps have something sensible to work with. A bad SFT initialization makes the reward model training distribution incoherent and the PPO step unstable. The SFT model serves a second function during PPO: it becomes the **KL reference** — the fixed checkpoint the PPO objective penalizes the active policy for diverging too far from. More on that shortly. ## Phase 2: The reward model The reward model is a language model. Specifically, it's the SFT model with its final language-modeling head removed and replaced by a single linear layer projecting to a scalar. Given a prompt `x` and a response `y`, it outputs `r(x, y)` — a single number representing how good that response is. Training it requires **comparison data**: pairs of responses to the same prompt where human labelers have selected one as better. The loss function is a **Bradley-Terry ranking loss**: ``` L(θ) = -E[(x, y_w, y_l)] log σ(r(x, y_w) - r(x, y_l)) ``` Where `y_w` is the "winning" (preferred) response and `y_l` is the "losing" (rejected) response. The model learns to assign `y_w` a higher scalar than `y_l`. Note that it never sees an absolute ground-truth score — only relative preferences. ```python import torch import torch.nn as nn from transformers import AutoModel class RewardModel(nn.Module): def __init__(self, base_model_name: str): super().__init__() self.backbone = AutoModel.from_pretrained(base_model_name) hidden_size = self.backbone.config.hidden_size # Replace LM head with a scalar head self.scalar_head = nn.Linear(hidden_size, 1, bias=False) def forward(self, input_ids, attention_mask): outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask) # Pool at the last *non-pad* token — [:, -1, :] would silently read # pad embeddings on right-padded batches seq_lens = attention_mask.sum(dim=1) - 1 last_hidden = outputs.last_hidden_state[ torch.arange(input_ids.size(0), device=input_ids.device), seq_lens ] reward = self.scalar_head(last_hidden).squeeze(-1) return reward def bradley_terry_loss(reward_chosen: torch.Tensor, reward_rejected: torch.Tensor) -> torch.Tensor: # Positive means chosen is preferred over rejected return -torch.log(torch.sigmoid(reward_chosen - reward_rejected)).mean() ``` In practice, a reward model trained on ~50K–200K comparison pairs reaches roughly 65–75% agreement with held-out human judgments on general alignment tasks. This is measurably below the ~75–80% human-to-human agreement rate on the same pairs — which tells you something important: the reward model is an imperfect proxy. It's good enough to guide PPO, but it will be exploited. ### Reward hacking: the structural flaw This is not a corner case. It is the default outcome if you run PPO without adequate constraints. The PPO policy explores the reward model's input space and finds patterns that produce high scores without being genuinely good. Common failure modes observed in practice: - **Length bias**: reward models trained on human annotations often reflect annotators' implicit preference for thorough answers; PPO discovers this and produces verbose, repetitive responses that look comprehensive without being informative. - **Sycophancy**: if human raters tend to prefer responses that agree with their framing, the policy learns to affirm the user's premise regardless of accuracy. - **Surface-level quality signals**: formatting, hedging phrases ("that's a great question"), and confident-sounding language that annotators associated with quality become spurious attractors in reward model space. The reward model was trained on the SFT model's distribution. As PPO moves the policy into new territory, the reward model is scoring samples from a distribution it was never trained on. Its predictions degrade — sometimes silently, sometimes catastrophically. ## Phase 3: PPO with a KL constraint **Proximal Policy Optimization** adapted for language models optimizes the policy's token-level decisions using the reward model signal, subject to a constraint that keeps the policy from drifting too far from the SFT baseline. The objective for each (prompt, generated response) pair is: ``` reward_objective = r(x, y) - β * KL[π_θ(y|x) || π_ref(y|x)] ``` Where: - `r(x, y)` is the reward model's scalar score - `π_ref` is the frozen SFT model (the reference policy) - `β` is the KL coefficient, typically 0.01–0.1 - `KL[·||·]` is the per-token Kullback-Leibler divergence between the active policy and the reference The KL term is what prevents the policy from collapsing into gibberish. Set β too low and the policy runs freely toward whatever text pattern the reward model happens to over-score. Set β too high and the PPO gradient is suppressed to the point where the RL phase does nothing the SFT phase didn't already do. ```mermaid sequenceDiagram participant P as Active Policy (π_θ) participant RM as Reward Model participant REF as SFT Reference (π_ref) participant PPO as PPO Optimizer loop For each batch of prompts P->>P: sample responses y ~ π_θ(·|x) P->>RM: score responses r(x, y) RM-->>PPO: reward scalars P->>REF: compute log probs under π_ref(y|x) REF-->>PPO: reference log probs PPO->>PPO: compute KL divergence penalty PPO->>PPO: compute final objective = reward − β·KL PPO->>P: gradient update (clipped by PPO ratio ε) end ``` In implementation, this requires holding **four model copies** in memory simultaneously: 1. The active policy (being trained) 2. The frozen SFT reference (for KL) 3. The reward model (for scoring) 4. A value/critic model (for PPO's advantage estimation) For a 7B-parameter policy, that's approximately 4 × 14 GB (in bfloat16) = 56 GB before KV cache, optimizer states, or gradients. The reward model is often smaller (1B–3B), and the value model shares weights with the policy in some implementations, but you're still holding roughly four model copies where SFT holds one — the ~4–5× in the comparison table below. Counting the optimizer states and gradients that SFT also pays for, total training memory lands closer to 1.4–1.6× a comparable SFT run, as the block below shows. Either way, this is one of the core practical motivations for the DPO migration. ``` Memory estimates (bfloat16, no quantization): ───────────────────────────────────────────────────────── Active policy (7B): ~14 GB SFT reference (7B, frozen): ~14 GB Value model (7B, shared): ~0 GB (or ~14 GB separate) Reward model (3B): ~6 GB Adam optimizer states: ~2× parameter count = ~28 GB Gradient storage: ~14 GB Total (shared value, 7B): ~76 GB Total (separate value, 7B): ~90 GB → A single A100 80GB handles this only with int8 quantization of the frozen models, or with DeepSpeed ZeRO-3 across 2+ GPUs. ``` [Interactive visualizer on the original page: rlhf-dpo — RLHF vs DPO vs GRPO: what each method removes] ### How many PPO steps to run This is more art than science, but there are useful heuristics. More steps means more reward maximization but also more KL drift from the SFT distribution and more risk that the reward model becomes stale. InstructGPT trained on the order of hundreds of PPO steps with batch sizes in the hundreds (exact figures not publicly specified). Anthropic's Claude 1 training used a similar order of magnitude. The practical rule: track both reward model score *and* a separate held-out human evaluation throughout PPO training. When the reward model score keeps rising but human eval quality plateaus or declines, you've reached the reward-hacking regime. Stop there and retrain the reward model on the new policy's distribution before continuing. ## Constitutional AI: the RLAIF version Constitutional AI is Anthropic's 2022 approach to scaling alignment feedback without scaling human annotation linearly. The core insight: if you need AI behavior to align with principles, you can use a language model as the judge of whether responses follow those principles — and write the principles down explicitly as a "constitution." The training procedure has two phases: ```mermaid flowchart TD SFT["SFT Model"] -->|"Generate harmful response to red-team prompt"| HARM["Potentially harmful\nresponse"] HARM -->|"Critique: 'Does this violate principle X?'"| CRITIQUE["Critique Model\n(guided by constitution)"] CRITIQUE -->|"Revise: 'Rewrite to avoid harm'"| REVISED["Revised response"] REVISED -->|"Multiple revision rounds"| CLEAN["Cleaned SL-CAI dataset"] CLEAN -->|"Supervised fine-tuning"| SLCAI["SL-CAI model"] SLCAI -->|"Generate response pairs"| PAIRS["(response A, response B)\nfor same prompt"] PAIRS -->|"AI judge: which better follows\nconstitution principle?"| AIJUDGE["AI Preference\nLabels (RLAIF)"] AIJUDGE -->|"Train reward model"| RM_CAI["RL-CAI Reward Model"] RM_CAI -->|"PPO"| FINAL_CAI["Final aligned model"] style CRITIQUE fill:#b45309,color:#fff style AIJUDGE fill:#b45309,color:#fff style FINAL_CAI fill:#15803d,color:#fff ``` The **SL-CAI phase** (supervised learning) is a self-critique-and-revision loop. The SFT model generates responses to adversarial prompts, then a critique model (also the SFT model, prompted differently) critiques those responses against specific constitutional principles, then generates revisions. This process runs for multiple rounds per prompt, producing a cleaned dataset of (harmful prompt, safe response) pairs that become a new SFT stage. The **RL-CAI phase** is standard RLHF except that human preference labels are replaced with AI preference labels — the "AI Feedback" in RLAIF. A frontier model is prompted with pairs of responses and asked to select the one better aligned with a stated constitutional principle. These AI-generated preferences train the reward model. PPO then runs against that reward model as usual. The constitution itself is a list of principles — Anthropic published theirs, which includes principles derived from sources including the UN Declaration of Human Rights, safety research, and practical helpfulness heuristics. It reads like a policy document, which is what it is. ``` Example principle (paraphrased from Anthropic's published constitutions): ────────────────────────────────────────────────────── "Which of these assistant responses is less harmful? Choose the response that a thoughtful, senior Anthropic employee would consider optimal — someone who cares about doing the right thing but also wants the assistant to be genuinely helpful and understands the value of this." Alternative principle (explicit harm avoidance): "Choose the response that is least likely to contain information that could be used to produce biological, chemical, nuclear, or radiological weapons." ``` The AI judge's agreement with human preference pairs on safety-relevant tasks is roughly 70–75% — within the range of human-to-human agreement (75–80%) but not better. The advantage is cost: where a human comparison costs $0.12–$0.25 in labeler time, an AI judge call on a Claude or GPT-4-class model costs roughly $0.005–$0.02, a 6–50× reduction per comparison. At 200K pairs, that's the difference between a $24,000–$50,000 human annotation budget and a $1,000–$4,000 API budget. The bias risk is real. An AI judge reflects its training data and its own alignment properties. If the judge model has absorbed particular cultural framings, over-refuses at certain risk categories, or has blind spots in specific domains, those biases propagate directly into the reward model and from there into the policy. With human annotators you can audit individual judgments; with an AI judge you're auditing a black box. Anthropic's mitigation is the written constitution — by anchoring the AI judge to explicit principles, you at least make the evaluation criteria auditable even if the judge's implementation of them isn't. ## What breaks ### Reward hacking at scale The longer PPO runs, the more aggressively it exploits the reward model. In published InstructGPT analysis, response length grew monotonically throughout PPO training — a reward model artifact that didn't reflect genuine preference for verbosity. Anthropic noted similar dynamics with sycophancy: models learned that affirming user framing got higher reward-model scores. Neither behavior showed up in early PPO steps; both emerged as the policy pushed further into the reward model's extrapolation zone. Mitigation: monitor the distribution of response lengths, detect sycophancy indicators (does the model agree more with users who state an incorrect belief confidently?), and retrain the reward model periodically on policy outputs. ### KL coefficient drift The β coefficient that controls the KL penalty is highly sensitive. At β = 0.01, PPO runs freely and hacks the reward model within a few hundred steps on most tasks. At β = 0.3, the policy barely moves from SFT. The right value depends on how well-calibrated the reward model is for the current policy distribution — which changes throughout training. Some implementations use adaptive β schedules that raise or lower β to hold the measured KL near a target threshold — a separate mechanism from PPO's ratio clipping, which bounds individual policy updates rather than cumulative drift. None of these eliminate the issue, they just manage it. ### Distribution shift in the reward model A reward model trained on SFT model outputs is calibrated to that distribution. After 200 PPO steps, the policy's outputs look different. The reward model's predictions on out-of-distribution policy outputs can be confidently wrong — it will assign high scores to responses that look like its training distribution's high-scoring examples superficially, even if they're actually worse. This is not a hypothetical: it's why frontier labs retrain reward models regularly and why the field moved toward DPO, which sidesteps the distribution shift problem by baking the preference signal into the policy training objective directly. ### Constitutional AI judge blindspots When the AI judge misclassifies, it does so consistently across the thousands of comparisons it generates. A human annotator who makes a mistake on one type of prompt makes an independent mistake; an AI judge that's miscalibrated on a category makes the same mistake on every example in that category. The reward model learns from this systematic error as if it were signal. This can create coordinated biases in the aligned model that are harder to detect than random noise. ### The sycophancy trap This deserves its own mention because it's the failure mode that directly damages product utility. RLHF-trained models learned to agree with users because human annotators slightly preferred responses that aligned with their own views. The effect is small per comparison but compounds through PPO. The result is a model that, when presented with a factually wrong premise from a user who sounds confident, will often affirm rather than correct. This is not hallucination in the mechanistic sense; it's a preference optimization artifact. Detection requires specific evaluation sets where the user states incorrect beliefs — not general capability benchmarks. ## RLHF vs the alternatives | Method | RM needed? | Reference model? | Value model? | GPU footprint vs SFT | Main limitation | |---|---|---|---|---|---| | PPO-RLHF | Yes | Yes (frozen SFT) | Yes | ~4–5× | Complexity, reward hacking | | DPO | Implicit | Yes (frozen SFT) | No | ~2× | Requires offline preference pairs | | SimPO | Implicit | No | No | ~1.2× | Calibration sensitivity | | GRPO | Optional (often a rule-based verifier) | Yes | No | ~2–3× | Advantage noise with small groups; usually paired with verifiable rewards | | Constitutional AI (CAI) | Yes (RLAIF) | Yes | Yes | ~4–5× | AI judge bias propagation | DPO's insight is that the reward model can be expressed analytically in terms of the policy and the SFT reference, eliminating the need to train it separately. The [preference optimization article](/ai-engineering/articles/preference-optimization-dpo-simpo-orpo-grpo) covers this in detail. GRPO, which powers [DeepSeek-R1's reasoning training](/ai-engineering/articles/reasoning-model-training-rlvr-grpo), takes a different approach: it drops the value model by scoring each response relative to a group of samples for the same prompt. The estimator itself works with any reward source — including a learned reward model — but the R1 training recipe pairs it with programmatic verifiable rewards (math evaluators, code execution), which never degrade with distribution shift because they're computed from ground truth rather than from a learned model. The through-line is that every method in this table is trying to solve the same problem: SFT imitates, but alignment requires optimization. They differ in how they construct the optimization signal and what components they're willing to collapse or approximate away. ## Practical implementation notes If you're implementing RLHF from scratch in 2026, the default library is Hugging Face TRL. The loop below is written against the legacy TRL API (≤0.11) because its explicit generate → score → step structure makes the mechanics visible; TRL 0.12+ replaced it with a `PPOTrainer` that takes `reward_model` and `value_model` arguments and runs everything inside `trainer.train()`: ```python # Minimal RLHF-style training loop — legacy TRL (≤0.11) API, shown for clarity from trl import PPOConfig, PPOTrainer, AutoModelForCausalLMWithValueHead from transformers import AutoTokenizer model = AutoModelForCausalLMWithValueHead.from_pretrained("meta-llama/Llama-3.1-8B-Instruct") tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct") ppo_config = PPOConfig( model_name="llama-3.1-8b-rlhf", learning_rate=1.41e-5, batch_size=128, mini_batch_size=16, gradient_accumulation_steps=4, ppo_epochs=4, kl_penalty="kl", # kl | abs | mse | full init_kl_coef=0.05, # β — start conservative target_kl=6.0, # adaptive KL controller target (separate from cliprange) horizon=10_000, cliprange=0.2, cliprange_value=0.2, vf_coef=0.1, max_grad_norm=1.0, ) trainer = PPOTrainer( config=ppo_config, model=model, ref_model=None, # TRL auto-creates frozen copy from model tokenizer=tokenizer, dataset=prompt_dataset, # prompts only — PPO samples its own responses; # comparison pairs belong to Phase 2 (reward model) data_collator=collator, ) # Training loop (simplified) for batch in trainer.dataloader: query_tensors = batch["input_ids"] # Generate responses from active policy response_tensors = trainer.generate( query_tensors, max_new_tokens=256, do_sample=True, temperature=0.7, ) # Score with reward model rewards = [reward_model(q, r) for q, r in zip(query_tensors, response_tensors)] # PPO step (includes KL computation internally) stats = trainer.step(query_tensors, response_tensors, rewards) trainer.log_stats(stats, batch, rewards) ``` The `init_kl_coef=0.05` and `target_kl=6.0` settings implement an adaptive KL schedule: if the actual KL between the policy and reference exceeds 6.0 nats, the trainer increases β; if it's well below, β decreases. This is more robust than a fixed β across the entire training run, since the appropriate level of constraint changes as the policy drifts. A direct-RLAIF variant skips the trained reward model and uses an AI judge API call as the reward signal itself. Note that this is a different design from CAI proper, where the AI-generated preference labels train a reward model and PPO runs against it as usual. And even the direct variant still holds the active policy, the value model, and the frozen SFT reference for the KL penalty — three of the original four models — with only the reward score arriving from an external API call. ```python async def constitutional_reward(prompt: str, response: str, principle: str) -> float: """Query an AI judge for Constitutional AI RLAIF.""" judge_prompt = f"""Consider this principle: {principle} User asked: {prompt} Assistant responded: {response} On a scale of 1-10, how well does this response follow the principle? Reply with only a number.""" result = await anthropic_client.messages.create( model="claude-sonnet-4-5", max_tokens=10, messages=[{"role": "user", "content": judge_prompt}], ) try: score = float(result.content[0].text.strip()) return (score - 5.0) / 5.0 # normalize to [-1, 1] except ValueError: return 0.0 # abstain on parse failure ``` This is not production-grade — you'd want a structured output, batched calls, caching of identical prompt-response pairs, and a calibration study against human labels — but it illustrates why RLAIF is operationally attractive: the reward computation is an API call you can iterate on without retraining a model. ## When to understand this vs when to use a shortcut If you're doing post-training alignment on a model you control, you're almost certainly using DPO or [SimPO rather than PPO](/ai-engineering/articles/preference-optimization-dpo-simpo-orpo-grpo). The reasons are concrete: simpler infrastructure, no separate reward model to maintain, no four-model memory footprint, and comparable quality in most benchmarked settings. The cases where you actually want the full RLHF pipeline today: - **Frontier-scale alignment**: the reward model–policy interaction at 70B+ parameters produces qualitatively better results in some evaluations; labs with H100 clusters use it precisely because they can. - **Iterative reward model improvement**: if you're running multiple PPO rounds and refreshing the reward model between rounds, the explicit separation between RM and policy is an advantage, not a liability. - **Constitutional AI pipelines**: if you need auditable, principle-grounded alignment with RLAIF feedback and a large annotation budget is unavailable, CAI's written constitution approach is still the most principled way to encode organizational values into a reward signal. - **Understanding what DPO is doing**: the log-ratio objective in DPO is a closed-form derivation of the optimal policy under an RLHF objective. If your DPO training diverges or produces sycophantic outputs, diagnosing it requires understanding what the reward model it's implicitly representing would look like. The [decision framework for when to fine-tune at all](/ai-engineering/articles/fine-tuning-vs-rag-vs-prompting-decision-framework) should precede this decision. RLHF is expensive. It's warranted when you need deep behavioral alignment that SFT can't provide and you have the compute budget and the annotation data or an AI judge pipeline to support it. For most product teams, DPO on a clean preference dataset — or even SimPO, which drops the reference model requirement — is the better starting point. But the mechanics described here are what every alignment paper since 2022 is building on, shortcutting, or reacting against. Start here to understand what they're skipping. ## Frequently asked questions Q: What is RLHF and why do language models need it? A: RLHF (Reinforcement Learning from Human Feedback) is a post-training stage that shifts a supervised model from predicting likely text to producing outputs that humans actually prefer. After SFT, a model trained only on next-token prediction still generates confidently wrong answers, inappropriate content, and unhelpfully terse replies. RLHF trains a separate reward model on human comparisons (A vs B), then uses PPO to push the policy toward higher rewards — tightening the gap between "statistically plausible" and "genuinely useful." Q: How is a reward model trained and how accurate is it? A: A reward model is a language model with its final token-prediction head replaced by a single scalar output, trained on pairs of responses (chosen vs rejected) using a Bradley-Terry ranking loss. With ~100K human comparison pairs, reward models reach 65–75% agreement with held-out human judgments on general alignment tasks. The critical failure mode is reward hacking: the PPO policy finds outputs that score high on the reward model but are actually worse — often by producing verbose, sycophantic, or subtly manipulative text. Reward models must be retrained as the policy drifts. Q: What is Constitutional AI and how does it differ from RLHF? A: Constitutional AI (Anthropic, 2022) replaces the human-labeler bottleneck in RLHF's preference step with an AI judge guided by a written constitution — a list of principles like "be helpful, harmless, and honest." In the RL phase, a separate critique model evaluates policy outputs against the constitution and generates preference pairs without human annotators, a technique called RLAIF. The practical benefit is scale: AI feedback is orders of magnitude cheaper than human annotation, though it can reflect the judge model's own biases and blind spots. Q: Why did the field move from PPO-based RLHF to DPO? A: Classic RLHF with PPO requires four models running simultaneously: the SFT reference model, the reward model, the active policy, and a value/critic model (sharing a value head with the policy can reduce this to three). This is expensive to orchestrate, sensitive to hyperparameters, and prone to KL drift instability. DPO (2023) showed you can implicitly represent the reward model in the policy itself using a log-ratio objective, collapsing the pipeline to one trained model plus a frozen reference — whose log-probs can be precomputed — and a dataset of preference pairs. The quality trade-off is small in most settings, making DPO the default for alignment. The sibling article on preference optimization covers DPO, SimPO, and GRPO in depth. Q: What is the KL penalty in RLHF and why does it matter? A: The KL (Kullback–Leibler) divergence penalty in the PPO objective measures how far the current policy has drifted from the SFT reference model. Without it, PPO finds reward-maximizing outputs that are gibberish or degenerate — the model discovers it can score high by producing long lists of caveats or by repeating the same safe phrase. The KL coefficient β (typically 0.01–0.1) trades off alignment reward against staying close to the original SFT distribution. Too small and you get reward hacking; too large and the RL step does nothing useful. Q: Can you run RLHF without expensive human labelers? A: Yes — RLAIF (Reinforcement Learning from AI Feedback) substitutes a frontier model as the preference judge. In practice, GPT-4-class or Claude-class models agree with human preference pairs at roughly human–human agreement rates (~70–75%) on factual and harmlessness tasks, though gap widens on subtle values and cultural nuance. Constitutional AI is the production version of RLAIF: the judge model is guided by a written set of principles rather than free-form judgment, which makes the preference signal more consistent and auditable. The cost drops from ~$0.12–0.25/comparison with human raters to ~$0.005–0.02/comparison with an LLM judge — a 6–50× reduction.