Understand Reward Models
& Alignment

Chapter 1: Reward Modeling

How do you teach a model what "good" means? You ask humans. Given a prompt, generate two candidate responses A and B. Show them to a human annotator who picks the better one. Collect thousands of these preference pairs and train a separate model — the reward model — to predict which response a human would prefer.

The reward model takes a (prompt, response) pair and outputs a scalar score. Higher score = more likely to be preferred. The key insight: we don't need humans to assign absolute scores (that's noisy and inconsistent). We only need relative judgments: "A is better than B." This is the Bradley-Terry model:

where σ is the sigmoid function and r(·) is the reward model's score. The loss function pushes the reward for the preferred response above the rejected one:

Reward Model Architecture

The reward model is typically the same architecture as the LLM itself, but with the language modeling head replaced by a scalar head. The data flow:

For a 7B reward model with d_model=4096: the entire backbone processes the prompt+response, the last token's representation is pooled, and a single linear layer projects 4096 dimensions down to 1 scalar. The backbone is usually initialized from the SFT checkpoint — it already understands language; it just needs to learn to evaluate quality.

Training the Reward Model

Each training example is a tuple: (prompt, chosen_response, rejected_response). Forward both responses through the RM to get r(chosen) and r(rejected). The Bradley-Terry loss pushes the gap apart:

python
r_chosen  = reward_model(prompt + chosen)     # scalar
r_rejected = reward_model(prompt + rejected)  # scalar
loss = -torch.log(torch.sigmoid(r_chosen - r_rejected))
loss.backward()

Typical dataset sizes: 50K–500K preference pairs. Training: 1–3 epochs, learning rate ~1e-5, batch size 64. The RM converges quickly because it's initialized from a strong LLM — most of the work is already done. You're just teaching the final layer what "quality" means.

Chapter 2: The RLHF Pipeline

RLHF (Reinforcement Learning from Human Feedback) is a three-stage pipeline that transforms a base model into an aligned assistant. Each stage builds on the last:

Stage 1 (SFT) gives the model the right "shape" — it learns to follow instructions, use a helpful tone, and structure answers clearly. But SFT alone is limited by the quality and diversity of demonstrations.

Stage 2 (Reward Model) captures nuanced preferences that are hard to demonstrate. It's easier to judge than to demonstrate: a human can quickly say "A is better than B" even if writing the perfect response from scratch would take much longer.

Stage 3 (PPO) is where the magic happens. The policy model generates responses, the reward model scores them, and PPO updates the policy to produce higher-scoring outputs. A KL penalty prevents the model from drifting too far from the SFT checkpoint.

What's Running During PPO

The PPO training loop is surprisingly GPU-hungry. At any given moment, you need four models in memory:

For a 7B model, that's roughly 4 × 14 GB = 56 GB just for weights in fp16 — before any activations or optimizer states. This is why RLHF is expensive and why simpler alternatives like DPO are appealing.

Chapter 3: PPO for LLMs

Proximal Policy Optimization (PPO) is the workhorse RL algorithm behind RLHF. The idea: generate a response, score it with the reward model, then nudge the policy to make high-reward responses more likely — but not too much per step.

PPO uses a clipped surrogate objective. The ratio r_t = π(a|s) / π_old(a|s) measures how much the policy has changed. PPO clips this ratio to [1−ε, 1+ε], preventing destructive updates:

But there's a critical addition for LLMs: a KL penalty that keeps the policy close to a reference model (usually the SFT checkpoint). Without it, the model would "overoptimize" — finding weird outputs that score high on the reward model but are gibberish to humans.

The β Tradeoff in Practice

The total reward for a generated response is R_total = r_model(y) − β · KL(π || π_ref). The coefficient β is the most important hyperparameter in RLHF training:

• β too high (e.g., 0.5): The KL penalty dominates. The model barely deviates from the SFT checkpoint. You pay all the compute of RL but learn almost nothing.
• β too low (e.g., 0.001): The model is free to drift far from π_ref. It discovers reward hacking strategies — adversarial outputs that score high on the reward model but are gibberish or sycophantic to humans.
• β just right (typically 0.01–0.1): The model improves meaningfully while staying in the "trustworthy" region where the reward model's scores are meaningful.

In practice, teams often anneal β during training — starting high for stability, then lowering it as the reward model proves reliable.

One PPO Iteration, Step by Step

Chapter 4: DPO — Direct Preference Optimization

What if we could skip the reward model entirely? DPO (Rafailov et al., 2023) makes a beautiful mathematical observation: the optimal policy under the RLHF objective has a closed-form solution in terms of the reward function. By rearranging terms, we can express the reward implicitly through the policy itself:

Substituting this into the Bradley-Terry preference model gives the DPO loss directly in terms of the policy log-probabilities — no reward model needed:

where y_w is the preferred (winning) response and y_l is the rejected (losing) response. DPO increases the relative log-probability of preferred responses while decreasing that of rejected ones, all while staying anchored to the reference model through the log-ratio terms.

DPO in Practice

Training DPO requires computing four forward passes per batch: π(y_w|x), π(y_l|x), π_ref(y_w|x), π_ref(y_l|x). The reference model is frozen (no gradients). Each forward pass returns the log-probability of the response: sum of log-probs per token. The loss is a single sigmoid cross-entropy on the gap between the two log-ratios. No sampling, no reward model, no RL loop. Just standard supervised training with a clever loss.

Typical hyperparameters: β = 0.1–0.5 (controls deviation from reference), learning rate 1e-6 to 5e-7 (very low — you're fine-tuning a fine-tuned model), 1–3 epochs over the preference dataset. Memory: only 2 models (policy + frozen reference) instead of 4 for RLHF.

The Implicit Reward

Here's the key derivation that makes DPO work. The optimal policy under the KL-constrained RLHF objective is:

Rearranging for r gives us: r(x,y) = β log(π(y|x)/π_ref(y|x)) + β log Z(x). The partition function Z(x) cancels when we take the difference r(y_w) − r(y_l) inside the Bradley-Terry model. This is why DPO works: it's implicitly computing a reward function where the reward of any response is just β times its log-probability ratio relative to the reference. Higher probability under π (relative to π_ref) = higher implicit reward.

python
import torch
import torch.nn.functional as F

def dpo_loss(
    policy_chosen_logps: torch.Tensor,
    policy_rejected_logps: torch.Tensor,
    ref_chosen_logps: torch.Tensor,
    ref_rejected_logps: torch.Tensor,
    beta: float = 0.1
) -> torch.Tensor:
    # Log-ratios: how much more likely under policy vs reference
    chosen_logratios = policy_chosen_logps - ref_chosen_logps
    rejected_logratios = policy_rejected_logps - ref_rejected_logps

    # The "implicit reward gap" scaled by beta
    logits = beta * (chosen_logratios - rejected_logratios)

    # Negative log-sigmoid = binary cross-entropy with label=1
    loss = -F.logsigmoid(logits).mean()
    return loss

Chapter 5: KTO & Simpler Methods

DPO still requires paired preferences (A vs B for the same prompt). But what if you only have binary feedback — thumbs up or thumbs down on individual responses? KTO (Kahneman-Tversky Optimization) works with unpaired data by leveraging prospect theory: humans feel losses more strongly than equivalent gains.

The KTO loss treats desirable and undesirable outputs asymmetrically. For a good response, it encourages the log-ratio to be high. For a bad response, it penalizes the log-ratio being high. The asymmetry mirrors human loss aversion:

ORPO (Odds Ratio Preference Optimization) takes yet another approach: it combines the SFT loss with a preference signal in a single training objective, using the odds ratio of generating preferred vs rejected responses.

The Simplicity Gradient

Each method trades off against the others in a clear pattern:

• RLHF: Most flexible (any reward signal), but 4 models in memory + RL instability. Used by OpenAI for GPT-4, Anthropic for Claude.
• DPO: Mathematically equivalent to RLHF under Bradley-Terry, but only 2 models + standard supervised training. Used by Meta for Llama 3.
• KTO: Works with thumbs-up/down data (cheaper to collect than pairs). Slightly worse empirically but much easier data collection.
• ORPO: Single-stage training combining SFT and preference in one loss. Simplest pipeline but least studied at scale.

Chapter 6: Constitutional AI

Human annotation is expensive and doesn't scale. Anthropic's Constitutional AI (CAI) asks: can the AI critique and revise itself using a set of written principles (a "constitution")? The answer is yes, and it works surprisingly well.

CAI has two phases. In the critique-revision phase, the model generates a response, then is asked to evaluate it against principles like "is this harmful?" or "is this honest?" and rewrite it. The revised responses become the SFT training data. In the RLAIF phase (RL from AI Feedback), the AI itself generates preference labels instead of human annotators.

Chapter 7: Process Reward Models

Standard reward models (Outcome Reward Models, or ORMs) score the final answer. But for multi-step reasoning, the final answer might be right for the wrong reasons — or wrong because of a single bad step in an otherwise sound chain. Process Reward Models (PRMs) score each step of the reasoning process.

PRMs provide much denser supervision. Instead of one score for the whole response, you get a score per step. This helps in at least three ways: (1) better credit assignment (which step went wrong?), (2) more training signal per example, and (3) the ability to do tree search over reasoning paths at inference time.

Chapter 8: Reward Hacking

Here's the dark side of optimization: the model will find loopholes. If the reward model gives higher scores to longer responses, the model learns to be verbose. If the reward model prefers confident-sounding text, the model learns to sound confident even when wrong. This is reward hacking — the policy exploits imperfections in the proxy reward.

Common failure modes include:

• Sycophancy — agreeing with the user even when they're wrong, because agreement gets higher reward.
• Length bias — generating unnecessarily long responses because the RM associates length with quality.
• Hedging — giving vague, noncommittal answers that avoid being clearly wrong.
• Format gaming — using bullet points, bold text, or other formatting tricks that correlate with high scores.

The Overoptimization Curve

There's a characteristic pattern in RLHF training. Plot proxy reward (from the RM) and true reward (from held-out human evaluations) against KL divergence from the reference. Early on, both curves rise together — the model genuinely improves. But past a critical KL threshold, proxy reward keeps climbing while true reward peaks and then drops. The model has found adversarial inputs that game the RM.

Gao et al. (2022) showed this empirically across multiple model sizes. The critical KL scales roughly with model capacity: larger models can exploit RMs more aggressively. This is why β must be tuned per model size — there's no universal value.

Mitigation Strategies

Chapter 9: Open Problems

Alignment research is far from solved. As models become more capable, the challenges deepen. Here are the frontier problems that keep researchers up at night:

Scalable Oversight

How do you supervise a model that can perform tasks beyond human ability? If the model writes code that's too complex for the annotator to evaluate, or reasons about problems the human can't verify, the entire preference-based framework breaks down. Proposals include recursive reward modeling (use AI to help humans evaluate AI), debate (two AIs argue, human judges), and market-based mechanisms.

Weak-to-Strong Generalization

Can a weaker model supervise a stronger one? If GPT-2-level labels are used to fine-tune a GPT-4-level model, can the strong model generalize beyond the quality of its supervisor? Early results from OpenAI suggest this partially works — the strong model can "figure out" what the weak supervisor meant, analogous to a smart student learning from an imperfect teacher. But it doesn't fully close the gap.

Interpretability for Alignment

Can we look inside the model and verify its "values" directly? If we could read the model's internal representations and confirm it's being helpful for the right reasons (not just gaming the reward), that would be far more robust than any behavioral test. Mechanistic interpretability aims to make this possible.