The Post-Training Frontier: RLHF & DPO

What You'll Master

LLM Training Overview

Building a useful language model is not one step — it's a four-stage pipeline. Each stage narrows the gap between "a thing that predicts text" and "a thing that helps humans." The stages differ in data quantity, data quality, and training objective.

What Pretraining Learns

A language model trained to predict the next word discovers far more than grammar. The pressure to minimize perplexity forces the model to build internal representations of the world. Let's catalog exactly what emerges.

Capabilities That Emerge from Next-Token Prediction

Syntactic structure: Subject-verb agreement across clauses, long-range dependencies, correct pronoun binding. Not memorized — generalized to novel sentences.

Factual knowledge: To predict "The capital of France is ___", the model must store the fact. Billions of such facts get compressed into parameters. It becomes a noisy encyclopaedia.

Coreference resolution: "The trophy doesn't fit in the suitcase because it is too big." Resolving "it" requires physical reasoning.

Lexical semantics & sentiment: Word relationships, analogy patterns, valence of statements.

Basic arithmetic: Addition, subtraction of small numbers. Breaks down for large operands (limited by positional encoding, not lack of "understanding").

Code synthesis: Patterns of programming: function definitions, loop structures, type annotations. Predicting next-token in code = informal type checking.

Language Models as World Models

Andreas (2022) argues that language models trained on text about agents' beliefs, goals, and actions develop implicit world models. The argument:

To predict what a character says next in a story, you must model their beliefs (what they know), goals (what they want), and actions (what they'd do given beliefs and goals). Next-token prediction is world-modeling, just compressed into the language domain.

The Alignment Gap

Here's the fundamental problem. Language modeling optimizes:

But what we want the model to optimize:

Language modeling ≠ assisting users. This mismatch is the alignment gap.

The GPT-3 "Explain the Moon Landing" Example

Ouyang et al. (2022) demonstrated this gap perfectly:

The model isn't stupid. It literally doesn't know you want an answer. In its training data, this pattern → more patterns. Finetuning fixes this.

Instruction Finetuning

The simplest fix: show the model examples of instructions paired with correct responses, then maximize likelihood on those. Same loss as pre-training, different data.

Scaling Up: Many Tasks

Key insight (Flan, T0, SuperNaturalInstructions): finetune on many diverse tasks, evaluate on unseen tasks. Generalization emerges from diversity.

The Limitation of SFT

You can only finetune on instructions for which you can write gold answers. But many preferences are:

• Contextual: "be concise" for simple questions, "be thorough" for complex ones.

• Comparative: humans can say "A is better than B" even when they can't write the perfect response.

• Implicit: users want certain tone, formatting, level of hedging — hard to specify.

This is where preference learning enters.

The RLHF Pipeline

RLHF (Christiano et al. 2017; Ouyang et al. 2022) has three stages. Think of it as building a critic, then using the critic to improve the actor.

Stage 1: Preference Data

The format is simple. For each comparison:

Humans compare responses. They don't need to write the perfect response — just judge which of two candidates is better. This is much easier: comparison is cheaper than generation.

Stage 2: The Reward Model

A neural network r_φ(x, y) → scalar. Takes a (prompt, response) pair and outputs a single number: "how good is this response?" Typically initialized from the SFT model with a linear head replacing the LM head.

Stage 3: RL Optimization

Use PPO to maximize:

We'll derive each piece rigorously in the next sections.

Reward Model Training

The reward model turns human comparisons into a differentiable signal. The key question: given that a human preferred y_w over y_l, what loss function trains r(x,y) to reproduce this preference?

The Bradley-Terry Model

Bradley and Terry (1952) proposed a model for pairwise comparisons. The probability that item A beats item B is determined by their "strengths":

The Reward Model Loss

Given a dataset of preferences D = {(x⁽ⁱ⁾, y_w⁽ⁱ⁾, y_l⁽ⁱ⁾)}, maximize log-likelihood:

Hand Calculation: Reward Model in Action

The KL-Constrained Objective

Now we have a reward model. The naive approach: just maximize reward.

Problem: the policy will exploit the reward model. It finds adversarial inputs — responses that score high on the learned reward but are gibberish or repetitive. This is reward hacking.

The Fix: KL Penalty

Keep the policy close to the reference (SFT) model. The complete RLHF objective:

The KL divergence term measures how far π_θ has drifted from π_ref. Expanding:

Deriving the Optimal Policy

What's the best possible policy under this objective? We can solve it in closed form.

Hand Calculation: Boltzmann Reweighting

DPO: Direct Preference Optimization

The RLHF pipeline works, but it's complex: three models, PPO with reward shaping, careful hyperparameter tuning. Rafailov et al. (2023) asked: can we skip the reward model entirely?

The answer is yes. The key insight: since we know the closed-form optimal policy, we can reparameterize the reward in terms of the policy — then substitute directly into the preference model.

The DPO Derivation

Understanding the DPO Gradient

Define the implicit reward of response y under the current policy:

Then DPO loss = −log σ(r̂_θ(x, y_w) − r̂_θ(x, y_l)). It pushes the implicit reward of winners above losers — exactly like training a reward model, except the "reward model" is the policy.

DPO in 5 Lines of PyTorch

python
def dpo_loss(pi_logps_w, pi_logps_l, ref_logps_w, ref_logps_l, beta):
    """Each input: log P(response | prompt) summed over tokens."""
    pi_ratio_w = pi_logps_w - ref_logps_w  # log(pi/ref) for winner
    pi_ratio_l = pi_logps_l - ref_logps_l  # log(pi/ref) for loser
    logits = beta * (pi_ratio_w - pi_ratio_l)
    return -F.logsigmoid(logits).mean()

That's it. Five lines. No reward model, no value function, no GAE, no PPO clipping. Just forward-pass your policy on both responses, forward-pass the reference on both, compute ratios, sigmoid, backprop.

Hand Calculation: DPO Loss

DPO vs RLHF Comparison

Both methods optimize the same objective (KL-constrained reward maximization). They differ in how they get there.

The Theoretical Equivalence

Under infinite data and perfect optimization, DPO and RLHF converge to the same solution: the Boltzmann optimal policy. They differ in finite-sample behavior:

• RLHF benefits from online generation: the policy generates new responses during training, getting preference signal on its own outputs. This covers the distribution it actually operates on.

• DPO trains on fixed offline data. If the preference dataset was generated by a very different policy, DPO may learn slowly on regions its current policy visits. This is the distribution mismatch problem.

Beyond DPO: Modern Methods

DPO opened the floodgates. Researchers asked: can we do even better by changing the preference model, the reference constraint, or the loss function?

IPO: Identity Preference Optimization

Azar et al. (2023) observed that DPO can overfit — driving the implicit reward gap to infinity on training pairs. IPO replaces the log-sigmoid with a squared loss that saturates:

This targets a finite margin (1/2β) instead of pushing the margin to infinity. Prevents overfitting on easy pairs.

SimPO: Simple Preference Optimization

Meng et al. (2024) asked: do we even need the reference model? SimPO uses length-normalized log-probabilities as the implicit reward, with no π_ref:

KTO: Kahneman-Tversky Optimization

Ethayarajh et al. (2024): what if you don't have paired preferences (y_w, y_l for same prompt)? KTO works with unpaired data: just "this response is good" or "this response is bad."

ORPO: Odds Ratio Preference Optimization

Hong et al. (2024): combine SFT and alignment in a single loss. The odds ratio of generating y_w vs y_l directly penalizes the loser:

Frontier Post-Training

What's happening at the cutting edge? Three major trends:

1. RLAIF / Constitutional AI

Bai et al. (2022): replace human annotators with an AI judge. The AI Feedback loop:

• Write a constitution (set of principles: be helpful, be harmless, be honest).

• Generate response pairs.

• Ask a separate LLM to judge which response better follows the constitution.

• Train on the AI-generated preferences.

2. Iterative / Online DPO

Standard DPO trains on a fixed preference dataset. But preferences from an old policy may not cover what the new policy generates. Online DPO closes this gap:

This gives DPO the online learning benefit of RLHF while keeping its implementation simplicity.

3. Process Reward Models

Instead of scoring the final answer, score each step in the reasoning chain. Lightman et al. (2023) showed that process-level supervision dramatically improves math reasoning — the model learns which reasoning steps are valid, not just whether the final answer is correct.

Summary & Connections

The Complete Pipeline

Key Formulas Cheat Sheet

Connections to Other Lessons

• Policy Gradients: RLHF uses PPO (a policy gradient method with clipping). Everything from Lecture 3 — surrogate objectives, baselines, KL constraints — directly applies to the RL stage.

• Reward Learning: The reward model is trained via maximum likelihood on pairwise comparisons. Same framework as learning reward functions from demonstrations in robotics.

• Offline RL: DPO can be viewed as a form of offline RL — learning from a fixed dataset of (state, action, preference) without environment interaction.

• KL-Regularized RL: The KL-constrained objective appears throughout RL: TRPO, PPO, SAC (entropy regularization). The Boltzmann policy is the optimal solution to all of these under different names.