Alignment: DPO and Intrinsic Curiosity

Chapter 1: The Closed-Form Policy

Chapter 0 ended on a promise: the KL-constrained reward-maximization objective has an exact, closed-form optimizer. This chapter proves it — the full derivation from Appendix A.1 of the paper, with every algebraic step shown, because “it is straightforward to show” (the paper's own phrase) is exactly the kind of sentence this course refuses to leave unexplained.

Setting up the objective

We start from the same objective as Chapter 0, but now think of it abstractly: for any reward function r(x,y), reference policy π_ref, and unconstrained policy class, what π maximizes

The KL-divergence term, written out, is 𝔼_y∼π[ log( π(y|x) / π_ref(y|x) ) ] — the expected log-ratio between the new policy's probability and the reference's probability for the same response. Substitute that definition in and pull both expectations under one E over y ∼ π(y|x):

Before the algebra: what a KL-divergence number actually feels like

It's easy to let 𝔻_KL stay an abstract symbol. Push one small example through it by hand first, so the algebra below has a concrete referent. Suppose there are only three possible responses to some toy prompt, and after a little RL the policy has shifted its probabilities slightly away from the reference:

𝔻_KL(π||π_ref) = ∑_y π(y) log(π(y)/π_ref(y)), summed term by term, using the natural log the way the paper's derivation does throughout:

Every term where π put more mass than π_ref contributed positively; every term where it put less mass contributed negatively; and the sum still came out strictly positive, because π is genuinely a different (more concentrated) distribution than π_ref. Recompute it the other direction — 𝔻_KL(π_ref||π) — and you'd get a different number entirely (try it: it comes out to about 0.0765 nats), which is the standard warning about KL: it is not a symmetric distance, it's a directed measure of “how surprised would you be, on average, using π_ref as your model of the world, if reality were actually distributed like π.” This is the number that grows every time πθ drifts further from π_ref in Chapter 0's objective, and it's the same number this chapter is about to prove has a closed-form minimizer.

Substitute the formal KL definition into the objective, and pull both expectations under one E over y ∼ π(y|x):

Now do two purely mechanical moves. First, flip the sign and turn the maximization into a minimization (maximizing f is the same as minimizing −f). Second, divide the bracket by β and pull the β out front — a maximization problem's optimal argument doesn't change if you scale the whole objective by a positive constant, so we can freely multiply through by 1/β:

That single algebraic step — sign flip plus a rescale — is the entire “trick.” What remains is to notice that the bracket looks almost like a KL-divergence between π and something, except for that stray −(1/β)r(x,y) term sitting where a second log-probability should be. The rest of the derivation is about constructing exactly the right “something” so the bracket becomes a real KL-divergence, term for term.

Manufacturing the missing distribution

Define a new quantity, the partition function:

This is a sum over every possible response y, of the reference probability times an exponentiated, temperature-scaled reward. It does not depend on which response we're currently looking at — only on the prompt x. That single property (no y-dependence) is the entire reason it will later cancel out of the preference model in Chapter 2, so hold onto it. For now, use Z(x) to define a candidate distribution:

Check that this is a legal probability distribution: every factor is positive (a probability times an exponential is always positive), and dividing by Z(x) — which is exactly the sum of the numerator over all y — guarantees ∑_y π*(y|x) = 1 by construction. That's the classic normalize-by-the-partition-function move from statistical mechanics, and it's the only reason Z(x) exists: it's a bookkeeping device to make π* sum to one.

Now substitute π_ref(y|x)·exp((1/β)r(x,y)) = Z(x)·π*(y|x) back into the minimization objective from above:

Split the log of a quotient into a difference of logs — log(a/(bc)) = log(a/c) − log(b) — and Z(x) pulls entirely out of the inner expectation, since it doesn't depend on y:

This is the payoff line. The bracket is now exactly a KL-divergence between our policy π and the manufactured distribution π*, minus a term (log Z(x)) that doesn't depend on π at all. Since we're minimizing over π, the log Z(x) term is along for the ride — it shifts the objective's value up or down uniformly but can never change which π achieves the minimum.

Why KL can never go negative, proven not asserted

That callout is doing a lot of work with a bare assertion, so let's actually prove 𝔻_KL(π||π*) ≥ 0, because the entire derivation's punchline rides on it being true, not just plausible. Start from the definition and flip the sign inside so it's phrased as a negated quantity:

The function log(·) is concave (it curves downward — picture the familiar log curve bending under any straight line drawn between two of its points). Jensen's inequality says that for any concave function f and any random variable X, the average of f applied to X is never more than f applied to the average of X: 𝔼[f(X)] ≤ f(𝔼[X]). Apply that here with X = π*(y)/π(y) and f = log:

Expand the right-hand expectation directly — it's a sum over y, weighted by π(y), of π*(y)/π(y):

The π(y) in the numerator and denominator cancel term by term, leaving exactly ∑_yπ*(y), which is 1 because π* is a legal probability distribution — the same normalization fact we checked two paragraphs ago. So the right-hand side of Jensen's inequality is log(1) = 0, which means 𝔼_y∼π[log(π*(y)/π(y))] ≤ 0, and negating both sides flips the inequality:

Equality in Jensen's inequality holds exactly when the random variable is constant almost everywhere — here, when π*(y)/π(y) is the same number for every y with π(y)>0, which combined with both distributions summing to 1 forces that constant to be exactly 1, i.e. π(y)=π*(y) everywhere. That's the rigorous version of “equals 0 iff the two distributions are identical.” Nothing here was asserted on faith; it falls out of one property of the log function (concavity) and one property of π* (it sums to one), the same two facts used to build π* in the first place.

So the optimal policy for the KL-constrained reward-maximization problem, for any reward function r(x,y), is:

Read this formula the way you'd read a physical law, because that's exactly its lineage — it has the shape of a Boltzmann distribution from statistical mechanics, where reward plays the role of negative energy and β plays the role of inverse temperature. In plain language: start from the reference policy's distribution over responses, then re-weight every response by how exponentially good its reward is, scaled by 1/β, then renormalize. High-reward responses get boosted; low-reward responses get suppressed; but nothing that had zero probability under π_ref can ever get positive probability under π_r — the reference model sets the support, the reward only reshapes the mass within it.

Why this formula, by itself, is useless

You might think Chapter 0's problem is already solved: just compute Eq. 4 and you have your optimal policy. It isn't that easy, and the paper is explicit about why: Z(x) sums over every possible response y — for language, that's every possible token sequence, an astronomically large (effectively infinite) set. There is no way to compute this sum, and no way to sample from π_r directly without already knowing Z(x). Eq. 4 tells you the shape of the answer but gives you no way to actually use it. This is exactly why the field turned to PPO in the first place: if you can't write down π_r, you have to search for it, and RL is a way of searching without ever computing Z(x).

Chapter 2 shows the actual escape route: instead of trying to compute π_r from r, DPO rearranges Eq. 4 algebraically to express r in terms of π_r, and then substitutes that expression into the Bradley-Terry preference model — at which point Z(x) will cancel algebraically, without ever being computed.

Watch what happens at the extremes. As β → ∞ (drag the slider far right), 1/β → 0, so exp((1/β)r(x,y)) → exp(0) = 1 for every y — the reward stops mattering and π_r collapses back to π_ref exactly. That's the KL leash winning completely: infinite penalty for deviation means zero deviation. As β → 0 (drag far left), 1/β explodes, and the exponential wildly amplifies whichever response has the single highest reward — π_r collapses onto a near-deterministic point mass on the best-scoring response, regardless of what π_ref thought. β is not a minor implementation detail; it is the dial between “stay exactly as I was” and “ignore everything except the reward.”

Verify the widget's default position by hand

Trust the formula, not just the animation. Set β=1.0 (the widget's default slider position) and compute π_r(y|x) for all four candidate responses by hand, using Eq. 4 with a uniform π_ref(y|x)=0.25 for every y. First, the un-normalized numerator for each response, 0.25·exp(r):

Sum these four numbers to get Z(x) — tractable here only because there are exactly four responses in this toy example, which is the entire reason real language-model Z(x) is not computable this way:

Divide each numerator by Z(x) to get the final π_r(y|x):

Check these four numbers against what the widget shows at β=1.0 — they should match to the first decimal place. Notice the reference model gave every response an equal 25% shot; the reshaped π_r still gives the wrong response a nonzero 2.0% probability (it never had zero probability under π_ref, so Eq. 4 can shrink its mass but never erase it entirely), while the best response nearly tripled its share, from 25% to 65.2%, purely from being re-weighted by its own reward relative to the other three.

Chapter 2: Killing the Reward Model

Chapter 1 left us with a formula for the optimal policy that we can't compute, because it requires an intractable sum. This chapter performs the algebraic move that makes the intractable sum disappear entirely — not by approximating it, but by arranging for it to cancel out exactly. This is Appendix A.2 of the paper, worked in full.

Step 1: invert the formula to solve for the reward

Eq. 4 expresses π_r in terms of r. We want the reverse: r in terms of π_r. Take the log of both sides of Eq. 4:

Rearrange for r(x,y) — move everything except the reward term to the other side, then multiply through by β:

Stare at this for a second, because it is the conceptual heart of the entire paper. It says: the reward that made π_r optimal can be reconstructed, up to an additive prompt-dependent constant, from nothing but the log-ratio of two policies. If you know how much more likely the optimal policy makes a response versus the reference policy, you know its reward — scaled by β. This is true for the ground-truth optimal policy π* and the ground-truth reward r* too, by the exact same algebra, since Eq. 4 held for any reward function:

Step 2: substitute into Bradley-Terry, and watch Z(x) die

Recall the Bradley-Terry preference model from Chapter 0 depends only on a difference of two rewards for the same prompt x:

Substitute the Eq. 5 expression for r* at both y₁ and y₂. Both substitutions carry the same β log Z(x) term, because Z(x) depends only on the prompt x, which is identical for both responses being compared:

The two β log Z(x) terms are identical and subtract to exactly zero. This is the whole trick, laid bare: Z(x) never had to be computed, because it was always going to cancel the moment you looked at a difference of two rewards for the same prompt instead of a single reward in isolation. Bradley-Terry only ever needs a difference — that structural fact is what makes the intractable partition function irrelevant. What survives is:

Read what this equation actually says: the probability that a human prefers y₁ over y₂ is completely determined by how much more the optimal policy π* upweights y₁ relative to y₂, compared to how the reference policy weighted them. There is no reward model anywhere in this expression. There is only a policy, a reference policy, and β. We reparameterized the reward entirely out of existence and replaced it with a policy's own log-probabilities.

A fully computable toy example: watch Z(x) actually cancel

Real language-model Z(x) can never be computed — it sums over every token sequence. But nothing stops us from building a toy world small enough to compute it directly, checking the cancellation with real numbers instead of taking Step 2's algebra on faith. Three responses, π_ref(y|x) = [0.5, 0.3, 0.2], rewards r(x,y) = [1.0, 0.0, −1.0], β=1.0. First, Z(x), the tractable way, since there are only three terms to sum:

Divide each numerator by Z(x) to get π_r(y|x) = [78.44%, 17.31%, 4.25%] (check: these sum to 100%). Now do something that should feel like cheating: shift every reward by the same constant, c=+3, so r′(x,y) = [4.0, 3.0, 2.0]. That's exactly the kind of prompt-dependent-only shift — r′(x,y) = r(x,y) + f(x) with f(x)=3, a constant here since there's only one prompt in this toy world — that Appendix A.5 of the paper calls two reward functions from the same equivalence class. Recompute Z′(x) with the shifted rewards:

Z′(x) is a completely different number — about 20× larger, exactly e³≈20.09 times larger, since every term in the sum picked up the same exp(3) multiplier. But divide through and check π_r′(y|x):

Identical to the unshifted policy, to four significant figures. This is Lemma 1 of the paper's Appendix A.5 made concrete: r and r′ differ by a function of the prompt alone, so the exp(f(x)) factor they each pick up appears in every term of Z(x) uniformly, and cancels out of every ratio the instant you divide by Z(x) to normalize. Two reward functions from the same equivalence class induce the exact same optimal policy — which means the exact same Bradley-Terry preference probabilities, since Eq. 6 is built entirely out of that policy. You could have picked any constant shift — c=3 was arbitrary — and gotten the same π_r. This is the deepest reason Z(x) is allowed to vanish from the DPO loss: not merely that the algebra happens to cancel it once, but that the entire equivalence class of reward functions it could represent collapses onto one policy, so there was never a unique “the” reward to begin with — only a family of them, all equally valid, all inducing identical behavior.

Step 3: turn the preference model into a trainable loss

Eq. 6 tells us the probability of a preference given the true optimal policy π*. We don't have π* — that's what we're trying to find. But we do have a dataset of real human preference pairs 𝒟 = {(x⁽ⁱ⁾, y_w⁽ⁱ⁾, y_l⁽ⁱ⁾)}_i=1^N, where y_w beat y_l in a real human comparison. So we do exactly what Phase 2 of RLHF did in Chapter 0 — maximum likelihood estimation — except now the thing we're fitting is a policy πθ standing in for π* inside Eq. 6, instead of a reward network. Negate the log-likelihood to get a loss to minimize:

This is the DPO loss, and everything you need to compute it is sitting in front of you right now: a batch of (prompt, winning response, losing response) triples, a policy network you can run a forward pass through to get log-probabilities, and a frozen copy of the same network (before any DPO training started) to get reference log-probabilities. That's it. No sampling during training — the y_w and y_l were already generated ahead of time, by whoever collected the preference data. No critic. No reward network. This single expression is a plain binary cross-entropy loss, computed with two forward passes and a subtraction, and it is provably equivalent to running the entire three-phase RLHF pipeline from Chapter 0 to convergence.

Concept → realization: what you'd actually implement

Compare the object you now have to build against the object described in Chapter 0. Instead of a reward network with a scalar output head, you need only your policy model loaded twice — once trainable, once frozen as π_ref — and a way to extract the sequence log-probability logπ(y|x) = ∑_t logπ(y_t | x, y_<t) by summing token log-probabilities along the completion. Everything downstream (the sigmoid, the loss, the gradient) is ordinary autodiff. Chapter 3 makes this completely concrete: a toy pair of responses, real log-probability numbers, and the DPO loss computed by hand, term by term, so you can verify your own implementation against it later.

Put the two pipelines side by side as code, and “no reward model to train” stops being a slogan and becomes a visible difference in what you have to build and maintain:

python
# Phase 2 of RLHF (Chapter 0): a SEPARATE network, its own training loop
class RewardModel(nn.Module):
    def __init__(self, backbone):
        self.backbone = backbone            # usually initialized from pi_SFT
        self.score_head = nn.Linear(backbone.hidden_size, 1)  # NEW parameters
    def forward(self, x, y):
        return self.score_head(self.backbone(x, y))    # scalar r_phi(x, y)

# train reward_model to minimize L_R from Chapter 0, THEN run PPO against it
# -- an entire extra training run, before RL fine-tuning even starts

# DPO (this chapter): NOTHING new to train separately
def dpo_step(policy, ref_policy, x, y_w, y_l, beta, optimizer):
    logp_w = policy.sequence_logprob(x, y_w) - ref_policy.sequence_logprob(x, y_w)
    logp_l = policy.sequence_logprob(x, y_l) - ref_policy.sequence_logprob(x, y_l)
    loss = -F.logsigmoid(beta * (logp_w - logp_l))
    loss.backward()          # gradients flow into `policy` only; ref_policy stays frozen
    optimizer.step()

The reward-model class needs its own architecture decision (what head to bolt on), its own training loop, its own convergence check, its own checkpoint to version and keep in sync with whichever policy it scores — an entire separate artifact with its own lifecycle. dpo_step needs nothing new: no extra parameters, no separate forward pass through a different network, no second convergence criterion to watch. It's the same policy you were already going to fine-tune, run twice (once trainable, once frozen), with a subtraction and a sigmoid in between.

Beyond pairs: what if you have a full ranking, not just a winner and a loser?

Everything above assumed preference data comes as pairs: one winner, one loser. Real annotation pipelines sometimes collect more — a human ranks K candidate responses from best to worst in one pass, which is richer information than K/2 independent pairwise comparisons. The paper doesn't leave this on the table. Appendix A.3 generalizes Bradley-Terry to the Plackett-Luce model, which assigns a probability to an entire ranking τ of K responses, not just a single winner-vs-loser call: at each step, the model picks the highest-remaining-reward response from whatever's left with probability proportional to exp(reward), then removes it and repeats for the next position in the ranking. Setting K=2 recovers exactly the Bradley-Terry model from Chapter 0 as a special case — a single “pick the winner from these two” step. The derivation runs through the identical two moves as Step 1 and Step 2 above: substitute the Eq. 5 reward-in-terms-of-policy expression into the Plackett-Luce likelihood, and Z(x) cancels for the same structural reason — it appears identically in every term of the ranking probability, since every term still shares the same prompt x. The resulting DPO-for-rankings loss is a straightforward sum over ranking positions of the same logσ(·) shape you already know. We won't need K-wise rankings anywhere in this lesson — Paprika's preference pairs, starting next chapter, are ordinary winner/loser pairs — but it's worth knowing the pairwise story isn't a special case DPO happens to handle; it's the K=2 instance of something the underlying method handles in general.

Chapter 3: The DPO Loss By Hand

Equations are convincing on paper and useless until you've pushed real numbers through them. This chapter does exactly that — one prompt, two candidate responses, real (if invented for teaching) log-probabilities, and the complete DPO loss computed by hand, matching every line of the reference implementation from Appendix B of the paper.

The scenario

Prompt x: “Write a function that reverses a string.” Two candidate completions were sampled from π_ref and a human labeled a preference between them.

Suppose we've run both responses through the current policy πθ and through the frozen reference π_ref, and summed token log-probabilities along each completion (exactly as Chapter 2 described) to get four scalar numbers:

Notice the setup deliberately mirrors a realistic mid-training snapshot: the policy has already started moving toward the concise winner (its log-prob rose from −1.60 to −1.20, i.e. it became more likely) and away from the verbose loser (from −1.30 to −2.10, less likely) — but training isn't finished, so there's still a real loss to compute and a real gradient to take. We'll use β = 0.1, the paper's own default value from Appendix B.

Step 1: the log-ratios

Compute how much each response's probability changed, policy versus reference, in log-space:

Already there's a story here: the winner's log-ratio is positive (the policy has moved toward it relative to the reference) and the loser's is negative (moved away). That's the direction DPO wants; the loss just measures how far there still is to go.

Step 2: the implicit rewards

Recall from Chapter 2 that β·log(πθ(y|x)/π_ref(y|x)) is the implicit reward r̂θ(x,y) this policy currently assigns. Multiply each log-ratio by β=0.1:

Right now, according to the policy's own implicit reward, the winner scores 0.040 and the loser scores −0.080 — the policy already ranks them correctly, but the margin (0.120) may not be as confident as the human preference deserves. That's exactly what the loss will push on.

Step 3: the margin and the loss

The DPO loss argument is the difference of implicit rewards, equivalently β times the difference of log-ratios:

Pass the margin through the sigmoid, then take the negative log — the standard binary cross-entropy shape, exactly like Phase 2's reward-modeling loss in Chapter 0:

Compare this to the loss before any training moved the policy, when πθ = π_ref exactly: every log-ratio is 0, the margin is 0, σ(0) = 0.5 exactly, and the loss is −log(0.5) = 0.693 (that's ln 2 — the loss of a perfectly uninformed 50/50 guess). Our trained policy's loss of 0.635 is lower — the policy has genuinely learned something about this preference, but it's a small step; a β=0.1, margin=0.12 nudge is nowhere near confident. That smallness is not a bug in our made-up numbers; it's DPO's actual behavior. A single preference pair moves the policy a little. Thousands of pairs, batched over many steps, are what add up to real alignment.

Verify it against the reference implementation

Here is Appendix B's PyTorch code for the DPO loss, unmodified. Confirm every number above appears inside it:

python
import torch.nn.functional as F

def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta):
    # pi_logps, ref_logps: sequence log-probs, shape (B,)
    pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs]     # -1.20, -2.10
    ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs] # -1.60, -1.30

    pi_logratios  = pi_yw_logps - pi_yl_logps    # -1.20 - (-2.10) = 0.90
    ref_logratios = ref_yw_logps - ref_yl_logps  # -1.60 - (-1.30) = -0.30

    losses  = -F.logsigmoid(beta * (pi_logratios - ref_logratios))
                                                  # beta * (0.90 - (-0.30)) = 0.1 * 1.20 = 0.120  -> matches our margin
    rewards = beta * (pi_logps - ref_logps).detach()
                                                  # r(y_w) = 0.1*0.40 = 0.040, r(y_l) = 0.1*(-0.80) = -0.080
    return losses, rewards

Notice the code computes the margin via pi_logratios − ref_logratios (a difference of differences) instead of subtracting our two per-response margins directly — algebraically these are identical: (logπθ(y_w)−logπθ(y_l)) − (logπ_ref(y_w)−logπ_ref(y_l)) rearranges to exactly (logπθ(y_w)−logπ_ref(y_w)) − (logπθ(y_l)−logπ_ref(y_l)), which is our r̂θ(y_w)/β − r̂θ(y_l)/β. Same four numbers, regrouped; same answer, 0.120 after multiplying by β.

A second worked example: run the same pair forward 500 more steps

Numbers in isolation don't show you training. Keep every other quantity fixed — π_ref is frozen, so logπ_ref(y_w)=−1.60 and logπ_ref(y_l)=−1.30 don't move — and imagine gradient descent has kept pushing on this exact pair for several hundred more steps, moving the trainable policy's log-probabilities further in the direction the gradient wants:

Redo Steps 1 through 3 with the new numbers. The log-ratios:

The implicit rewards, scaled by the same β=0.1:

The margin and loss:

Everything moved in the direction you'd hope: the margin widened from 0.120 to 0.285, and the loss dropped from 0.635 to 0.561. This isn't a new phenomenon — it's the same loss, the same pair, further along the same gradient trajectory, and it's exactly the behavior Chapter 4 is about to name precisely. Compute the gradient weight for this later step, σ(−margin) = σ(−0.285) = 1−0.5708 = 0.4292, and compare it to the weight for the earlier step you'll compute in Chapter 4: 0.4701. The weight shrank as the pair got more resolved — a preview, with real numbers already in hand, of Chapter 4's claim that DPO automatically spends less gradient budget on pairs it has already learned.

Push logπθ(y_w|x) all the way toward 0 (the policy becoming near-certain about the winner) while dragging logπθ(y_l|x) toward −40 (near-zero probability for the loser): the margin grows large and positive, σ(margin) approaches 1, and the loss approaches 0 — the policy has essentially resolved this one preference pair completely. Now try the opposite: push y_l's policy log-prob above y_w's. The margin goes negative, the sigmoid falls below 0.5, and the loss climbs past 0.693 — the policy is actively contradicting the human preference, and the gradient will push back hard.

A third worked example: the loss when the policy is actively wrong

Don't just take the widget's word for “the loss climbs past 0.693” — compute one point in that regime by hand, completing the full picture (below-baseline, at-baseline, above-baseline) with real arithmetic every time. Suppose, instead of the healthy direction Chapter 3 started with, training had pushed this pair the wrong way: the policy grew less confident about the winner and more confident about the loser.

Now the full picture, three points on the same curve, all computed by hand across this course: a healthy pair gives loss 0.635 (below the 0.693 baseline — genuine progress), an untrained pair gives exactly 0.693 (the coin-flip baseline, πθ=π_ref), and this actively-wrong pair gives 0.809 (above baseline — the policy has learned the wrong ranking and the loss says so, unambiguously, in the same units every time). Nothing about the loss function changed between these three cases; only the inputs did. That's the entire behavior of −logσ(·) laid bare: a smooth, monotonic scoreboard that rewards confidence in the right direction and punishes confidence in the wrong one, with “uninformed” sitting exactly at ln 2 in between.

Chapter 4: Secretly a Reward Model

Chapter 3 gave you numbers. This chapter gives you the mechanism — what the DPO loss's gradient actually does to the policy's weights, and why the paper's title is a literal, technical claim rather than a catchy phrase.

Deriving the gradient

Start from the loss (Eq. 7) and differentiate with respect to θ. Write u for the margin term inside the sigmoid, flipped in sign to match the derivative of logσ(·):

Before applying it, prove the identity being invoked, since “a standard sigmoid calculus fact” shouldn't mean “trust me.” Write logσ(x) explicitly, using σ(x) = 1/(1+e^−x):

Differentiate the right-hand side with the chain rule — the derivative of −log(1+e^−x) with respect to x is −1/(1+e^−x) times the derivative of (1+e^−x), which is −e^−x:

Divide numerator and denominator by e^−x and you get 1/(e^x+1), which is exactly σ(−x) by definition (substitute −x for x in σ(x)=1/(1+e^−x) and you get 1/(1+e^x), the same expression). So:

with that identity now proven rather than asserted, the chain rule through the loss produces:

Unpack this term by term, because every piece has a plain-English job.

The direction is exactly what you'd hand-design if someone asked you to write an alignment loss from scratch: push up the winner, push down the loser. The interesting part — the part that makes DPO more than “maximize logπ(y_w) minus logπ(y_l)” — is the weight σ(r̂θ(x,y_l) − r̂θ(x,y_w)) sitting in front.

The weight, using our own numbers

Plug in Chapter 3's implicit rewards: r̂θ(x,y_l) = −0.080, r̂θ(x,y_w) = 0.040.

Read this as: the model is currently wrong about this pair by roughly 47%'s-worth of sigmoid mass — it hasn't fully separated the two responses yet, so the gradient step still carries substantial weight. Now imagine a different pair, already resolved: say the current implicit rewards are r̂_w=3.0 and r̂_l=−3.0. Then weight = σ(−3−3) = σ(−6) ≈ 0.0025. The gradient step on that pair is almost zero — not because the example doesn't matter, but because the model has already learned it. DPO automatically stops spending gradient budget on preferences it has already resolved, and concentrates its updates on the pairs it currently gets wrong. The paper states this explicitly: the weight is “higher when [the] reward estimate is wrong.”

You already have a third data point for this trend, without any new arithmetic: Chapter 3's second worked example, the same pair 500 steps further into training, had implicit rewards r̂_w=0.075 and r̂_l=−0.210. Its weight is σ(−0.210−0.075) = σ(−0.285) = 0.4292 — lower than the step-0 weight of 0.4701, higher than the confidently-resolved example's 0.0025. Lay all three side by side and the pattern is unmistakable: as a pair's margin widens from 0.120 to 0.285 to 6.0, its gradient weight shrinks from 0.4701 to 0.4292 to 0.0025 — a smooth, monotonic decay, not a step function. DPO doesn't switch a pair's gradient off once it's “done”; it fades it continuously, in exact proportion to how wrong the model's current implicit ranking still is.

The title, made literal

Section 5.1 of the paper calls this “Your Language Model Is Secretly a Reward Model,” and by now you can see exactly what that means as a technical statement, not a marketing line. The quantity

is a fully-formed reward function — it takes a prompt and response and returns a scalar, exactly like rφ from Phase 2 of RLHF in Chapter 0. The difference is that it costs nothing extra to compute: it's already sitting inside the two forward passes you were doing anyway to compute the DPO loss. Once you've trained πθ with DPO, you can hand anyone this formula and they have a reward model — suitable for scoring new responses, for best-of-N reranking, for anything a Phase-2 reward model would have been used for — without ever having trained one. That is the sense in which the paper's central theorem (Theorem 1) says the reparameterization “does not constrain the class of learned reward models” — every reward function reachable by the old three-phase pipeline is reachable this way too, just represented differently.

Chapter 2's toy example already gave you the machinery to see exactly why “does not constrain the class” is true, not just plausible. Two reward functions r and r′=r+f(x) induce the identical optimal policy π_r (Lemma 1, worked out numerically back there). Run the DPO reparameterization on either one, and you recover the same implicit reward formula, up to that same additive constant — because r̂θ(x,y)=βlog(πθ(y|x)/π_ref(y|x)) only ever measures a policy's log-probability against a fixed reference, and a policy has no memory of which member of its equivalence class of rewards it was trained to represent. So “secretly a reward model” comes with a precise asterisk: DPO recovers a valid representative of the true reward's equivalence class, not a uniquely identified r*(x,y) — but since every member of that class induces the exact same behavior and the exact same preference probabilities, the asterisk costs you nothing in practice. Best-of-N reranking makes this concrete: sample N candidate responses from any base model, score every one with r̂θ(x,y)=βlog(πθ(y|x)/π_ref(y|x)), and keep the highest-scoring candidate — the exact same procedure you'd run with a Phase-2 reward network, except the “reward network” here is two log-probability lookups against a model you already have loaded.

Concept → realization, made literal: this is the entire implementation.

python
def implicit_reward(policy, ref_policy, x, y, beta):
    # Two forward passes, sum token log-probs along the completion (Chapter 2's
    # log pi(y|x) = sum_t log pi(y_t | x, y_<t)), then a subtraction and a scale.
    logp_policy = policy.sequence_logprob(x, y)
    logp_ref    = ref_policy.sequence_logprob(x, y)
    return beta * (logp_policy - logp_ref)

def best_of_n(policy, ref_policy, x, candidates, beta):
    # candidates: N responses sampled from ANY model, not necessarily `policy` itself
    scored = [(y, implicit_reward(policy, ref_policy, x, y, beta)) for y in candidates]
    return max(scored, key=lambda pair: pair[1])[0]  # highest-scoring candidate

Two things worth noticing about this snippet. First, ref_policy is the same frozen checkpoint used throughout DPO training — it never gets thrown away after training finishes, because every implicit reward is defined relative to it (Eq. 5, Chapter 2). Score the same completions against a different reference and you'd get different numbers, even though nothing about the policy changed. Second, candidates doesn't have to come from policy at all — you could rerank outputs from an entirely different, cheaper model using a DPO-trained policy's implicit reward as the judge, exactly as you would with any Phase-2 reward network. The only new cost, compared to plain inference, is a second forward pass through ref_policy per candidate.

There's a second half to Section 5 worth knowing about, even briefly: the paper also uses this same framework to explain why PPO-style actor-critic training is unstable in the first place (Section 5.2). Take the standard RL fine-tuning objective from Chapter 0, but plug in the DPO-equivalent reward instead of the raw scorer, and after some algebra it reorganizes into:

Look closely at f(rφ,π_ref,β): it's βlog∑_yπ_ref(y|x)·exp((1/β)rφ(x,y)) — which is exactly β·log Z(x), the very normalization constant from Chapter 1's derivation, wearing a different name. The paper calls this term “the soft value function of the reference policy” — a way of saying “how much expected reward is achievable from this prompt, averaged over the reference model's own behavior, temperature-scaled by β.” It doesn't depend on πθ, so it can't change which policy maximizes Eq. 10 — the same “along for the ride” argument from Chapter 1. But PPO doesn't get to ignore f(rφ,π_ref,β) the way DPO does, because PPO is estimating a policy gradient via sampling, not solving the KL-projection in closed form — and to compute an actual gradient step, something has to stand in for this term. Most PPO implementations either try to learn it with a value function (hard to optimize, another source of the critic's own instability from Chapter 0) or estimate it crudely with a single-sample Monte Carlo baseline — a human completion's reward, used as a rough proxy for “expected reward from this prompt.” Either approximation is noisy, and the paper's diagnosis is that this specific noisy estimate is what injects the high-variance policy-gradient signal that makes PPO finicky to tune. DPO doesn't estimate that term poorly; it never has to estimate it at all, because the reparameterization made an object that is literally the same Z(x) from Chapters 1 and 2 cancel algebraically, one more time, in one more place.

Chapter 5: Failure Modes, Honestly

Every chapter so far has made DPO look clean: no reward model, no PPO, a loss you can hand-compute on a napkin. That's real, and it's why the field adopted it fast. It is not the whole story. This chapter reports what the paper itself found when it stress-tested DPO — not speculation, its actual numbers. “Honestly” cuts both ways, so before the failures, here's what actually held up when the paper went looking for cracks.

What held up first: the reward/KL frontier, and robustness to sampling temperature

Concept → realization first, because “controlled sentiment generation” is doing a lot of work as a phrase. The actual setup (Appendix C.1): prompts are 2–8 token prefixes drawn from the IMDB movie-review dataset; the base model is gpt2-large; and critically, the ground-truth reward isn't hand-labeled at all — it's a real, separately trained sentiment classifier, siebert/sentiment-roberta-large-english, scoring how positive a completion sounds. The paper first runs ordinary supervised fine-tuning on a subset of IMDB for one epoch, then samples 4 completions for each of 25,000 prefixes from that SFT model and constructs 6 preference pairs per prefix from all 4C2 combinations, using the sentiment classifier's score to decide winner and loser automatically — no humans anywhere in this particular experiment's preference data, which is exactly what makes it useful as a controlled test: with a fixed, known, computable ground-truth reward, you can directly measure how close each method's learned policy gets to the objective's true optimum, without worrying about label noise or a judge's own biases (Failure 5, below, is precisely the case where you can't avoid that worry).

Section 6.1 of the paper runs the most direct test imaginable of whether DPO actually solves Chapter 0's KL-constrained objective as well as PPO does: sweep each method across multiple settings of its own conservativeness knob — target KL ∈ {3, 6, 9, 12} for PPO, β ∈ {0.05, 0.1, 1, 5} for DPO, α ∈ {0.05, 0.1, 0.5, 1} for the Unlikelihood baseline, and several random seeds for a Preferred-FT baseline — 22 runs total. Every 100 steps until convergence, they measure both the average true reward achieved and the average KL -divergence from π_ref, and plot reward against KL for every run. The result: DPO produces “by far the most efficient frontier” — the highest reward at every KL budget, dominating plain PPO, and notably still dominating PPO-GT, an oracle version of PPO given direct access to the true ground-truth reward function rather than a learned approximation of it. DPO and PPO are, in principle, optimizing the exact same objective from Chapter 0 — this result says DPO gets closer to that objective's true optimum in practice, not just cheaper.

On TL;DR summarization, a second axis of robustness shows up: sampling temperature. The paper sweeps completions from temperature 0.0 to 1.0 and measures GPT-4 win rate at each. DPO's win rate stays close to its peak across that whole range; PPO's degrades sharply as temperature rises, falling all the way to roughly what the untuned base GPT-J model achieves at high temperature. A method that only works at one carefully-chosen sampling setting is fragile in a way that matters operationally — you don't always get to serve every user at temperature 0 — and this is a place DPO is measurably the more forgiving choice.

Single-turn dialogue (Anthropic-HH) sharpens the point further. Starting from a pretrained Pythia-2.8B (no standard SFT checkpoint exists for this task, so the paper builds its own reference via Preferred-FT on the chosen completions first), DPO is, in the paper's own words, “the only computationally efficient method that improves over the preferred completions” in the dataset at all. It's checked against a Best-of-128 Preferred-FT baseline — sample 128 completions, keep the one the reward model likes best, a rough, compute-heavy proxy for PPO-level performance the paper adopts because the actual off-the-shelf PPO checkpoint they tried couldn't beat the untouched base Pythia-2.8B model at any prompt or temperature they could find. DPO matches or beats that 128-sample baseline while sampling exactly one completion, and Figure 3 shows it converges to its best performance quickly, early in training, not after some long fragile tail.

Failure 1: it does scale, but not for free

On TL;DR summarization, DPO reaches a 61% win rate against reference summaries at sampling temperature 0.0, versus PPO's best-case 57% at its own optimal temperature (also 0.0) — DPO wins here. But look at Table 1 of the paper, the out-of-distribution generalization test: both policies, trained only on Reddit TL;DR, evaluated on CNN/DailyMail news articles they never saw during training.

DPO still leads. But notice both methods drop hard leaving distribution — DPO falls from 61% (in-domain, different eval protocol) to 36%, a reminder that a preference-tuned policy has learned to satisfy a particular preference distribution, not summarization in the abstract. The paper is candid about this in its own limitations section: “How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? … more comprehensive study is needed.”

Do the relative-drop arithmetic by hand, because absolute win rates alone can be misleading here. DPO's drop is (61−36)/61 = 41.0% of its in-domain win rate, lost leaving distribution. PPO's drop is (57−26)/57 = 54.4% of its in-domain win rate. Both collapse hard, but DPO retains proportionally more of what it had — a smaller relative fall from a similar starting point. That's a genuinely different (and slightly more favorable to DPO) picture than “both drop hard” alone conveys, and it's exactly the kind of number you only get by actually doing the division instead of eyeballing two side-by-side percentages.

Failure 2: fluent but wrong, and GPT-4 notices

The paper's own qualitative analysis (Appendix Table 9) catches DPO producing a summary that is more detailed and more confidently written than the ground truth — and factually wrong. The paper's own description: “DPO's response is verbose and plausible, but contains factually incorrect information (the ‘coalition of the willing’ does not refer to events of WWII; the ‘all-inclusive association’ is not a real organization).” GPT-4, used as the paper's own judge, chose the ground truth over DPO specifically because of this. This is the sharp edge of what DPO actually optimizes: Eq. 7 rewards a response for being relatively preferred in the training distribution, which correlates with – but is not identical to – being factually correct. If human labelers in the training data systematically rewarded detail and confidence over concision and accuracy (a very plausible labeling bias), DPO will learn exactly that trade, faithfully.

Failure 3: the naive version degenerates completely

We flagged this in Chapter 4, but it's worth seeing exactly what the paper actually shows, word for word, because this is a place where sloppy reading is easy and costly. Strip the σ(·) weighting term out of the DPO gradient (this is the paper's Unlikelihood baseline, Appendix C.3) and the model doesn't converge — it runs away. Appendix Table 3 samples the Unlikelihood-trained policy on TL;DR prompts at temperature 1.0, and the completions are not sentences at all: for one prompt the model outputs the single token “when” repeated dozens of times in a row (“girl when when when when when when when when…”); for another it does the same thing with a different filler word. The table's own caption is blunt about it: “we find unlikelihood fails to generate meaningful responses for more complex problems such as summarization and dialogue.” There is no bracketed annotation anywhere in Table 3 — the paper doesn't need one, because a wall of the same repeated token speaks for itself.

The bracketed string “[maximum generation length reached]” does appear in the paper — twice, in fact — but it belongs to a completely different experiment. It's a post-hoc editorial annotation (the paper explicitly labels these annotations as not part of the model's actual output) marking where a DPO-generated response, not an Unlikelihood one, simply ran out of generation budget mid-sentence. One instance is in Table 7, a DPO response to “Can you help me write an essay on the Civil Rights Movement?” the other is in Table 9, a DPO response about what drew the United States into World War II (the same response Failure 2, above, quotes for its factual errors about a “coalition of the willing”). Both are from the dialogue qualitative-analysis appendix, where DPO is being compared against ground-truth Anthropic-HH responses — an entirely different ablation, on a different dataset, studying a different question, than the Unlikelihood-degeneration story this section is about.

Why does the distinction matter, beyond getting a citation right? Because they're two different failure modes with two different lessons. Unlikelihood without the σ(·) weighting collapses into literal, meaningless repetition — the model has nothing coherent left to say and the loss doesn't punish it for saying nothing. A DPO response hitting the token limit mid-thought is a completely mundane generation-budget artifact, sitting inside an otherwise fluent, on-topic (if occasionally factually wrong) paragraph — evidence for Failure 2's point about confident-but-wrong content, not for this section's point about degenerate weighting. Conflating the two would make you draw the wrong conclusion from each: you'd think DPO itself produces gibberish (it doesn't; Unlikelihood does), and you'd miss that the length-cutoff artifact has nothing to do with the weighting term at all. The lesson generalizes past this one baseline correctly, even with the citations untangled: preference losses that don't automatically down-weight already-resolved examples can over-sharpen a distribution past the point of coherence. DPO's weighting isn't decorative; remove it and repeated-token collapse, not a graceful length cutoff, is what you get.

Failure 4: β and the reference model are load-bearing, and mostly untuned

Appendix B's hyperparameters are strikingly minimal: β=0.1 by default, β=0.5 for TL;DR, batch size 64, RMSprop at learning rate 1e-6 with a 150-step linear warmup. The paper is explicit that it “did not meaningfully tune DPO's β hyperparameter,” meaning the reported results likely understate DPO's ceiling — a comfort for practitioners, but also a warning: β was not validated as robust across settings, it was mostly just not touched. And π_ref is not a free variable either. Every implicit reward in this lesson is relative to whichever π_ref you pick; the paper's own outline states plainly that π_ref should be initialized from π^SFT whenever available, and when it isn't, from a model fit by maximizing likelihood of just the preferred completions — because a mismatched reference distorts every log-ratio in Eq. 7 before training even starts.

Failure 5: even the judge needed checking

Every win-rate number in this chapter came from GPT-4 acting as an automated judge, comparing two summaries and picking a winner. That's convenient, but it's also a second model's opinion standing in for a human's, and the paper doesn't just assume that substitution is safe — Section 6.4 runs an actual human study to check it, with real participant counts: 272 respondents for one comparison, 122 for another, 199 for a third, comparing the highest-scoring policy (DPO, temperature 0.25), the lowest (PPO, temperature 1.0), and a midpoint. What they found is its own small, honest failure: GPT-4, prompted the simple way (“which summary better covers the important information”), systematically preferred longer, more repetitive summaries than actual human raters did. That's a judge-side bias, not a policy-side one — and it means every earlier win-rate number in this chapter was at some risk of quietly rewarding verbosity, the exact same trait Failure 2 already caught DPO's own policy exploiting. The paper's fix was methodological: introduce a second prompt variant, GPT-4 (concise), that explicitly also asks which summary is more concise, and report results from that stricter prompt rather than trusting the simple one uncorrected. It's a small thing to catch, and catching it is exactly the kind of rigor this chapter is built to model — before you trust any evaluation number, ask what the evaluator itself might be systematically biased toward, the same way you'd ask it of a reward model.

Chapter 6: Teaching Curiosity

Everything so far has been about one language model, one prompt at a time, choosing between two already-written responses. Now flip the setting entirely. Instead of “which of these two answers is better,” the question becomes: can a language model act — take a sequence of actions in an unfamiliar environment, receive feedback, and use that feedback to act better on the next turn, on a task it has never seen before? This is the problem Tajwar et al. tackle in Training a Generally Curious Agent (arXiv:2502.17543), and the method they call Paprika. We're about to see the exact same DPO machinery from Chapters 1–4 doing a completely different job.

Why this is hard, concretely

Prior work (Krishnamurthy et al. 2024, cited directly by the Paprika authors) showed that LLMs perform poorly even at the simplest possible sequential decision problem: a multi-armed bandit, where you repeatedly pick one of several unlabeled options and observe a noisy reward, and the only goal is to eventually figure out which option pays best. This isn't a hard reasoning problem in the usual sense — there's no clever proof, no tricky code. It requires something else: trying an action specifically because you don't yet know its outcome, tracking what you've learned across turns, and updating your strategy from feedback you generate yourself. That capacity is exploration, and it turns out not to fall out of ordinary instruction-following at all.

The obvious fix — generate expert trajectories from a known-good algorithm (e.g. the UCB algorithm for bandits) and fine-tune on those — was already tried (Nie et al. 2024) and it works, for bandits. The Paprika authors point out exactly why this doesn't scale as a general strategy: “(1) we want LLMs to perform strategic exploration and decision making in more complex settings, (2) for most tasks, there is no known algorithm like UCB to generate good synthetic trajectories from, (3) it can be infeasible to collect data for all tasks that we care about.” You cannot pre-solve every task the model will ever face and distill each one's optimal algorithm. What you can do is teach the model a general disposition toward exploration, on a diverse enough set of tasks that it transfers to tasks it's never seen.

Ten task groups, deliberately incompatible

Paprika's first move is to design a suite of ten distinct task groups — each one a family of related problems that share a required strategy but not a shared optimal policy (guessing “apple” in twenty questions is a different task from guessing “Paris,” but both reward the same kind of informative-question-asking). Every task is formalized as a partially observable Markov decision process (a POMDP): at each turn the agent emits a text action a_t, the environment emits a text observation o_t, and at the end of the episode the environment emits a single scalar score r(h) for the whole trajectory h.

The “partially observable” part is not decoration — it's the entire reason any of this is hard. In an ordinary (fully observable) Markov decision process, the agent sees the complete state of the world at every step; in a POMDP, it doesn't. In Twenty Questions, the true hidden state is the secret topic itself, and the agent never observes it directly — only o_t, the yes/no answers it manages to extract, one bit of information at a time. The whole skill of the game is choosing a_t to maximize how much the next o_t narrows down the hidden state, which is precisely what “exploration” means formally: acting not to directly collect reward, but to reduce uncertainty about a state you can't see. Write h=(o₀,a₀,…,o_H,a_H) for a full episode of length H, and h_:t for everything that happened strictly before turn t — that's the “history” notation Chapter 7's multi-turn loss conditions each action on, because in a POMDP the optimal next action depends on the entire history of past observations, not just the current one.

Two design choices matter more than they look. First, for tasks needing real-world knowledge to generate plausible feedback (twenty questions, customer service, murder mystery), the “environment” is itself another LLM — GPT-4o-mini, prompted to answer honestly and to flag when the agent has actually won. For rule-based tasks (Wordle, Mastermind, Battleship), the paper deliberately uses hardcoded programs instead of an LLM judge, “similar to DeepSeek-AI et al. 2025,” because they found LLM-generated feedback for rule-governed games less reliable than a simple verifier. Second, several games (Wordle, Mastermind, Minesweeper) additionally let the agent think in chain-of-thought before committing to an action — the paper reports this “improves its performance significantly” on tasks requiring multi-step deduction from structured feedback.

Concept → realization: what the agent and the environment actually see

“An LLM plays twenty questions against another LLM” is an abstraction until you look at the actual text both sides receive. Here, condensed but verbatim in spirit, is the agent's system prompt for Twenty Questions: “You are playing a game of 20 Questions. Your goal is to guess the name of a thing or person by asking up to 20 yes-or-no questions… If you're confident, you can make a guess before reaching 20 questions.” The environment — a separate call to GPT-4o-mini — receives a completely different prompt, one that includes the secret answer the agent never sees: “You are the environment for a game of 20 Questions. You will be given a topic… your role is to answer ‘Yes’ or ‘No’ to questions about the topic… If the user guesses the correct answer, respond with ‘Goal reached.’” Two separate model calls, two separate prompts, one shared hidden fact (the secret topic) that only one of them is allowed to know — that's the entire mechanism behind “the environment emits an observation o_t.”

Notice the failure mode this setup invites: what stops the environment LLM from hallucinating “Goal reached” when the agent hasn't actually guessed correctly? The paper doesn't trust the environment LLM's self-report at face value. It runs a third model call, a dedicated judge prompt, that receives the full exchange and the true secret topic and must reply with exactly <VALID> or <NOTVALID> before a “win” is accepted — and separately, any environment response that isn't strictly “Yes,” “No,” or “Goal reached” (checked by string matching) causes the whole trajectory to be discarded after five retries. This is real engineering against a real failure mode — an LLM-simulated environment that reports success too eagerly would silently poison every downstream training signal, so the paper builds a second, independent check specifically to catch it.

For the hardcoded-feedback games, contrast the observation format directly. Wordle's agent is told to answer inside <Think>…</Think> and <Answer>…</Answer> tags (the chain-of-thought scaffold from the design-choices paragraph above, made concrete), and the environment's reply is not free text generated by a language model at all — it's a deterministic per-letter report from a Wordle-rules program: for secret word “toast” and guess “boost,” the environment returns something like “First letter, b, is not in the target word. Second letter, o, is correct and in the correct position…” — a fixed, rule-derived string, the same every time for the same guess, with zero risk of the hallucination problem Twenty Questions has to guard against. That's the entire reason the two feedback types exist side by side in the same suite: LLM-generated feedback for tasks that genuinely need world knowledge to judge (nobody can hardcode “is this a sensible follow-up question about a city”), hardcoded feedback for tasks with an exact, checkable rule (nobody needs an LLM to tell you whether a Wordle letter is in the right position).

Formally, the performance of a policy π on a task group G is the average expected score across every task in it:

Read this the way you'd read any average: for every task τ in the group, sample a trajectory h by running the policy π against it (that's what π∘τ means — policy composed with task), take the expected score 𝔼_h∼π∘τ[r(h)] for that one task, then average across all |G| tasks in the group. A tiny illustration with invented numbers: if a task group has 3 tasks and the policy's expected scores on them are 0.80, 0.50, and 0.90, then Perf(G) = (0.80+0.50+0.90)/3 = 0.733 — one number summarizing performance across an entire family of related-but-not-identical problems. This is exactly the quantity Chapter 8's headline “+47%” result compares, before and after Paprika fine-tuning, averaged not over 3 toy tasks but over 10 real task groups.

The entire experiment is defined by one split: the agent trains on a set of groups 𝒢_train and is evaluated — often having never seen the target group at all — on 𝒢_test.

Generating the data: diversity is the point

Here is where the design gets deliberate about exploration as a property of the data, not just the loss. For each training task, Paprika samples n_sample=20 independent trajectories (100 for Mastermind, since its search space is larger) using Min-p sampling at a high temperature (1.5) — a decoding method that adaptively truncates the vocabulary relative to the top token's probability, which the authors chose specifically because it “enables us to generate diverse yet coherent trajectories at a higher temperature” than ordinary top-p or temperature sampling would allow without collapsing into gibberish.

From those 20 rollouts, Paprika builds one preference pair per task: h_w is the highest-scoring trajectory (the one that succeeded, and did so in the fewest turns); h_l is randomly sampled from whatever scored lower — not necessarily the single worst one. That randomization is deliberate: always picking the worst-possible failure would make every losing example trivially distinguishable from the winner, and the paper notes prior work (Pal et al. 2024) already showed DPO struggles when the edit distance between y_w and y_l is too small to matter or the pair is too easy to be informative. Randomizing h_l keeps the training distribution harder and more diverse.

Make this concrete with a toy round of 20 rollouts on one Twenty Questions task. Suppose the scores (say, 21 minus the number of questions used, or 0 for an outright failure to guess within 20 turns) come back as: 6 rollouts fail outright (score 0), 9 rollouts succeed slowly, in 14–19 questions (scores 2–7), and 5 rollouts succeed efficiently, in 6–11 questions (scores 10–15). h_w is simply whichever single rollout has the highest score — say the one that solved it in 6 questions, score 15. h_l is drawn at random from the other 19, not restricted to the 6 outright failures: it might land on one of the slow 14-question successes (score 4) just as easily as on a score-0 failure. Either way, the resulting pair still teaches something — “this specific fast, information-dense line of questioning outscored this specific slower one” is a real, if noisier, signal than “a coherent strategy beat a broken one,” and across 20 rollouts × roughly 1,500 training tasks in Twenty Questions alone, that noisier signal aggregates into exactly the kind of general disposition Chapter 6 opened by promising.

Chapter 7: Multi-Turn Preferences

Chapter 2 derived a DPO loss for a single prompt and a single-turn response. Paprika's data is a multi-turn trajectory — a whole sequence of (observation, action) pairs. This chapter shows exactly how the loss had to be rebuilt, term by term, to handle that, and introduces the second machine Paprika needs: a curriculum that decides which tasks are even worth sampling from.

Two objectives, then a combination

Paprika actually trains in two stages, using two losses defined directly on the trajectory data from Chapter 6.

Stage 1 — supervised fine-tuning on the winners. Treat the highest-scoring trajectories as expert demonstrations and simply maximize their likelihood, discarding the losers entirely:

Read the normalization carefully: dividing by the total number of action tokens ∑_t|a_t^w| means a trajectory isn't weighted by how many turns it took — it's weighted per token, so a 3-turn success and a 15-turn success contribute comparably per unit of text generated. This step is exactly rejection-sampling fine-tuning (also called STaR, Self-Taught Reasoner — train on your own past outputs that happen to have reached a correct answer — or RAFT, Reward-rAnked FineTuning, in prior literature): keep only the trajectories that worked, train on those as if they were ground truth. The minimization target this loss is written to maximize is a raw log-likelihood, so ℒ_SFT here is a negative log-likelihood in the same sense as ordinary cross-entropy training: you minimize its negation. Written that way, to match Chapter 2's sign convention:

which is precisely negative log-likelihood, minimized by gradient descent — the same shape as any ordinary supervised fine-tuning loss you've seen elsewhere, just restricted to the agent's own action tokens and normalized per token instead of per trajectory.

Stage 2 — multi-turn DPO on the pairs. This is the real rebuild. Ordinary DPO (Eq. 7 from Chapter 2) computes one log-probability ratio for one response. A trajectory has many turns, and crucially, some tokens in the trajectory were generated by the environment, not the agent — you must never train the policy to predict the environment's own words. The paper's multi-turn DPO loss sums the implicit reward over only the agent's action tokens, across every turn of the episode:

Compare this term by term against Chapter 2's Eq. 7. Where single-turn DPO had one log-ratio per response, this has a sum of log-ratios, one per turn, only over the agent's own tokens — the “implicit reward” of an entire winning trajectory is the sum of the implicit rewards of every individual decision inside it. Everything else — the leading minus sign turning a log-likelihood into a loss to minimize, the sigmoid, the binary cross-entropy shape, the weight that's higher when the model is more wrong — is identical to Chapter 4's derivation, because the underlying Bradley-Terry-over-trajectories argument goes through exactly the same way, just with “response” generalized to “trajectory.”

Worked example: sum the implicit reward over a toy trajectory by hand

Push real numbers through the multi-turn formula the way Chapter 3 did for the single-turn one. Say the winning trajectory h^w has 2 turns and the losing trajectory h^l has 3 — trajectories don't need to be the same length, since the loss sums however many action tokens each one actually has. Use β=0.1 throughout, matching every earlier worked example in this course.

Notice every one of the five log-ratios in these two tables is negative or small — each individual decision, taken alone, looks like the policy drifting slightly away from or barely toward the reference. But the trajectory-level implicit reward is a sum over decisions, and a string of small negative per-turn log-ratios in the losing trajectory compounds into a much more negative total than any single turn shows. Now the margin and loss, exactly as in Chapter 3:

Same shape, same arithmetic, one new wrinkle: what got summed inside the sigmoid wasn't one log-ratio per response, it was a whole trajectory's worth — and note that neither table above included a single environment-generated token. That omission is doing real work, not just following a formula: if turn t=1's “action” had actually been the environment's Wordle feedback string rather than the agent's own guess, including its log-ratio in this sum would mean training the policy to imitate text it never generated and has no business being scored on.

Why DPO, and why add SFT back in

The authors are explicit about a practical reason for choosing DPO over online RL here: “DPO allows us to decouple the data collection and policy improvement steps and offload them on different machines” — exactly the memory argument from Chapter 0, now cashed out as an engineering convenience: generate trajectories on one cluster, train on another, with no synchronized rollout loop required.

But DPO alone has a known failure the paper cites directly: unintentional unalignment (Razin et al. 2024) — DPO's gradient, recall from Chapter 4, only cares about the difference between winner and loser log-probabilities, so it can satisfy that difference by decreasing both log-probabilities, as long as the loser's drops faster. In the worst case, the “preferred” trajectory's own probability can fall during training even as the loss goes down. The fix Paprika uses is RPO (Pang et al. 2024), which simply adds the SFT loss back in as a regularizer:

The SFT term is a direct, unconditional pull toward higher likelihood for the winning trajectory, which counteracts DPO's tendency to only care about the margin. This is exactly the kind of practical patch that shows up once you leave the clean theory of Chapters 1–4 and start training on real, messy, self-generated data.

Concept → realization: the actual training recipe

Every piece described so far — task groups, rollout sampling, the two-stage loss — has to become an actual training run somewhere, and the paper reports the exact recipe. Data generation uses Min-p sampling at temperature 1.5 and a specific Min-p truncation parameter of 0.3, applied across n_sample=20 rollouts per training task (100 for Mastermind, as Chapter 6 noted). After filtering out malformed or environment-hacked trajectories, this yields 17,181 trajectories for the Stage 1 SFT step and 5,260 trajectory pairs for the Stage 2 RPO step, pooled across all ten task groups. Training itself uses a learning rate of 1×10⁻⁶ for SFT and a much smaller 2×10⁻⁷ for RPO, batch size 32, and the AdamW optimizer with a cosine-annealing schedule and a 4% warmup ratio — SFT is always run first, then RPO fine-tunes further on top of the SFT checkpoint, never the reverse. At evaluation time, the paper samples 4 trajectories per test task (Min-p 0.3, temperature 0.7) and reports the average success rate, with pass@4 (credit if any of the 4 succeeded) reported separately as a more forgiving metric. All of this is run on Llama-3.1-8B-Instruct as the primary base model, with a second full pass on Gemma-3-12B-IT reported as a robustness check — the headline results this lesson quotes hold on both, not just one cherry-picked backbone.

The curriculum problem: which tasks are worth sampling?

Here's the part that has no analogue in single-turn alignment DPO at all. Generating one trajectory in, say, Murder Mystery costs an LLM rollout through up to 20 turns of dynamic environment simulation — that's expensive, and the paper is candid that “the major cost for training is actually data generation rather than model updates,” a complete inversion of ordinary deep learning where compute is dominated by backward passes. If some tasks are much harder than the current model can handle, sampling 20 rollouts from them yields 20 failures and zero learning signal — wasted compute. You want to spend your sampling budget on tasks the model can partially solve, where the outcomes still vary.

The paper's measure of “is this task worth sampling from right now” is the coefficient of variation of the model's reward on that task — its standard deviation divided by its mean:

where R_π(τ) = 𝔼_h∼π∘τ[r(h)] is the average score and σ²_π(τ) is its variance. If a task is so easy the model always succeeds, or so hard it always fails, its variance is near zero, ν is near zero, and there's nothing new to learn — every one of the 20 sampled rollouts will look the same, and DPO literally cannot form a meaningful preference pair from 20 identical outcomes. If the model succeeds sometimes and fails sometimes, variance is high relative to the mean, and that's precisely where a winner/loser pair carries real information. Normalizing by the mean matters because different task groups have wildly different reward scales; without it you couldn't compare “this Twenty Questions task” against “this Battleship task” on the same footing.

Turning task selection into a bandit problem, on top of a bandit-solving agent

There's a pleasing recursion here: to decide which task group to sample from next, Paprika treats task-group selection itself as a multi-armed bandit problem, solved with the classic UCB (Upper Confidence Bound) algorithm — the same family of algorithm the agent is being trained to eventually handle inside the environment. Each task group is an “arm.” Pulling an arm means: sample one task from that group, roll out C trajectories, measure ν̂_π(τ) from those samples, and use it as the observed “reward” for that arm. The UCB score for arm k after n_k pulls with cumulative score s_k is:

The first term is the empirical average learning-potential observed so far for group k; the second is an exploration bonus that grows for arms pulled rarely — UCB's whole idea is “optimism in the face of uncertainty”: prefer arms with either a high known average or too little data to be sure they're bad. At each round, pick k* = argmax_kθ_k, sample from that group, and update its running statistics.

Work one round by hand. Suppose after some pulls, Twenty Questions has s=3.2, n=5, and Wordle has s=0.9, n=3, with ∑n_j=20 total pulls so far across all groups:

Close — Twenty Questions edges it out (1.735 vs 1.713) despite Wordle's exploration bonus being larger (fewer pulls), because Twenty Questions' higher empirical average still wins out this round. Sample from Twenty Questions next; the moment its n_k climbs and its bonus shrinks, an under-sampled arm like Wordle can overtake it. That's the whole mechanism — Chapter 8 puts it in motion, live.

Chapter 8: The Curriculum Showcase

Time to see the whole loop running, and to see what it actually bought Paprika in the paper's own numbers.

What kind of improvement this actually is

One distinction is worth being precise about before the numbers arrive, because it changes what “transfers to an unseen task group” even means. The paper's own framing, right in the abstract, is that a Paprika-tuned model can “effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training” and “adapt their behavior on a new task based on environment feedback in-context without more gradient updates.” Read that carefully: the gradient updates all happened once, during the training run this chapter is about to quantify. When the fine-tuned model later encounters Bandit Best-Arm Selection for the first time in the leave-one-out experiment below, it does not get retrained on bandits — it walks in with frozen weights and has to adapt purely from the observations it accumulates within that one episode, exactly the way a human who has played nine different guessing games might walk into a tenth one without ever having seen its specific rules, and still play it more competently than someone who's never played any guessing game at all. What Paprika is claiming to have trained isn't “knowledge of ten specific games” — it's a reusable, in-context adaptation skill that a fixed set of weights can apply to a game it has literally never been shown.

Run it for thirty or forty rounds. Two things should become visible. First, the algorithm doesn't lock onto a single “best” arm and hammer it forever — every arm gets pulled occasionally, because the exploration bonus keeps rising for anything neglected, until it's competitive again. Second, arms with genuinely low learning potential (near-zero true ν, meaning the model already always succeeds or always fails on that group) still get sampled early, while uncertainty is high, but fall out of rotation once their low value becomes statistically clear — exactly the behavior that saves wasted rollouts on already-solved or hopelessly-hard task groups.

Continue Chapter 7's hand-worked example one more round

Verify the “arms can switch places” claim from Chapter 7 with one more round of real arithmetic, instead of trusting the widget's animation alone. Recall where we left off: Twenty Questions had s=3.2, n=5, θ=1.735; Wordle had s=0.9, n=3, θ=1.713; Twenty Questions won that round and got sampled. Suppose that new pull returned an observed ν̂_π(τ) of 0.50 — a fairly ordinary result, neither a great nor a terrible round for that arm. Update Twenty Questions' running statistics: s becomes 3.2+0.50=3.70, n becomes 6. Wordle is untouched: still s=0.9, n=3. The total pull count ∑n_j is now 21. Recompute both θ values for the next round:

The arms switched places. Twenty Questions' θ fell from 1.735 to 1.624 — not because its empirical average collapsed (0.617 is close to its old 0.640), but because n grew from 5 to 6 and the exploration bonus shrank accordingly, from 1.095 down to 1.007. Wordle's θ barely moved on paper (1.713→1.725, since ∑n_j ticking from 20 to 21 nudges its bonus up slightly even though Wordle itself wasn't sampled), but that was enough: Wordle now leads, 1.725 to 1.624, and gets sampled next. This is exactly the “an under-sampled arm can overtake it” behavior promised at the end of Chapter 7, confirmed with the same arithmetic, two rounds running. Watch the live widget above for the same signature: a bar's height gaining slightly every round it isn't picked (the bonus term alone growing), until it eventually overtakes whatever's currently winning.

What the curriculum actually bought, in the paper's numbers

The paper ran this exact mechanism on real Twenty Questions data: 3 rounds, 250 tasks sampled per round via Algorithm 1, using number-of-turns-to-solve as the reward proxy for ν_π. Tasks were first classified by GPT-4o-mini into easy / medium / hard difficulty tiers (477 / 726 / 296 in the train split). Comparing this curriculum against uniformly sampling 250 random tasks per round, after three rounds:

Modest numbers, and the paper doesn't oversell them — but remember what they're buying: the same final accuracy for less total rollout compute, because uniform sampling keeps spending budget on tasks that yield no signal.

Notice pass@4's advantage (+3.3pp) is more than double average success rate's (+1.4pp) — not a coincidence, but a direct consequence of what “pass@4” measures: credit if any of 4 independent attempts at a task succeeds, not the average outcome across all 4. If a single attempt succeeds with probability p, the probability at least one of 4 independent attempts succeeds is 1−(1−p)⁴, not p itself. Work a toy case: p=0.40 gives an average success rate of 40%, but a pass@4 rate of 1−(0.60)⁴ = 1−0.1296 = 87.0% — more than double the single-attempt number, purely from the arithmetic of “at least one of four.” Small shifts in per-attempt success probability get amplified nonlinearly once you're asking “did any attempt work,” which is exactly why a curriculum that nudges the model's per-attempt reliability up by a modest amount can produce a noticeably larger jump in pass@4 than in the raw average — the same underlying improvement, reported through a lens that compounds it.

Ruling out the boring explanation first

Before trusting a 47% relative improvement, it's worth asking the skeptical question: is this just “fine-tuning on more multi-turn conversation data helps,” dressed up in exploration language? The paper checks this directly (Appendix I.10). They took the same base model and fine-tuned it on 100,000 ordinary multi-turn conversations sampled from WildChat, real GPT-4/human chat logs with no game structure, no success/failure labels, no curriculum — using the exact same training hyperparameters as the real Paprika recipe. If “more multi-turn data” were the active ingredient, this should help too. It doesn't: the paper reports significant performance degradation on every task group from this fine-tune, and hypothesizes why — ordinary chat data optimizes for being coherent and agreeable turn by turn, not for strategically probing an unfamiliar environment to gather information, and training on it can actively erode whatever exploratory disposition the base instruct model already had. This is exactly the kind of negative control a careful empirical claim needs: it isn't the volume of multi-turn text that produces the 47% number, it's the specific property Chapter 6 built into the data — diverse tasks, an automatically computed success/failure signal, and a loss that rewards the strategy that gathered information fastest.

A second check (Appendix I.9) asks a narrower version of the same question about the loss itself: does the Stage 2 RPO step actually matter, or would Stage 1 SFT alone (imitating only the winning trajectories, no preference signal at all) capture most of the benefit? Across six task groups tested this way, RPO “improves performance beyond the SFT model” in every single one — the negative examples, the trajectories that scored lower, are carrying real information that pure imitation of the winners throws away. Put the two checks together and you get a fuller picture of what's actually doing the work: not “more data” (the WildChat check rules that out), and not “imitation alone” (the SFT-only check rules that out) — specifically the combination of diverse, structured task data and a preference loss that also learns from what didn't work.

What the full method bought overall

Zoom out to the headline results. Training on filtered trajectories from all 10 task groups — 17,181 supervised fine-tuning trajectories plus 5,260 RPO preference pairs, about 22,500 trajectories total, generated from a Llama-3.1-8B-Instruct base model — the paper reports:

And critically, this transfers to tasks Paprika never trained on. In a leave-one-out experiment where every task group except Bandit Best-Arm Selection was used for training, success on the untouched bandit group rose from 42.25% to 62.25% — a relative gain of (62.25−42.25)/42.25 ≈ 47.3%, achieved with zero bandit-specific training data, purely from strategies learned on nine other, structurally different games. Compare that to Chapter 6's earlier point: prior work needed a known optimal algorithm (UCB) specifically for bandits to get this kind of improvement; Paprika got a comparable gain with no bandit-specific algorithm at all — the generalizable strategy (probe broadly first, then commit sampling budget to the leading candidates) transferred zero-shot from completely different games.

And the fine-tuning didn't cost general capability: on standard benchmarks, Llama-3.1-8B-Instruct scored MT-Bench 7.88 / AlpacaEval 33.6 / GPQA 33.5 / MATH(Hard) 24.6 / MMLU-Pro 46.7 / IFEval 84.4, and after Paprika fine-tuning: MT-Bench 8.14 / AlpacaEval 33.5 / GPQA 32.8 / MATH(Hard) 25.3 / MMLU-Pro 46.2 / IFEval 85.4 — every change within noise (standard errors in parentheses in the original table), confirming this multi-turn exploration training didn't quietly damage the model's ordinary instruction -following.

Chapter 7 already flagged that the headline numbers also hold on a second base model, Gemma-3-12B-IT, not just Llama-3.1-8B-Instruct. The paper pushes this robustness check one step further (Appendix I, Figure 18): running the same recipe on three different roughly-7-8B-parameter instruct models — Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3 — evaluated on three representative task groups, with error bars from three random seeds each. This matters for the same reason the WildChat negative control in the next section matters: a single-model result always leaves open the question of whether you've found a genuine method or an accident of one particular base model's quirks. Reporting the same qualitative improvement across three different pretraining recipes, three different tokenizers, three different instruction-tuning pipelines is evidence the effect is coming from Paprika's training recipe itself, not from some idiosyncrasy of Llama-3.1 specifically.

The honest counter-result, and what “regret” means precisely

The 42.25%→62.25% leave-one-out result above uses Paprika's own Bandit Best-Arm Selection task group — success/fail on identifying the best arm within the turn limit. The paper also checks a completely different, harder bandit benchmark: the original one from Krishnamurthy et al. 2024, which doesn't just ask “did you eventually find the best arm,” it measures empirical regret — the total reward you left on the table, summed over every single decision, by not pulling the optimal arm from the very first pull:

where r* is the true best arm's reward, â_t is the arm the policy actually chose at timestep t, and T is the total number of decisions. This is a much less forgiving metric than “solved the task or didn't” — it punishes every single suboptimal pull along the way, averaged across 100 trials in the paper's evaluation, not just the final outcome.

Chapter 9: Alignment vs. Curiosity

We've now watched the identical mathematical object — a Bradley-Terry-derived, KL-anchored preference loss — do two jobs that sound like opposites. Alignment DPO (Chapters 1–5) makes a model converge: narrow its distribution toward the responses humans preferred, and stay leashed near a reference policy so it doesn't wander. Paprika (Chapters 6–8) uses the same loss to make a model better at diverging: trying unfamiliar actions, gathering information, adapting to environments it's never encountered. That tension is real, it's structural, and both papers hand you the pieces needed to see exactly where it lives and how it gets managed.

Where the tension actually comes from

Go back to Eq. 3, the objective both methods ultimately descend from: maximize reward, minus β times KL-divergence from π_ref. The DPO paper is explicit about what that KL term is for: it prevents “mode-collapse to single high-reward answers” and preserves “generation diversity.” That sentence is doing double duty. In the alignment setting, diversity is a safety valve — you don't want the model latching onto one sycophantic phrase pattern that happens to score well. In the exploration setting, diversity is not a safety valve, it is the entire point: an agent that has collapsed onto one confident strategy has, by definition, stopped exploring.

Here is the sharper version of the problem. DPO's loss, from Chapter 2, can only ever rank trajectories it already has preference labels for — it is fundamentally a converger, pulling probability mass toward y_w and away from y_l. It has no internal mechanism for generating a novel action it has never seen labeled. So where does the actual drive to explore come from in Paprika, if not from the loss? Look back at Chapter 6: it comes from upstream of the loss, in how the training data itself is generated — Min-p sampling at temperature 1.5, specifically chosen because ordinary high-temperature sampling degenerates into incoherence while Min-p stays diverse and readable. The DPO/RPO loss then converges the model toward whichever of those diverse rollouts happened to succeed fastest. The exploration lives in the sampling; the loss just decides what to keep.

Three concrete places the tension bites

Reference-policy anchoring. Every implicit reward in this lesson is defined relative to π_ref (Chapter 2, Eq. 5). In alignment, π_ref is the SFT model — a sensible anchor, since you want to stay close to a model that already writes fluent, on-topic text. In Paprika, the paper's own discussion section names this as a genuine limitation: “the starting model need[s] to exhibit good behavior within a reasonable generation budget, so Paprika would perform worse in the absence of a good base model.” Rejection sampling can only ever amplify behavior the base model already sometimes produces; if π_ref essentially never tries an effective exploration strategy, no amount of DPO re-weighting can manufacture one from nothing. The same anchoring that makes alignment DPO safe (never straying into territory the reference wouldn't recognize) is what caps Paprika's exploration at whatever the base model was already occasionally capable of.

Unintentional unalignment, twice over. Chapter 7 introduced Razin et al.'s finding that DPO's gradient can satisfy the margin by lowering both log-probabilities, as long as the loser drops faster — and that Paprika patches this with RPO's added SFT term. In the alignment setting this same failure mode has a well-documented cousin: a DPO-tuned chat model whose probability of its own preferred answer quietly falls during training is a model becoming less confident, not more curious, about what it says. In both settings, the fix is structurally identical — add back an unconditional likelihood term that anchors the winning behavior in place, rather than trusting the margin alone. The same instability, and the same patch, shows up whether “winning” means “the reply a human liked” or “the trajectory that solved the puzzle fastest.”

What counts as evidence that the loss actually worked. Chapter 5's entire honesty audit depended on GPT-4-as-judge, and even that needed a human study to catch the judge's own bias toward verbosity — “did DPO improve the response” is ultimately a matter of comparing two pieces of open-ended text against a proxy for human taste, and every proxy can be gamed or biased in ways you have to go looking for. Paprika's evaluation is structurally different: “did the agent guess the secret word,” “did it identify the best arm,” “did it sink all the ships” are machine-checkable facts, not judgments — Chapter 6's hardcoded-feedback task groups don't need a GPT-4 judge at all, and even the LLM-simulated ones (Twenty Questions, Customer Service) reduce to a strict <VALID>/<NOTVALID> call rather than an open-ended preference. That's not a minor implementation detail; it changes what kind of “honest failure mode” each paper can even discover. DPO's failures in Chapter 5 are subtle and require careful qualitative reading (a factually wrong but fluent paragraph, a judge with a length bias). Paprika's failure in Chapter 8 (the arm-count-dependent gap on the standard bandit benchmark) is a hard number computed from a hard-coded regret formula, no judge involved. Alignment's evaluation problem is partly a measurement problem; curiosity's evaluation problem, in this suite at least, mostly isn't.

Watch unintentional unalignment happen, with numbers

Don't take “can satisfy the margin by lowering both probabilities” on faith — run it through the same arithmetic Chapter 3 used, now with a training trajectory where the “preferred” response's own probability falls. Start, as always, from πθ=π_ref at initialization: logπ_ref(y_w)=logπ_ref(y_l)=−1.00, so every log-ratio starts at zero. Now suppose training moved the policy this way instead of the healthy direction Chapter 3 showed:

Notice the winner's log-ratio is negative — the policy has made y_w, the response a human preferred, less likely than the reference model thought it should be. That's the failure mode by name: the “preferred” trajectory's own probability fell during training. Compute the loss anyway, with β=0.1 as always:

Margin 0.120 — look back at Chapter 3's very first worked example: it also produced a margin of exactly 0.120, and the same loss of 0.635, but by moving the winner's log-probability up (log-ratio +0.40) while the loser barely moved. DPO's loss cannot tell these two training trajectories apart. It sees the same number either way, because the loss only ever depends on the difference between the two implicit rewards, never on their absolute levels. Gradient descent is free to satisfy that difference however is cheapest, and “push the loser down faster than the winner” is just as valid a way to widen the margin as “push the winner up.” That indifference is precisely Razin et al.'s finding, and it's why RPO's added SFT term matters: ℒ_SFT cares about logπθ(y_w) in isolation, not relative to y_l, so it directly penalizes the scenario in this table even though the DPO term alone would happily accept it.

Full circle: why neither paper just reaches for online RL

Chapter 0 opened with PPO's cost, and both papers, read side by side, converge on a shared answer to “why not just run reinforcement learning properly, then?” The DPO paper's whole argument is that you don't have to, for single-turn preference data, because the closed-form trick makes an entire class of RL machinery unnecessary. Paprika's argument is narrower and more pragmatic: for its setting, DPO is chosen specifically because it “allows us to decouple the data collection and policy improvement steps and offload them on different machines” — an engineering convenience, not a claim that DPO is strictly better than online RL here. In fact the paper says the opposite, plainly, in its own words: following prior work showing online RL outperforms offline algorithms in similar settings, the authors “expect doing Paprika with online RL would lead to even stronger results,” and leave it as future work. That's a genuinely different posture from the DPO paper's: DPO claims to match or exceed PPO while deleting its machinery (and Chapter 5 showed real evidence for that claim); Paprika claims DPO is a reasonable, compute-decoupled compromise, while suspecting a more expensive, more synchronized online method would do even better. Both papers avoid the four-networks-in-memory problem from Chapter 0 — but for two different reasons, one a mathematical equivalence, the other an engineering tradeoff its own authors flag as leaving performance on the table.

Why this isn't a contradiction, just a design axis

Both papers, read together, point at the same resolution: DPO is a mechanism-neutral tool for converging a policy toward whatever behavior your preference pairs encode. What it converges toward is entirely determined by two things outside the loss itself — how y_w and y_l get generated, and what β and π_ref anchor the result to. Point the data generator at “two static answers, ranked by a human who wants safety and helpfulness” and you get an aligned assistant that stays close to its SFT distribution. Point the exact same loss at “dozens of self-played trajectories across ten structurally different games, ranked by which one gathered information fastest,” sampled with enough temperature to stay diverse, and you get an agent that generalizes exploratory strategy to games it has never played. Neither paper claims to have solved the other's problem. What they jointly demonstrate is that the tension between “converge toward known-good behavior” and “stay diverse enough to discover unknown-good behavior” is not resolved inside the DPO loss at all — it is resolved entirely upstream, in the sampling temperature, the task design, and the choice of what counts as y_w.

Where this leaves you

A speculative closing question, ours, not either paper's

Neither paper sets out to build a single system that is both safely aligned and genuinely curious, so take this as this lesson's own synthesis, not a claim either set of authors makes. Everything in this chapter traces back to one dial: β, and what π_ref it's measured against. What if that dial weren't a single global number, but varied by context — a tight, alignment-style leash (small effective β-freedom, or a π_ref reflecting careful, safety-vetted behavior) on the content of what a model says to a person, and a loose, Paprika-style leash on how it explores within a sandboxed tool-use or information-gathering loop, before it ever produces a user-facing response? Chapter 2 showed the KL term's only job is deciding how far a policy may wander from a reference; nothing in the math says that reference has to be the same model, or that single number has to be the same, in every situation a system finds itself in. Building that well — a leash that tightens exactly where safety matters and loosens exactly where exploration is the point — is a genuinely hard systems problem this lesson has given you the vocabulary to describe precisely, even though neither paper attempts to solve it.

Feynman's version of this lesson's whole argument might be: an equation doesn't know what it's for. DPO's loss is nine lines of algebra derived from a change of variables; it has no opinion about whether “good” means polite or means curious. That opinion lives entirely in the data you feed it — which is, in the end, the most useful thing to remember the next time you reach for a preference -optimization loss and assume the hard part is the math.

References

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” NeurIPS 2023. arXiv:2305.18290
2. Tajwar, F., Jiang, Y., Thankaraj, A., Rahman, S. S., Kolter, J. Z., Schneider, J., Salakhutdinov, R. “Training a Generally Curious Agent.” 2025. arXiv:2502.17543
3. Bradley, R. A., Terry, M. E. “Rank analysis of incomplete block designs: I. The method of paired comparisons.” Biometrika, 1952.
4. Razin, N. et al. “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization.” 2024. (cited by Paprika for the RPO motivation)
5. Pang, R. et al. “Iterative Reasoning Preference Optimization” (RPO loss). 2024. (cited by Paprika)
6. Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., Slivkins, A. “Can large language models explore in-context?” 2024. (cited by Paprika for the bandit-exploration baseline)