Why DPO Is a Misspecified Estimator

Chapter 1: The Key Insight

Here is the paper's central insight, stated plainly: DPO loss minimization is equivalent to a weighted KL-projection of the true reward function onto the manifold of rewards that the policy class can express.

Let's unpack that one piece at a time.

What is an "implicit reward"?

Given any policy π_θ and a reference policy π_θ0, DPO defines an implicit reward function:

This is just the log-probability-ratio, scaled by β. Every policy parameter θ induces exactly one implicit reward. The set of all such implicit rewards, as θ varies over R^d, forms a manifold R^β inside the full reward space R^m (where m = |S| · |A|).

What does DPO actually minimize?

When you minimize the DPO loss with infinite data, you're finding the implicit reward in R^β that is closest to the true reward r* — where "closest" means minimizing a weighted sum of KL divergences between Bernoulli preference probabilities. The weights are the preference pair counts n_s,a,a'.

When does this go wrong?

If r* happens to live on the manifold R^β (i.e., some policy parameter θ yields an implicit reward equal to r*), then DPO finds the RLHF-optimal policy. No problem.

But if r* is off the manifold — which is the generic case when d << m — then DPO projects r* onto R^β. This projection depends on the weights (preference counts), and can land at a point on the manifold that corresponds to a worse policy than doing nothing at all.

The fix: AuxDPO

The paper then studies the geometry of two-stage RLHF and discovers that RLHF partitions all reward functions into equivalence classes: rewards that differ by a vector in the nullspace of a certain matrix all yield the same optimal policy. DPO can only search the column space. AuxDPO adds auxiliary variables that search the nullspace too, expanding the search from a d-dimensional manifold to the full m-dimensional reward space. The result: the projection is no longer misspecified.

Chapter 2: Prerequisites

Before we dive into the proofs, let's make sure every tool is sharp. This chapter covers the three building blocks: the Bradley-Terry preference model, the DPO loss, and the geometry of misspecified estimation.

The Bradley-Terry-Luce (BTL) Model

Given a prompt s and two responses a, a', how do we model which one a human prefers? The BTL model says: each response has a latent "reward" r*(s, a), and the probability that a is preferred over a' is:

where σ(z) = 1/(1 + e^−z) is the sigmoid function.

Let's build some intuition with numbers. Suppose r*(a₁) = 3 and r*(a₂) = 1. Then:

So a₁ is preferred 88% of the time. The bigger the reward gap, the stronger the preference. If rewards are equal, preference is 50-50 (since σ(0) = 0.5).

The KL-Regularized RLHF Objective

Given a reward function r* and a reference (base) policy π_ref = π_θ0, the RLHF objective maximizes expected reward while penalizing deviation from the base:

Here β > 0 controls how far the aligned policy can stray from the base. Large β means "stay close to the base" (conservative alignment). Small β means "chase the reward aggressively."

Worked example: Suppose you have one prompt, two responses, π_ref(a₁) = π_ref(a₂) = 0.5, and r*(a₁) = 1, r*(a₂) = 0. If β = 1:

Setting p = π_θ(a₁):

Taking the derivative and setting to zero yields the optimal p* = e^1/β / (1 + e^1/β) = σ(1/β). For β = 1, that's σ(1) ≈ 0.73. So the optimal policy puts 73% probability on the preferred response — not 100%, because the KL penalty holds it back.

The DPO Reparameterization

For a tabular policy class, the RLHF objective has a closed-form maximizer:

Rearranging for the reward:

Substituting this into the BTL model, the Z*(s) terms cancel (since they're the same for both responses at the same prompt), giving:

DPO says: instead of learning r* first and then optimizing the policy, directly learn θ by treating the above as a binary classification loss. The DPO loss is:

Misspecified Estimation

In statistics, an estimator is misspecified when the true data-generating distribution doesn't belong to the model family being fit. Classic example: fitting a linear model to quadratic data. The fit converges (with infinite data) to the best linear approximation — but that approximation can be misleading.

White (1982) showed that misspecified maximum likelihood estimators converge to the KL-projection of the truth onto the model family. The projection is consistent (it converges) but not to the right thing. DPO exhibits exactly this phenomenon: it's a misspecified estimator in reward function space.

Chapter 3: Setup & Definitions

Now let's formalize everything. We need precise definitions because the paper's results are about exact mathematical objects, not hand-wavy intuitions.

The Data

We have a dataset D of n preference triplets (s⁽ⁱ⁾, a_w⁽ⁱ⁾, a_l⁽ⁱ⁾), where a_w ≻ a_l means "a_w is preferred over a_l at prompt s." Prompts come from S (finite set), responses from A (finite set). Let m = |S| · |A| be the total number of (prompt, response) pairs.

The Policy Class

π_θ : S → Δ(A) is parameterized by θ ∈ R^d. The base policy is π_θ0. Two key cases:

• Tabular: d = m, each θ_s,a is a free parameter, π_θ(a|s) ∝ θ_s,a. The policy can represent anything.
• Parametric (e.g., neural softmax): d << m. π_θ(a|s) = exp(f_θ(s, a)) / ∑_a' exp(f_θ(s, a')), where f_θ is a neural network. The policy lives on a d-dimensional manifold in the m-dimensional simplex.

The Implicit Reward

For any policy parameter θ, define the implicit reward function:

Note that r_θ0^β ≡ 0 by definition (the base policy has zero implicit reward). This makes sense: we're measuring reward relative to the base.

The Implicit Reward Manifold

The set of all achievable implicit rewards forms:

For a tabular policy, R^β = R^m (you can reach any reward). For a parametric policy, R^β is a d-dimensional manifold embedded in R^m, where d << m. This dimensional mismatch is the source of all trouble.

The Population DPO Loss

With infinite data (population setting), the number of preferences for triplet (s, a, a') is n_s,a,a' and the fraction that prefer a over a' is p_s,a,a'^BTL(r*). The population DPO loss is:

This is a weighted cross-entropy loss. Each pair (s, a, a') contributes a binary cross-entropy term, weighted by how many times that pair was compared (n_s,a,a'). The "label" is the true BTL preference probability; the "prediction" is the preference probability induced by the policy's implicit reward.

The Jacobian Matrix A_θ0

A critical object: the d × m matrix where the (s, a)-th column is ∇ log π_θ0(a|s). This is the score function of the base policy, evaluated at every (prompt, response) pair.

Why does this matter? Because the linearized implicit reward is:

So the column space C(A_θ0^T) is the linearized implicit reward manifold. Whatever isn't in this column space is invisible to DPO.

Chapter 4: DPO as KL-Projection

This is the paper's first main theorem (Proposition 1). It reveals what DPO actually does in the population setting.

The Theorem

Derivation: Why This is True

Let's derive this step by step, filling in the gaps the paper skips.

Step 1. The population DPO loss is:

where p* = p_s,a,a'^BTL(r*) is the true preference probability and q(θ) = p_s,a,a'^BTL(r_θ^β) is the model's predicted preference probability.

Step 2. Rewrite using the identity H(p, q) = H(p) + d_KL(p || q), where H(p, q) is binary cross-entropy and H(p) is entropy:

The entropy term H(p*) doesn't depend on θ, so:

Step 3. Since θ determines r_θ^β, and r_θ^β ranges over R^β as θ varies, this is equivalent to:

This is a weighted KL-projection of r* onto R^β. QED.

What This Means

Three critical implications:

1. Realizable case (r* ∈ R^β): The KL divergence is zero at r* itself, so the projection trivially finds r*, and DPO recovers the RLHF-optimal policy. This is the tabular case where DPO was designed to work.

2. Misspecified case (r* ∉ R^β): The projection finds the closest point on R^β in the weighted KL sense. But "closest in weighted KL" doesn't mean "best policy." The weights n_s,a,a' — how many times each pair was compared — control where the projection lands. Different preference distributions give different answers.

3. The weights are the problem: In RLHF, the preference counts affect how well you learn the reward (statistical efficiency), but the learned reward is still the right one with enough data. In DPO, the preference counts determine which wrong answer you converge to. This is fundamentally different.

Worked Numerical Example

Let's trace through the paper's 3-response example from Proposition 3. One prompt, three responses a₁, a₂, a₃ with true rewards r* = [1, 2, 0] (after shifting so the minimum is 0). Policy: π_θ = [e^θ, e^−θ, 1]/Z, base: θ₀ = 0 (uniform).

The Jacobian: A_θ0 = [1, −1, 0]. So the implicit reward manifold is span([1, −1, 0]) — a line in R³.

The true reward r* = [1, 2, 0] is not on this line (it would need to be of the form [c, −c, 0]). DPO must project.

Now the projection depends on the preference counts. If n_1,3 (comparisons between a₁ and a₃) dominates, DPO is primarily trying to match the a₁ vs. a₃ preference probability. Since r*(a₁) − r*(a₃) = 1 > 0, DPO will set the implicit reward to have r_θ(a₁) − r_θ(a₃) > 0, which means r_θ ≈ [α, −α, 0] for some α > 0. This makes a₁ preferred over a₂ — a preference reversal since the true ordering is a₂ ≻ a₁ ≻ a₃.

Chapter 5: Failure Modes

Now we see DPO fail. Not because of bad data, not because of poor optimization, not because of insufficient training — but because of the geometry of misspecification.

The 3-Response Counterexample (Proposition 3)

This is the paper's most striking result. Let's set it up carefully.

Setup: One prompt. Three responses a₁, a₂, a₃. True rewards: r* = [2, 3, 1]. True preference order: a₂ ≻ a₁ ≻ a₃. Policy: π_θ = [e^θ, e^−θ, 1]/Z with a single parameter θ. Base policy: θ₀ = 0 (uniform, each action gets probability 1/3).

Since BTL only cares about differences, we shift to r* = [1, 2, 0]. The Jacobian at θ₀ = 0:

So R^β ≈ span([1, −1, 0]). The manifold is a line through the origin in the direction [1, −1, 0].

When n_1,3 Dominates

Suppose the dataset is imbalanced: lots of (a₁ vs. a₃) comparisons, few (a₁ vs. a₂) or (a₂ vs. a₃) comparisons. Then DPO mostly cares about matching the a₁ vs. a₃ preference.

True preference: P(a₁ ≻ a₃) = σ(1 − 0) = σ(1) ≈ 0.73. DPO finds an implicit reward on the line [α, −α, 0] that matches this. Setting r(a₁) − r(a₃) = α − 0 = α, DPO needs σ(α) ≈ 0.73, giving α ≈ 1.

The resulting implicit reward: r_DPO ≈ [1, −1, 0]. Now check the implied preference order:

• r_DPO(a₁) = 1, r_DPO(a₂) = −1, r_DPO(a₃) = 0
• Implied order: a₁ ≻ a₃ ≻ a₂
• True order: a₂ ≻ a₁ ≻ a₃

Reward Degradation

It gets worse. The expected reward under the DPO policy is lower than under the base policy.

With θ = α > 0:

Let's compute for α = 1:

Compare with the base policy (θ = 0):

So DPO achieves expected reward 0.845 vs. the base policy's 1.0. DPO made things worse.

Two-stage RLHF, by contrast, would learn r* accurately in stage 1, then optimize the policy in stage 2. RLHF's expected reward can only increase from the base policy (the base policy is always a feasible point for the KL-regularized objective, so the optimizer does at least as well).

Sensitivity to Preference Distribution

The killer: which failure mode you get depends entirely on which pairs are compared more often.

• If n_1,2 dominates: DPO needs σ(r(a₁) − r(a₂)) ≈ σ(−1) ≈ 0.27, so r(a₁) − r(a₂) ≈ −1. On the line [α, −α, 0], this means 2α ≈ −1, so α < 0 and θ < 0. The policy correctly favors a₂! Success.
• If n_1,3 dominates: Preference reversal and reward degradation (as shown above). Failure.
• If n_2,3 dominates: DPO needs σ(r(a₂) − r(a₃)) ≈ σ(2) ≈ 0.88. On the line: r(a₂) − r(a₃) = −α − 0 = −α, so −α ≈ 2, α ≈ −2, θ ≈ −2. Again favors a₂. Success, but degree of success depends on the count.

Not a Coverage Problem

Remark 4 in the paper emphasizes: this failure is not due to insufficient coverage. Song et al. (2024) argued that DPO needs a coverage condition max_s,a π_θ(a|s)/π_θ0(a|s) ≤ C. In our example, C = 3 for the uniform base policy — coverage is perfect. But DPO still fails. The problem is geometric (misspecification), not statistical (coverage).

Chapter 6: RLHF Geometry

Having seen DPO fail, the paper now asks: what does two-stage RLHF actually compute, geometrically? Understanding RLHF's local behavior will reveal the path to fixing DPO.

Local Approximation of the RLHF Objective

We approximate J(θ; r*) around the base policy θ₀ using Taylor expansions. The expected reward is linear in θ (first-order), and the KL penalty is quadratic (second-order):

Let's define the key matrices:

• D_ρ,θ0: Diagonal matrix with entries ρ(s)π_θ0(a|s) — the base distribution over (s, a) pairs.
• A_ρ,θ0 = A_θ0 D_ρ,θ0: The "scaled Jacobian" — columns are ρ(s) ∇π_θ0(a|s) (note: ∇π, not ∇logπ).
• F_ρ,θ0 = A_θ0 D_ρ,θ0 A_θ0^T = A_ρ,θ0 A_θ0^T: The Fisher information matrix. This is the "natural metric" on policy space — it measures how quickly the KL divergence grows as you move θ.

The Natural Gradient Solution

Taking the gradient of the quadratic approximation and setting it to zero:

Solving:

Equivalence Classes of Reward Functions

Here's the key insight for fixing DPO. The RLHF solution θ* depends on r* only through the product A_ρ,θ0 r*. This means:

All reward functions in the same equivalence class produce the same RLHF-optimal policy. Two rewards r₁, r₂ are equivalent if and only if r₁ − r₂ ∈ N(A_ρ,θ0) — they differ by a nullspace element.

Worked example: For our 1D policy with A_θ0 = [1, −1, 0] and uniform base policy, A_ρ,θ0 = (1/3)[1, −1, 0]. Its nullspace N(A_ρ,θ0) = {[a, a, b] : a, b ∈ R} — any reward where a₁ and a₂ have the same reward (regardless of a₃'s reward).

So r* = [1, 2, 0] and r* + [c, c, d] = [1+c, 2+c, d] all yield the same RLHF policy. That's because the 1D policy can only control the ratio of a₁ to a₂, and any reward with r(a₂) − r(a₁) = 1 gives the same θ*.

The DPO-RLHF Connection (Proposition 7)

DPO's implicit reward r_θ^β is the minimum-norm representative of the RLHF equivalence class R_eq^β(θ), measured in the Mahalanobis norm ||r||_Dρ,θ0.

In other words: for each θ, there's a whole affine subspace of rewards that would make RLHF choose θ. DPO picks the shortest one. This is elegant — but it means DPO is constrained to the column space C(A_θ0^T), while the true reward r* may require a nullspace component to be properly represented.

Chapter 7: The AuxDPO Algorithm

Now the fix. The authors have identified the problem: DPO searches the column space C(A_θ0^T), but the true reward has a nullspace component that distorts the projection. The solution: search both spaces simultaneously.

The Core Idea

Introduce auxiliary variables δ ∈ N(A_ρ,θ0) that represent the nullspace component of the reward. Optimize the DPO loss jointly over θ (column space) and δ (nullspace):

where L(θ, δ) is the DPO loss but with the reward replaced by r_θ^β + δ:

where r_θ,δ^β = r_θ^β + δ.

Why This Works (Proposition 9)

By the rank-nullity theorem:

So searching column space (θ) + nullspace (δ) covers the full m-dimensional reward space R^m. The reward r* = r_θ*^β + δ* is now realizable in the augmented representation. The misspecification vanishes.

From Theory to Practice: The Empirical Loss

The theoretical formulation requires knowing N(A_ρ,θ0) and optimizing δ over it. In practice, we can't compute the nullspace of a matrix with millions of rows. The paper uses a clever relaxation:

Step 1: Discretize δ. Instead of defining δ over all m = |S| · |A| pairs, define it only at the 2n data points that appear in the dataset: δ = {δ(s⁽ⁱ⁾, a_w⁽ⁱ⁾), δ(s⁽ⁱ⁾, a_l⁽ⁱ⁾)}_i=1ⁿ ∈ R²ⁿ.

Step 2: Enforce the nullspace constraint via a penalty. Replace the hard constraint δ ∈ N(A_ρ,θ0) with a penalty term ||A_ρ,θ0 δ||². Since A_ρ,θ0 δ = E_ρ,πθ0[δ(s, a) ∇ log π_θ0(a|s)], this can be estimated from the dataset.

The empirical AuxDPO loss:

Implementation Details

Pseudocode

# AuxDPO Training Loop
def auxdpo_loss(theta, delta, batch, ref_model, beta, lam):
    # batch: (s, a_w, a_l) triplets, indices i

    # 1. Compute implicit rewards (same as DPO)
    log_ratio_w = log_prob(theta, s, a_w) - log_prob(ref_model, s, a_w)
    log_ratio_l = log_prob(theta, s, a_l) - log_prob(ref_model, s, a_l)
    r_w = beta * log_ratio_w   # shape: [batch_size]
    r_l = beta * log_ratio_l   # shape: [batch_size]

    # 2. Add auxiliary variables (per-datapoint scalars)
    delta_w = delta[2*i]        # shape: [batch_size]
    delta_l = delta[2*i + 1]    # shape: [batch_size]

    # 3. AuxDPO logit = DPO logit + delta correction
    logit = (r_w - r_l) + (delta_w - delta_l)

    # 4. Binary cross-entropy loss (same as DPO)
    bce = -log_sigmoid(logit).mean()

    # 5. Nullspace penalty: ||A_{rho,theta_0} delta||^2
    # score_w = grad log pi_{theta_0}(a_w | s), shape: [batch, d]
    # Pre-computed and cached for reference model
    penalty = (delta_w[:, None] * score_w + delta_l[:, None] * score_l).mean(0)
    penalty = (penalty ** 2).sum()

    return bce + lam * penalty

Chapter 8: Experiments

Theory says AuxDPO should fix DPO's misspecification. Does it work in practice? The paper tests on two fronts: a didactic bandit setting (where we can verify the theory exactly) and real LLM alignment tasks.

Didactic Bandit Setting

The 3-response, 1-parameter example from Proposition 3 is implemented with a log-linear policy. With n_1,3 dominating, DPO produces preference reversal as predicted. AuxDPO with λ = 1.0 correctly recovers θ < 0, favoring a₂. The auxiliary variable δ absorbs the nullspace component, steering the projection to the correct equivalence class.

LLM Alignment: Datasets

• RewardBench v2 (Malik et al., 2025): 1.87K prompts testing factuality, instruction following, and focus. Each prompt has a chosen and rejected response.
• MMLU-Pro (Wang et al., 2024b): 12K multi-task understanding questions, 10 possible answers each. Converted to preference format by pairing the correct answer (chosen) with each incorrect one (rejected).

Training data: UltraFeedback (Cui et al., 2024) and its binarized version. Models: Llama3.1-8B, Llama3.2-1B, Qwen3-0.6B.

Main Results

Values show % change in mean accuracy relative to the base policy. Bold = best. Negative = degradation from base policy.

Reading the Results

Several patterns jump out:

1. AuxDPO wins everywhere that matters. On all 12 (model, dataset, setting) combinations, AuxDPO is either first or tied for first. DPO is never best.

2. The OOD gap is huge. On Llama3.2-1B RewardBench v2 OOD, AuxDPO achieves +43.27% vs. DPO's +14.11%. That's a 3x improvement in OOD generalization. This suggests that DPO's misspecification causes it to overfit to distribution-specific artifacts, while AuxDPO finds more robust alignments.

3. DPO can be catastrophically bad. On Qwen3-0.6B RewardBench v2 OOD, DPO scores −8.16% — meaning it's worse than the base model. This is exactly the reward degradation predicted by the theory (Proposition 3). AuxDPO scores +18.36%, a swing of over 26 percentage points.

4. Model size matters. The smaller the model (fewer parameters d relative to task complexity m), the worse the misspecification. Qwen3-0.6B has the most severe DPO failures, which makes sense: fewer parameters means a lower-dimensional manifold, means more reward functions are unrealizable.

Per-Subject Breakdown (MMLU-Pro)

The paper also reports per-subject accuracies on MMLU-Pro. Some highlights:

• Biology OOD: AuxDPO gets 75.07% vs. DPO's 55.93% — a 19-point gap.
• Engineering ID: AuxDPO gets 32.18% vs. DPO's 10.22% — tripling the improvement.
• Law OOD: DPO gets 23.61% (degraded from base), AuxDPO gets 34.50% (improved).

The gains are not uniform across subjects, but AuxDPO is never worse than DPO in any subject.

Chapter 9: Connections

Cheat Sheet: Every Key Equation

Related Papers & Lessons

• DPO (Rafailov et al., 2023): The original paper that this work analyzes and fixes. See our Reward Alignment lesson for DPO basics.
• IPO (Azar et al., 2023): An alternative to DPO that uses a squared loss instead of log-sigmoid. Not a fix for misspecification per se, but sidesteps some of DPO's sensitivity issues.
• DPOP (Pal et al., 2024): Fixes likelihood displacement (where chosen response probability decreases) but doesn't address the geometric misspecification that this paper identifies.
• RLHF (Ziegler et al., 2019): The two-stage approach that DPO was designed to replace. This paper shows RLHF's local behavior is a natural gradient step, and its equivalence class structure is what makes it robust to misspecification.
• Natural Policy Gradient (Kakade, 2001): The RLHF solution (Eq. 5) is exactly a natural gradient update. See our Policy Gradients lesson.
• Misspecified MLE (White, 1982): The statistical foundation. Misspecified MLEs converge to KL projections — consistent in the wrong sense.

Open Questions

• How bad is misspecification for specific architectures? The paper shows misspecification exists for any parametric class. But how much does it matter for a 70B Transformer vs. a 1B one? Is there a sweet spot where d is large enough that misspecification is negligible?
• Can we estimate misspecification? Is there a diagnostic that tells you "DPO is heavily misspecified on your data" without running AuxDPO? Perhaps by measuring the nullspace penalty term after DPO training.
• Scaling the penalty computation. The score function cache for the nullspace penalty grows as O(n × d). For very large datasets and models, this could be prohibitive. Can the penalty be approximated with a sketch or random projection?
• Beyond BTL. The analysis assumes the Bradley-Terry preference model. Real preferences are nosier, inconsistent, and context-dependent. Does AuxDPO still help when the BTL assumption itself is misspecified?

The Big Picture

This paper reveals a fundamental tension in direct alignment: DPO's elegance comes from reparameterizing the reward in terms of the policy, but this reparameterization constrains the reward space to a low-dimensional manifold. The constraint is invisible in the tabular case (where the manifold is the whole space) but becomes a source of systematic error for parametric policies.

AuxDPO's fix is principled and minimal: add auxiliary variables that search the directions the policy can't reach. The cost is small (2n extra scalars, one hyperparameter), and the theory guarantees recovery of the RLHF solution. The practical gains are substantial, especially for smaller models where misspecification is more severe.

The deeper lesson: whenever you reparameterize one quantity in terms of another (reward in terms of policy, here), you inherit the latter's capacity limitations. If those limitations don't match the true data, you're misspecified — and no amount of data will save you. Only expanding the representational capacity (here, via auxiliary variables) can fix a misspecification problem.