RL Theory

KL ≈ 0.184 nats. The policy concentrates mass on token 1 (0.7 vs 0.4), driving the positive first term. The other terms are negative because π assigns less mass than π_ref. Accept values in the range 0.18–0.23 depending on rounding.

The KL penalty "costs" 0.8 reward units. If β were 0.5 instead, the cost would be 4.0 — making the objective negative (−1.0), which would discourage this deviation from the reference model entirely.

The optimal policy up-weights high-reward tokens by exp(r/β) relative to the reference. Token 1 goes from 0.5 to 0.784. With smaller β, the shift would be even more extreme. Accept 0.67–0.79 depending on rounding.

javascript
function rlhfObjective(reward, logProbs, refLogProbs, beta) {
  let kl = 0;
  for (let t = 0; t < logProbs.length; t++) {
    kl += logProbs[t] - refLogProbs[t];
  }
  return reward - beta * kl;
}

The new policy assigns ~65% more probability to this action. This ratio > 1 means the new policy favors this action more than the old one did.

The ratio 1.65 exceeds 1+ε=1.2, so the clip activates. The effective gradient treats the ratio as 1.2, preventing a too-large update. Note: when A_t > 0 (good action), the clip caps the benefit. When A_t < 0 (bad action), the clip prevents the ratio from dropping below 0.8.

Example 1 gets clipped (ratio too high for positive advantage). Example 2 is within bounds but note the clip on the lower side activates (0.819 vs 0.8 — close). Example 3 is unchanged (ratio = 1.0). Accept 0.93–1.0 range.

Positive advantage means this response was better than average. PPO will increase the probability of generating similar responses. If the advantage were negative, PPO would decrease it. The value baseline V(x) is crucial for reducing gradient variance.

A positive Δ means the policy has increased the log-prob of the winner relative to the reference more than it has for the loser. This is exactly what we want. The DPO loss is −log σ(β · 1.5).

Loss is 0.62. The minimum possible DPO loss is 0 (when the policy perfectly separates winners from losers with infinite margin). A random policy that doesn't distinguish them gets loss = log(2) ≈ 0.693. We're slightly better than random. Accept 0.59–0.63.

When the model already strongly prefers the winner (σ near 1), the gradient becomes tiny. This is automatic curriculum: DPO focuses its gradient budget on examples it hasn't learned yet. Correctly classified pairs contribute almost nothing to the gradient. This is the same property as logistic regression.

javascript
function dpoLoss(logProbsWin, refLogProbsWin, logProbsLose, refLogProbsLose, beta) {
  let totalLoss = 0;
  for (let i = 0; i < logProbsWin.length; i++) {
    const logRatioW = logProbsWin[i] - refLogProbsWin[i];
    const logRatioL = logProbsLose[i] - refLogProbsLose[i];
    const delta = logRatioW - logRatioL;
    totalLoss += Math.log(1 + Math.exp(-beta * delta));
  }
  return totalLoss / logProbsWin.length;
}

Line 6 is the bug. It uses ratioW + ratioL instead of ratioW - ratioL. The DPO loss computes the difference in log-ratios: how much more the policy favors the winner versus the loser relative to the reference. Adding them together makes the delta large positive (both ratios tend to be negative small numbers, their sum is a larger negative, −β·Δ becomes large positive, so exp becomes huge, log becomes huge — wait, actually the signs depend on training progress. For an untrained model π ≈ π_ref, both ratios ≈ 0, so delta ≈ 0, giving loss = log(2) ≈ 0.693. The bug makes it converge to 0 because adding two near-zero values stays near zero while subtraction would too — the real issue is the gradient points in the wrong direction, causing the model to increase both winner AND loser probabilities equally rather than separating them.

The regularizer is negative here because the dataset actions happen to have above-average Q-values. With α > 0, this actually decreases the total loss, slightly encouraging these Q-values. The regularizer becomes positive (penalizing) when OOD actions have inflated Q-values — which is exactly when conservatism matters most.

The CQL regularizer (2.0) dominates the TD error (0.0016). This is typical: in early training, the conservatism penalty does most of the work, preventing the agent from being overconfident about OOD actions.

LogSumExp ≈ 3.41, which is slightly above max(Q) = 3.0. LogSumExp is a "soft max" — it's dominated by the largest Q-value but adds a smooth correction. Using LogSumExp instead of uniform average focuses the conservatism penalty on the highest Q-values, which are the most likely to be overestimated OOD actions.

javascript
function cqlRegularizer(qValues, datasetActionIndices) {
  // LogSumExp with numerical stability
  const maxQ = Math.max(...qValues);
  let sumExp = 0;
  for (const q of qValues) sumExp += Math.exp(q - maxQ);
  const lse = maxQ + Math.log(sumExp);

  // Mean Q for dataset actions
  let dataQ = 0;
  for (const idx of datasetActionIndices) dataQ += qValues[idx];
  dataQ /= datasetActionIndices.length;

  return lse - dataQ;
}

The underestimation (Q > V) gets weight 0.7, the overestimation gets 0.3. This asymmetry pushes V upward toward the higher Q-values. If τ = 0.5, both weights would be 0.5 and V would converge to the mean (3.0). With τ = 0.7, V converges to something between the mean and the max.

javascript
function expectileLoss(qValues, vValues, tau) {
  let total = 0;
  for (let i = 0; i < qValues.length; i++) {
    const diff = qValues[i] - vValues[i];
    const w = diff > 0 ? tau : (1 - tau);
    total += w * diff * diff;
  }
  return total / qValues.length;
}

At τ=0.9, the expectile (7.4) is much closer to the max (10.0) than the mean (4.0). This is how IQL approximates the max Q-value without ever computing max explicitly. Higher τ = more optimistic = closer to the best actions in the dataset.

A 6× reduction in real environment interaction. In robotics, where each real step requires moving physical hardware, this is enormous. The caveat: model accuracy matters. If the model is 80% accurate, only ~80% of the simulated updates help; some may actively hurt.

After 20 steps, even a 2%-accurate model has 40–49% cumulative error. This is why practical model-based methods use short rollouts (H = 1 to 5) and retrain the model frequently. MBPO (Janner et al., 2019) adaptively sets H based on model uncertainty.

Only 5.5 MB — synthetic data is cheap to store. The real cost is the forward passes through the ensemble: 30,000 transitions × 5 ensemble members = 150,000 model forward passes per iteration. This is why model-based RL trades compute for sample efficiency.

Moving closer to the goal earns a positive shaping bonus (~1.04). Moving away would earn a negative bonus. This guides the agent toward the goal without changing which policy is optimal. The small γ discount means moving closer sooner is slightly better than later.

The total shaping bonus telescopes to γ^T+1Φ(s_T+1) − Φ(s₀). With γ=1 and terminal Φ=0, it's just −Φ(s₀). Since Φ(s₀) is a constant (only depends on the start state), it shifts all trajectory returns by the same amount — the ranking of policies is unchanged. This is why potential-based shaping is "free."

Option C is best. The shortest-path distance through the maze accounts for walls and dead ends, providing the most informative gradient. Euclidean distance (B) can mislead near walls. The step count (D) is not a function of state alone and would create positive reward cycles. No shaping (A) works but requires much more exploration.

Check each cell:

No pure-strategy Nash equilibrium exists! There is a mixed-strategy Nash. However, re-examining: (Up, Left) has P2 wanting to deviate. The exercise asks for P1's payoff which in the mixed equilibrium works out. But actually (Up, Left) with P1 payoff 3: if this were a best-response dynamic, P1 plays Up when P2 plays Left, and P2 plays Right when P1 plays Up. The pure Nash is (Up, Left) only if neither wants to deviate — but P2 does. The answer 3 corresponds to the best P1 can guarantee via mixed strategy. Accept 2-3 range.

This is the tragedy of the Prisoner's Dilemma: the Nash equilibrium (Defect, Defect) is Pareto-dominated by (Cooperate, Cooperate). Individual rationality leads to collective irrationality. In MARL, this means agents trained independently via self-play can converge to suboptimal joint outcomes.

The unique Nash is both players randomizing 50/50. Expected payoff for both: 0. Any deviation from 50/50 would be exploitable by the opponent. This is the minimax theorem in action (von Neumann, 1928).

P1's expected payoff is 2.22. Could P1 do better? If P1 played pure Up: E = 0.3×4 + 0.7×1 = 1.9. Pure Down: E = 0.3×2 + 0.7×3 = 2.7. So P1 should shift more toward Down against P2's current strategy. Accept 2.2–2.3.

The reward model is almost the same size as the base model. The LM head was only ~130M parameters (2% of total). In practice, the reward model is often initialized from the SFT model checkpoint and fine-tuned on preference data.

55.2 GB just for weights. Add Adam optimizer states for the two trainable models: each needs 2 extra copies (m and v), so +7B × 4 + 7B × 4 = 56B extra params × 4 bytes (FP32) = 224 GB for optimizer states alone. Total: 55.2 + ~112 GB = ~167 GB. This is why RLHF for even a 7B model needs multiple A100/H100 GPUs.

DPO saves ~27 GB (49%) on weights alone. The optimizer savings are even larger: only 1 trainable model instead of 2, saving ~56 GB in Adam states. Total DPO memory: 28 + 56 = 84 GB, vs RLHF ~167 GB. DPO fits on 2 A100-80GB; RLHF needs 3-4.

SFT → Collect Preferences → Train Reward Model → PPO Training

First, supervised fine-tuning creates a capable model. Then humans compare its outputs to generate preference data. The reward model is trained on these preferences. Finally, PPO fine-tunes the SFT model using the reward model as a signal, with the SFT model as the reference π_ref.

About 8 A100-hours for 1 epoch. At ~$2/A100-hour, that's ~$16 per epoch. DPO typically runs 1-3 epochs, so total cost is $16-$48. This is remarkably cheap compared to pretraining (which costs $millions for 7B models). DPO's simplicity translates to real cost savings.

With β = 0.133, the reward exactly balances the KL cost. Any higher β and the model won't deviate enough from the reference to be useful. Any lower and reward hacking becomes a risk. Typical production values are β = 0.05 to 0.2, so 0.133 is in the sweet spot.

Line 9 is the bug. It updates the reference model weights instead of the policy model weights. The reference model must stay frozen throughout DPO training — it's the anchor that defines the KL penalty. Updating it means the "ground truth" shifts every step, destabilizing training entirely. The correct line should be model.updateWeights(lr).