Fine-tuning Large Language Models

Chapter 1: Prompting Strategies

Before we modify any weights, there’s a cheaper trick: change the input. By carefully designing the prompt, we can steer a pre-trained model toward useful behavior without training at all. This family of techniques is called prompting.

Zero-Shot Prompting

Just ask the question directly: “Translate ‘cheese’ to French.” No examples, no context. Large models (100B+ parameters) can often handle this thanks to the sheer breadth of patterns seen during pre-training. Smaller models struggle — they haven’t seen enough instruction-like text to recognize the intent.

Few-Shot Prompting

Prefix the question with worked examples: “English: dog → French: chien. English: cat → French: chat. English: cheese → French:” The model sees the pattern and completes it correctly. GPT-3 showed that few-shot performance improves dramatically with scale — a 175B model given 5 examples outperforms a 1.3B model given 100.

Chain-of-Thought (CoT)

Chain-of-thought prompting adds “Let’s think step by step” or includes a worked reasoning trace in the few-shot examples. Instead of jumping to the answer, the model generates intermediate steps. Wei et al. (2022) showed this unlocks reasoning abilities that flat prompting misses entirely — especially on math and logic problems.

Chapter 2: Instruction Tuning

Prompting is free but fragile. To truly teach a model to follow instructions, we fine-tune it on instruction-response pairs: (prompt, desired response) examples that explicitly demonstrate the behavior we want.

What Is Instruction Tuning?

Instruction tuning (also called supervised fine-tuning, SFT) takes a pre-trained LLM and continues training it on a curated dataset of instructions paired with ideal responses. The loss function is the same as pre-training — cross-entropy on next-token prediction — but the data is radically different: instead of raw web text, it’s task-specific demonstrations.

Where x is the instruction, y is the target response, and we sum over all tokens in y. We only compute loss on the response tokens, not the instruction tokens — because we want the model to generate good responses, not parrot back instructions.

Where Does the Data Come From?

Three sources, in order of quality and cost:

Concrete Example: InstructGPT

OpenAI’s InstructGPT started with a 175B GPT-3 base model and fine-tuned it on ~13,000 human-written instruction-response pairs. The result: a 1.3B InstructGPT was preferred by humans over the 175B base GPT-3. A 100x smaller fine-tuned model beat a 100x larger base model. That’s the power of alignment.

Chapter 3: The RLHF Pipeline

Instruction tuning teaches the model to follow instructions, but it can’t capture preferences. Which of two grammatically correct summaries is more helpful? Which response is less toxic? These are judgment calls that supervised loss can’t express — you need a signal that says “response A is better than response B.”

Reinforcement Learning from Human Feedback (RLHF) solves this in three stages. Each stage builds on the previous one, and the entire pipeline transforms a base model into an aligned assistant.

Why RL? Why Not Just More Supervised Data?

Supervised fine-tuning requires explicit demonstrations: “given this prompt, produce this response.” But for many tasks, it’s much easier for a human to compare two responses than to write the perfect one. RLHF leverages this asymmetry. Labelers rank outputs; a reward model learns from the rankings; RL optimizes the LLM to maximize that learned reward.

Christiano et al. (2017) proved the concept in Atari and MuJoCo: they trained agents to match human preferences using less than 1% of the environment interactions that standard RL required. The key insight was training a reward model from human rankings, then using it as a dense reward signal for RL.

The Data Flow

RLHF Dramatically Improves Scaling

Stiennon et al. (2020) showed that RLHF-trained summarization models outperform supervised baselines at every model size. More remarkably, a small RLHF model can match a much larger supervised model — RLHF shifts the entire scaling curve upward.

Chapter 4: Reward Modeling

Stage 2 of RLHF is where human judgment gets encoded into a neural network. The goal: train a model R_θ(x, y) that takes a prompt x and response y and outputs a scalar score — higher means “a human would prefer this response.”

Collecting Preferences

The SFT model generates two (or more) responses to each prompt. Human labelers see both and pick the better one, evaluating on criteria like helpfulness, harmlessness, and truthfulness. This produces comparison data: (x, y_w, y_l) where y_w is the preferred (“winner”) response and y_l is the rejected (“loser”) response.

The Bradley-Terry Model

We model human preferences using the Bradley-Terry model: the probability that a human prefers response y₁ over y₂ depends on the difference in their reward scores, passed through a sigmoid:

Where σ(z) = 1/(1 + e^−z) is the sigmoid function. If R gives y₁ a much higher score than y₂, the sigmoid pushes the probability toward 1 — the model is confident y₁ is preferred.

Deriving the Loss

We want to maximize the probability of the observed human preferences. Taking the negative log-likelihood:

Step by step: for each comparison triplet, compute the reward difference Δ = R(x, y_w) − R(x, y_l). Pass through sigmoid to get a probability. Take −log. Average over the dataset. Minimizing this loss pushes R_θ to assign higher scores to preferred responses.

Concrete Worked Example

Suppose for prompt “Explain gravity to a 5-year-old”, the reward model gives:

Δ = 2.3 − (−0.5) = 2.8. σ(2.8) = 0.943. Loss = −log(0.943) = 0.059. The model is already quite confident the simple answer is preferred. If the scores were closer, the loss would be higher, pushing the model to separate them more.

Chapter 5: PPO for Language Models

We now have a reward model R_θ that scores responses. Stage 3 uses this as the reward signal in a reinforcement learning loop. The algorithm of choice is Proximal Policy Optimization (PPO), adapted for language generation.

Framing Language Generation as RL

Here’s the mapping from language to RL:

The PPO Objective for LLMs

Standard PPO maximizes a clipped surrogate objective. For LLMs, we add a critical constraint: a KL penalty that prevents the policy from drifting too far from the SFT model. Without it, the policy would find degenerate outputs that exploit the reward model (called reward hacking).

Let’s unpack each term:

• R_φ(x, y) — the reward model’s score for the generated response. We want this to be high.
• D_KL(π_θ ‖ π_SFT) — the KL divergence between the current policy and the SFT reference. Measures how much the policy has changed. We want this to stay small.
• β — the KL penalty coefficient. A trust slider: high β keeps the model close to SFT (conservative); low β lets it explore more aggressively.

Why the KL Penalty Is Essential

Without the KL penalty, the model quickly discovers reward-hacking strategies: repetitive phrases the reward model scores highly, or adversarial outputs that exploit blind spots. The KL penalty says “maximize reward, but don’t become a different model.” It’s like a leash — the policy can explore, but it can’t run away.

Worked Example: One PPO Update

Given prompt x = “What is photosynthesis?”, the policy generates y = “Plants convert sunlight into energy using chlorophyll.”

Reward model gives R(x, y) = 1.8. The SFT model would have assigned this response probability p_SFT(y|x) = 0.03. The current policy gives p_θ(y|x) = 0.07.

KL divergence (simplified, per-sequence): D_KL ≈ log(0.07/0.03) = log(2.33) ≈ 0.85.

With β = 0.2: J = 1.8 − 0.2 × 0.85 = 1.8 − 0.17 = 1.63.

The policy gets credit for the high reward, with a small penalty for diverging from SFT.

Chapter 6: Constitutional AI & RLAIF

RLHF works, but it’s expensive. Collecting human preference labels requires hiring annotators, designing interfaces, running quality control. Anthropic’s Constitutional AI (CAI) asks: what if the AI itself could provide the feedback?

The Constitution

Instead of 10,000+ human comparison labels, CAI uses ~10 human-written principles — the “constitution.” These are natural-language rules like:

The Two-Phase Pipeline

CAI works in two phases that mirror RLHF but replace humans with AI:

Phase 1: Supervised Self-Critique

The LLM generates a response. Then it critiques its own response using a constitutional principle. Then it revises the response to address the critique. This critique-revision loop can be repeated multiple times. The revised responses become the SFT training data.

Phase 2: RLAIF — RL from AI Feedback

Instead of human labelers comparing responses, the AI compares them using the constitution. For each prompt, generate two responses, ask the AI “which response better follows this principle?” and use the AI’s judgment to train a preference model. Then run PPO as before, but with the AI-trained preference model.

Scaling Advantages

Chapter 7: Parameter-Efficient Fine-tuning

Fine-tuning a 70B model means storing 70B parameters, their gradients, and optimizer states (Adam keeps 2 extra copies per parameter). That’s easily 10× the model size in memory — 700 GB+ for a single training run. And if you want a different fine-tuned model for each task (summarization, coding, medical Q&A), you need separate 70B copies. This doesn’t scale.

Parameter-Efficient Fine-Tuning (PEFT) solves this by updating only a tiny fraction of the parameters — often less than 0.1% — while freezing the rest. The frozen parameters require no gradients and no optimizer states, slashing memory by 10-100×.

Three Families of PEFT

LoRA: Low-Rank Adaptation

LoRA (Hu et al., 2021) is the most widely used PEFT method. The key insight: fine-tuning weight changes have low intrinsic rank. Instead of updating a full d×d weight matrix W, LoRA decomposes the update into two small matrices:

Where A is [r × d] and B is [d × r], with rank r ≪ d (typically r = 4 to 64, while d = 4096 to 12288).

Deriving the Parameter Savings

A full weight matrix W has d × d parameters. The LoRA update BA has (d × r) + (r × d) = 2dr parameters. The compression ratio:

For d = 4096 and r = 8: ratio = 16 / 4096 = 0.39%. We’re updating less than half a percent of the parameters.

The Forward Pass

During training: h = Wx + BAx. The frozen W handles the bulk of computation. BA adds a small correction learned during fine-tuning. During inference, we can merge the matrices: W_LoRA = W + BA. After merging, the forward pass is just h = W_LoRAx — zero additional latency.

Prompt Tuning: Soft Prompts

Prompt tuning (Lester et al., 2021) prepends m learnable “soft prompt” tokens to the input embedding. Only these m × e parameters are trained (where e is the embedding dimension). The entire model is frozen. At scale (10B+ params), prompt tuning matches full fine-tuning performance with ~0.01% of the trainable parameters.

Trainable parameters: m × e. For m = 20 and e = 4096, that’s 81,920 parameters — compared to billions in the full model.

Prefix Tuning and P-Tuning v2

Prefix tuning (Li & Liang, 2021) extends prompt tuning to deeper layers: instead of only prepending tokens at the input, it injects learnable key-value pairs at every attention layer. P-Tuning v2 showed this approach matches full fine-tuning on smaller models where prompt tuning alone falls short.

python
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, linear, r=8, alpha=16):
        super().__init__()
        self.linear = linear          # frozen original layer
        d_out, d_in = linear.weight.shape
        self.A = nn.Parameter(torch.randn(r, d_in) * 0.01)
        self.B = nn.Parameter(torch.zeros(d_out, r))
        self.scale = alpha / r        # scaling factor
        linear.weight.requires_grad = False

    def forward(self, x):
        base = self.linear(x)         # W @ x (frozen)
        lora = (x @ self.A.T) @ self.B.T  # BA @ x (trainable)
        return base + lora * self.scale

Chapter 8: Connections & What’s Next

We’ve traced the full arc from a raw pre-trained model to an aligned, efficient assistant. Here’s the complete development flow:

Cheat Sheet

Method Comparison

What We Didn’t Cover

• DPO (Direct Preference Optimization) — skips the reward model entirely, directly optimizes the policy from preference data. Simpler than PPO, increasingly popular.
• QLoRA — combines LoRA with 4-bit quantization, enabling fine-tuning of 65B models on a single 48GB GPU.
• Mixture of Experts + PEFT — routing different inputs to different LoRA adapters.
• RLHF at scale — distributed PPO across hundreds of GPUs, managing reward model staleness.