Efficient Adaptation

Chapter 1: In-Context Learning

In 2020, GPT-3 demonstrated something remarkable: you could make it do new tasks without changing a single weight. No gradient updates. No training loop. You just write the task description in the prompt, and the model figures out what to do.

This is in-context learning (ICL) — the model "learns" the task from examples placed in its context window. The word "learns" is in quotes because the weights never change. Everything happens through the attention mechanism attending over the examples you provide.

Three Modes of In-Context Learning

Zero-shot: Describe the task, provide no examples. The model relies entirely on its pre-training knowledge. "Translate the following English text to French: 'The cat sat on the mat.'" The model has seen enough translation pairs during pre-training to handle this — but accuracy drops sharply on unusual formats or niche domains.

One-shot: Provide exactly one example, then the actual query. "English: 'Hello, how are you?' French: 'Bonjour, comment allez-vous?' English: 'The cat sat on the mat.' French:" The single example teaches the model the desired input-output format.

Few-shot: Provide 4-32 examples. More examples generally improve accuracy, but consume context window tokens. Each example "costs" tokens that could be used for the actual input. With a 2048-token context window, you might fit 10-20 short examples before running out of space for the actual query.

The GPT-3 paper showed dramatic improvements as you go from zero to few-shot, especially on tasks requiring specific output formats. On the SuperGLUE benchmark, zero-shot GPT-3 scored 55.4, while 32-shot GPT-3 scored 71.8 — a 16-point jump without touching the weights.

Why Does ICL Work?

This is still debated, but the leading theory is that during pre-training, GPT-3 encountered millions of sequences where context implied a task. Wikipedia articles implicitly teach "continue writing about this topic." StackOverflow threads implicitly teach "answer the question." Translation corpora implicitly teach "translate between languages."

When you provide few-shot examples, you're activating circuits the model already learned during pre-training. The examples don't teach new knowledge — they steer the model toward a pre-existing capability. Think of it as finding the right "mode" within a model that already knows how to do many things.

A compelling formal theory: Garg et al. (2022) showed that transformers trained on random linear regression tasks learn to implement ridge regression in their forward pass. The attention mechanism can, in principle, implement gradient descent over the provided examples. This means ICL may literally be performing a form of optimization internally — just not gradient-based optimization on the weights.

Scaling Laws for ICL

ICL ability improves with scale. The GPT-3 paper showed clear trends:

The pattern: ICL benefit grows superlinearly with model size. Small models barely benefit from examples. Large models extract far more from the same examples. This is an emergent ability — it appears to "switch on" above a certain model size threshold rather than increasing linearly.

ICL vs. Fine-Tuning: The Tradeoff

When should you use ICL instead of fine-tuning? The answer depends on three factors:

1. Data availability. ICL needs only a handful of examples (4-32). Fine-tuning typically needs 1K+ examples for SFT quality. If you have fewer than 100 labeled examples, ICL is often your only option.

2. Latency budget. ICL is available immediately — no training required. Fine-tuning takes hours to days. If you need to deploy in minutes (a new customer request, a rapidly changing task), ICL wins.

3. Accuracy requirements. For high-stakes applications (medical, legal, financial), ICL's ~70-80% accuracy ceiling may not be sufficient. Fine-tuning (or LoRA) can push accuracy to 90-95%+ on domain-specific tasks. The accuracy gap narrows with better prompting but never fully closes.

Many production systems use a hybrid approach: start with ICL for rapid prototyping, measure accuracy, then fine-tune with LoRA only if ICL falls short of requirements. This "ICL-first" workflow avoids premature optimization and often reveals that the task is simpler than expected.

Limitations

ICL is cheap and fast, but it hits a ceiling:

ICL in Practice: The Data Flow

Let's trace exactly what happens during a few-shot forward pass. Say you have 4 examples, each ~100 tokens, plus a 50-token query. Total input: 450 tokens.

python
# Few-shot prompt construction
examples = [
    ("The food was excellent", "Positive"),
    ("Terrible service", "Negative"),
    ("I loved the ambiance", "Positive"),
    ("Would not return", "Negative"),
]

prompt = "Classify the sentiment:\n"
for text, label in examples:
    prompt += f'{text} → {label}\n'
prompt += "The pasta was bland →"

# Input shape: [1, 450]  (batch=1, seq_len=450)
# Every token attends to all previous tokens
# The model's prediction for the last position
# is influenced by ALL 449 preceding tokens
logits = model(tokenize(prompt))  # [1, 450, vocab_size]
prediction = logits[0, -1]       # last token's distribution

The critical insight: attention at the final position can attend to every example token. The model doesn't have a special "example memory" — it uses the same attention mechanism it uses for all text. Examples are just more context.

For tasks where ICL falls short, we need methods that actually modify the model — but without the cost of full fine-tuning. The next chapters explore how.

Chapter 2: Prompt Engineering

Small changes in how you phrase a prompt can produce enormous accuracy swings. This isn't a quirk — it's a fundamental property of how language models process input. The model doesn't "understand" your intent the way a human does. It generates tokens conditional on the exact token sequence you provide. Change a single word and you shift the entire conditional distribution.

Format Matters More Than Content

Zhao et al. (2021) showed that simply reordering the same few-shot examples could swing accuracy on SST-2 (sentiment classification) from 54% to 93%. Same model, same examples, different permutation. The model was treating example order as a signal about the task — recency bias meant the last example disproportionately influenced the prediction.

Other formatting choices that have measured impact:

The Art of Prompt Design

Good prompt engineering follows a few principles:

Be explicit about output format. "Answer with exactly one word: Positive or Negative." Without this, the model might produce "I think this is positive because..." and your parser breaks.

Use the right verbalizer. A verbalizer maps between label names and the words the model uses to express them. If you're classifying sentiment, "Positive/Negative" works better than "1/0" because the model has seen far more text using those words in a sentiment context.

Select representative examples. The examples you choose should cover the distribution of inputs you expect. Don't use all easy examples — include edge cases. Don't use all similar examples — include diverse domains.

Calibrate against biases. Models have a tendency to favor certain labels regardless of input (the "majority label bias"). Zhao et al. proposed contextual calibration: measure the model's bias on a content-free input like "N/A", then adjust predictions accordingly.

Automatic Prompt Optimization

Humans are slow prompt engineers. Several methods automate the process:

Prompt tuning (Lester et al., 2021): learn a small set of continuous "virtual tokens" prepended to the input. These aren't real words — they're learned embeddings optimized via gradient descent. Only 0.01% of parameters are trainable. The rest of the model is frozen.

Prefix tuning (Li & Liang, 2021): similar idea, but prepend learned vectors to every layer's key-value pairs, not just the input embedding. This gives the "virtual prefix" direct influence over attention at every layer.

Both methods bridge the gap between pure prompting (zero parameters) and adapters (millions of parameters). They're effective for tasks where the pre-trained model already has the knowledge but needs steering — translation, classification, summarization.

Contextual Calibration

Zhao et al. (2021) proposed a simple fix for prompt sensitivity: contextual calibration. The idea:

Concretely: if the model assigns 70% to "Positive" on a content-free input, it has a strong positive bias. Dividing all future "Positive" probabilities by 0.7 (and "Negative" by 0.3) re-centers the predictions. This simple trick reduced accuracy variance across prompt formats from 40% to under 5%.

python
# Contextual calibration in code
# Step 1: Get bias probabilities
bias_prompt = "Classify: N/A\nSentiment:"
bias_probs = model.predict_probs(bias_prompt)
# e.g., {"Positive": 0.7, "Negative": 0.3}

# Step 2: Calibrate real predictions
real_prompt = "Classify: 'Great movie!'\nSentiment:"
raw_probs = model.predict_probs(real_prompt)
# e.g., {"Positive": 0.9, "Negative": 0.1}

calibrated = {k: raw_probs[k] / bias_probs[k]
              for k in raw_probs}
# Normalize so they sum to 1
total = sum(calibrated.values())
calibrated = {k: v / total for k, v in calibrated.items()}
# {"Positive": 0.81, "Negative": 0.19} — less biased

Chapter 3: Chain-of-Thought

In 2022, Jason Wei et al. at Google discovered something surprising: if you simply add the words "Let's think step by step" to a math prompt, GPT-3's accuracy on the GSM8K benchmark jumps from 18% to 57%. Include a few examples with worked-out reasoning chains, and it climbs to 79%. Same model. Same weights. Same test set. Only the prompt changed.

This is chain-of-thought (CoT) prompting — a technique where you ask the model to show its reasoning steps before giving the final answer. The key insight: language models process information sequentially, token by token. If a problem requires multiple reasoning steps, the model must perform all of them within a single forward pass — unless you give it "scratch space" in the form of generated intermediate tokens.

Why CoT Works

A standard prompt asks the model to go directly from question to answer:

prompt
Q: Roger has 5 tennis balls. He buys 2 more cans of
tennis balls. Each can has 3 tennis balls. How many
tennis balls does he have now?
A: 11

The model must compute 5 + (2 × 3) in a single "step." For simple arithmetic, this works. But for multi-step word problems, the required computation exceeds what a single forward pass can do reliably.

With CoT, the model generates intermediate steps. Each step becomes part of the context for the next step, effectively giving the model a working memory:

prompt
Q: Roger has 5 tennis balls. He buys 2 more cans of
tennis balls. Each can has 3 tennis balls. How many
tennis balls does he have now?
A: Roger started with 5 balls. He bought 2 cans with
3 balls each, so 2 × 3 = 6 new balls.
5 + 6 = 11. The answer is 11.

Each generated token becomes context for the next token. The model isn't doing harder computation — it's breaking one hard problem into many easy problems, solving them sequentially through autoregressive generation.

Zero-shot CoT

The most surprising finding: you don't even need hand-crafted reasoning examples. Kojima et al. (2022) showed that just appending "Let's think step by step" to any prompt triggers reasoning chains. This is zero-shot CoT — no examples, no manual chain-writing, just a magic phrase that activates latent reasoning capabilities.

On MultiArith (arithmetic word problems): Standard zero-shot = 18%. Zero-shot CoT = 79%. That's a 61-percentage-point improvement from five words.

When CoT Helps (and When It Doesn't)

CoT also scales with model size. In small models (<10B parameters), CoT often hurts performance — the model generates plausible-sounding but incorrect reasoning steps. The benefit emerges only in large models that have internalized enough world knowledge to reason accurately.

Variants of Chain-of-Thought

The original CoT idea spawned an entire family of prompting techniques:

All of these share the same underlying principle: give the model more "thinking tokens" between question and answer. The tokens can be natural language (CoT), code (PoT), or interleaved with tool calls (ReAct). The key is that each intermediate token creates a stepping stone the model can attend to when generating the next token.

Chapter 4: The Lottery Ticket

So far we've adapted models without changing any weights at all. But what if we want to actually train some weights — just not all 175 billion of them? How many weights do we really need?

In 2019, Jonathan Frankle and Michael Carlin published "The Lottery Ticket Hypothesis," making a striking claim: a dense neural network contains a sparse subnetwork that, when trained in isolation from the same initialization, matches the full network's accuracy. They called this subnetwork the winning ticket.

The Hypothesis

Think of training a neural network like buying lottery tickets. A dense network with 100 million weights is like buying 100 million tickets. Most of those tickets (weights) are losers — they don't meaningfully contribute to the final function the network computes. But somewhere in those 100 million, there's a winning subnetwork of maybe 10 million weights that does all the real work.

Frankle and Carlin proved this experimentally with a technique called iterative magnitude pruning:

The result: on CIFAR-10 with VGG-19, they could prune 90% of weights and still match the original accuracy. The winning ticket was there all along — you just needed to find it.

The iterative part is crucial. You can't just prune 90% of a randomly initialized network in one shot — you don't know which weights are important yet. The pruning must be guided by a full training run that reveals which weights naturally gravitate toward zero. The magnitude of a trained weight is a proxy for its importance: weights that end up large were consistently reinforced by gradients; weights that end up near zero were irrelevant to the loss.

python
# Simplified iterative magnitude pruning
import torch

def iterative_prune(model, train_fn, prune_pct=0.2, rounds=5):
    initial_weights = {n: p.clone() for n, p in model.named_parameters()}
    mask = {n: torch.ones_like(p) for n, p in model.named_parameters()}

    for r in range(rounds):
        train_fn(model, mask)  # train with current mask

        # Prune smallest-magnitude weights
        for n, p in model.named_parameters():
            alive = mask[n].bool()
            threshold = torch.quantile(
                p[alive].abs(), prune_pct
            )
            mask[n][p.abs() < threshold] = 0

        # Reset to original initialization
        for n, p in model.named_parameters():
            p.data = initial_weights[n] * mask[n]

    return mask  # the winning ticket

Why This Matters for Efficient Adaptation

The lottery ticket hypothesis tells us something profound about parameter efficiency: most parameters in a neural network are redundant. If 90% of weights can be removed without hurting accuracy, then maybe we don't need to update all of them during fine-tuning either.

This observation directly motivates adapter methods and LoRA. If the useful information lives in a low-dimensional subspace, we can fine-tune only within that subspace. The next two chapters show exactly how.

Aghajanyan et al. (2021): Intrinsic Dimensionality

This paper measured the intrinsic dimensionality of fine-tuning: the minimum number of parameters needed to achieve 90% of full fine-tuning accuracy. For a 280M-parameter RoBERTa model fine-tuned on MRPC (paraphrase detection), the intrinsic dimensionality was about 200. Not 200 million. Two hundred.

That's 0.00007% of the total parameters. The fine-tuning "update" lives in an extremely low-dimensional subspace. This is the theoretical foundation for LoRA.

Pruning at Scale

The original lottery ticket experiments used small networks on CIFAR-10. Does the hypothesis hold at scale? Subsequent work revealed a nuance: for large networks, you can't reset to the original initialization — you need to reset to weights from early in training (e.g., after 0.1% of total training). This is the late resetting variant, and it works robustly at scale.

The practical implication: large pre-trained language models are massively over-parameterized. Studies on BERT showed that 40-90% of attention heads can be pruned with negligible accuracy loss on downstream tasks. For GPT-style models, structured pruning of entire layers (not just weights) can remove 25-30% of layers while maintaining 95%+ of the original accuracy.

This over-parameterization is a feature, not a bug: it makes training easier (more paths to good solutions) and enables transfer learning (different tasks use different subsets of the weights). But at deployment time, most of those parameters are wasted compute.

Chapter 5: Adapters

Now we get to methods that actually modify the model — but surgically, with a scalpel rather than a sledgehammer. Adapters (Houlsby et al., 2019) insert small trainable modules between the frozen layers of a pre-trained transformer. The original weights don't change. Only the adapter weights are trained.

From Prompting to Training: The Leap

The previous three chapters used zero trainable parameters. Prompting, few-shot, and chain-of-thought all leave the model weights untouched. This is elegant but limited — you're constrained to whatever the model can do with just the right prompt. When you need the model to reliably produce specific output formats, handle domain-specific terminology, or maintain consistent quality across thousands of requests, you need to actually train some weights.

The question: which weights? Adapters answer this by training new weights inserted between existing layers, while leaving all original weights frozen.

Architecture

An adapter module is a simple bottleneck: down-project → nonlinearity → up-project, with a residual connection around the whole thing.

Concretely, for a hidden dimension of d = 1024 and a bottleneck dimension of m = 64:

Where W_down ∈ R^{d × m} (1024 × 64), W_up ∈ R^{m × d} (64 × 1024), and f is a nonlinearity (usually ReLU or GELU). The residual connection h + (...) ensures that if the adapter weights are initialized to near-zero, the module is approximately an identity function — the model starts where the pre-trained model left off.

Where Do Adapters Go?

Houlsby et al. placed two adapter modules in each transformer layer: one after the multi-head attention sublayer, one after the feed-forward sublayer. Both sit after the sublayer and before the residual connection to the layer norm.

With d = 1024 and m = 64, each adapter has 2 × 1024 × 64 = 131,072 parameters. Two adapters per layer × 24 layers = ~6.3M trainable parameters. For a 340M-parameter BERT-large, that's 1.8% of total parameters. For a 175B GPT-3, it would be roughly 0.004%.

Adapters in Code

python
class Adapter(nn.Module):
    def __init__(self, d_model, bottleneck):
        super().__init__()
        self.down = nn.Linear(d_model, bottleneck)
        self.up   = nn.Linear(bottleneck, d_model)
        self.act  = nn.GELU()
        # Initialize up-projection near zero
        # so adapter starts as identity
        nn.init.zeros_(self.up.weight)
        nn.init.zeros_(self.up.bias)

    def forward(self, x):
        # x: [batch, seq_len, d_model]
        return x + self.up(self.act(self.down(x)))

The key detail: W_up is initialized to zero, making the entire adapter output zero at initialization. The residual connection means the adapter starts as a pass-through — the model behaves identically to the original pre-trained model on the first forward pass. Training then "grows" the adapter's contribution gradually.

This zero-initialization is critical for training stability. If adapters were randomly initialized, the model's outputs would be completely disrupted on the first forward pass — the pre-trained model's learned representations would be corrupted by random adapter noise. Starting from zero means the model can never be worse than the pre-trained baseline, and gradients gently guide the adapter toward task-specific modifications.

Choosing the Bottleneck Size

The bottleneck dimension m controls the capacity vs. efficiency tradeoff:

The original paper found that m = 64 was sufficient for all 26 GLUE/SQuAD tasks. Reducing to m = 8 still worked for simpler classification tasks but degraded on complex QA. The choice should be guided by task complexity: simple classification needs less capacity than open-ended generation.

Multi-Task Deployment with Adapters

The killer feature of adapters is modular task composition. You store one frozen base model (say, 350 GB for GPT-3) and swap in tiny adapter checkpoints per task:

That's a 26x storage reduction. At serving time, you load the base model once into GPU memory and hot-swap adapter weights based on which task the incoming request requires. The swap takes milliseconds — just overwriting a few megabytes of weights.

The Inference Cost Problem

Adapters have one drawback: they add sequential computation during inference. Every forward pass must go through the adapter modules, adding latency proportional to 2 × L adapter forward passes (where L is the number of layers). For a model serving millions of requests per day, even a few milliseconds per adapter matters. The additional latency compounds: two adapters per layer × 96 layers = 192 extra sequential operations.

This latency issue motivated the development of LoRA, which adds zero inference cost by merging the learned parameters directly into the base weights.

Chapter 6: LoRA

In 2021, Edward Hu et al. at Microsoft published LoRA: Low-Rank Adaptation of Large Language Models — and it became the most widely used PEFT method within a year. The idea is elegant: weight updates during fine-tuning have low intrinsic rank. If the update matrix ΔW is low-rank, decompose it into two small matrices and train only those.

The Core Idea

Adapters work, but they add new sequential layers to the model. LoRA takes a fundamentally different approach: instead of adding new layers, it modifies existing ones. The key observation from Aghajanyan et al. is that fine-tuning updates are low-rank — the matrix ΔW = W_final - W_pretrained has far fewer independent dimensions than its size would suggest.

In standard fine-tuning, you update a weight matrix W₀ ∈ R^{d × k} to W₀ + ΔW. The update ΔW has the same shape as W₀ — for GPT-3's attention weights, that's 12288 × 12288 = 150 million parameters per matrix.

LoRA constrains ΔW to be low-rank by decomposing it:

Where B ∈ R^{d × r} and A ∈ R^{r × k}, and r << min(d, k). For r = 8 and d = k = 12288:

That's a 770x reduction in trainable parameters for r = 8. Across all attention matrices in GPT-3 (Q, K, V, O × 96 layers), LoRA with r = 8 trains about 4.7 million parameters instead of 175 billion — a 37,000x reduction.

How It Works at Inference

Here's the beautiful part: LoRA adds zero inference latency. During training, the forward pass computes:

After training, you merge: W_merged = W₀ + BA. Now you have a single weight matrix with no extra computation. The adapter is "baked in." To switch tasks, just swap the BA matrices and re-merge. This is why LoRA dominates: adapters add latency at every forward pass, but LoRA adds none.

Initialization

A is initialized with random Gaussian values, B is initialized to zero. This means ΔW = BA = 0 at the start — the model begins as the original pre-trained model, identical to adapter initialization logic.

LoRA also uses a scaling factor α/r to control the magnitude of the update:

Typically α = r, so the scaling is 1. When r increases, each individual rank-1 component contributes proportionally less, which stabilizes training.

LoRA in Code

python
class LoRALinear(nn.Module):
    def __init__(self, linear, r=8, alpha=8):
        super().__init__()
        self.linear = linear             # frozen original layer
        d, k = linear.weight.shape
        self.A = nn.Parameter(torch.randn(r, k) * 0.01)
        self.B = nn.Parameter(torch.zeros(d, r))
        self.scale = alpha / r

    def forward(self, x):
        # x: [batch, seq_len, k]
        base = self.linear(x)            # W_0 @ x
        lora = (x @ self.A.T) @ self.B.T # (BA)x
        return base + self.scale * lora

    def merge(self):
        # Bake LoRA into W_0 for zero-cost inference
        self.linear.weight.data += self.scale * (self.B @ self.A)

LoRA Training Loop

The training loop for LoRA is identical to standard fine-tuning, except only LoRA parameters receive gradients:

python
# Freeze base model, train only LoRA parameters
for name, param in model.named_parameters():
    if 'lora' not in name:
        param.requires_grad = False  # freeze base

optimizer = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad],
    lr=2e-4  # higher LR than full FT (typically 5e-5)
)

for batch in dataloader:
    loss = model(batch["input_ids"], labels=batch["labels"]).loss
    loss.backward()   # gradients only flow to LoRA params
    optimizer.step()
    optimizer.zero_grad()

# After training: merge for zero-cost inference
for module in model.modules():
    if hasattr(module, 'merge'):
        module.merge()

Notice the learning rate: 2e-4 vs. the typical 5e-5 for full fine-tuning. With far fewer parameters, each parameter needs to move more per step to achieve the same total update magnitude.

Which Weights to Adapt?

The original paper found that applying LoRA to the query and value projections (W_Q and W_V) in attention works best. Adapting all four attention matrices (Q, K, V, O) gives marginal improvement. The feed-forward layers are less important — attention is where task-specific adaptation happens.

Choosing the Right Rank

Rank r is the single most important hyperparameter in LoRA. The original paper tested r from 1 to 64 on GPT-3 for various tasks. Key findings:

A practical heuristic: start with r = 8. If accuracy is insufficient, double to r = 16. If r = 16 isn't enough, the task may require full fine-tuning or a fundamentally different approach. Going above r = 64 rarely helps — you're past the intrinsic dimensionality of most fine-tuning updates.

The scaling factor α is typically set equal to r, giving a scale of 1.0. Some practitioners use α = 2r for a stronger initial learning signal. The learning rate for LoRA parameters is usually 10-100x higher than for full fine-tuning, since you're training far fewer parameters.

LoRA vs. Full Fine-Tuning: When Does It Fall Short?

LoRA's low-rank constraint is a double-edged sword. For most downstream NLP tasks (classification, summarization, translation), the update is indeed low-rank and LoRA matches full fine-tuning. But there are cases where it struggles:

Dramatically new domains. If the task requires knowledge fundamentally absent from pre-training (e.g., a new programming language invented after the cutoff), a rank-8 update may not have enough capacity to encode the new patterns. Increasing rank to 32-64 or using full fine-tuning may be necessary.

Very long-context tasks. Adapting the model to handle much longer sequences than it was pre-trained on often requires changes that span many dimensions of the weight space — not a low-rank update.

Multi-modal extensions. Adding a new modality (vision, audio) requires updating embedding layers and cross-attention in ways that exceed low-rank decomposition. LoRA is best for task adaptation within the model's existing modality.

The rule of thumb: if you're steering the model's existing capabilities toward a specific task, LoRA works beautifully. If you're trying to teach the model fundamentally new capabilities, you may need higher rank or full fine-tuning.

Chapter 7: PEFT Playground

You've learned four ways to adapt a model: in-context learning, prompt tuning, adapters, and LoRA. But how do they compare head-to-head? This playground lets you train a simulated model with each method and watch the differences in real time.

Select a method, then click "Train" to start a simulated training run. Watch which weights update (colored vs. gray), and track four metrics: trainable parameters, GPU memory, accuracy, and training time. Try all four methods to see the tradeoffs.

Pay particular attention to the loss curve (bottom right): all methods converge, but at different speeds and to different final accuracies. Full fine-tuning converges slowest but to the best accuracy. Prompt tuning converges fastest but to a lower ceiling. LoRA and adapters sit in between — near full fine-tuning accuracy with prompt tuning speed.

The numbers tell the story:

Several patterns emerge:

Full fine-tuning wins on accuracy when you have enough data and compute. It can modify any weight to fit any pattern. But the cost is enormous — 2 TB of memory, weeks of training, and a full copy of the model per task.

LoRA is the sweet spot for most practitioners. Near-full-fine-tuning accuracy with 0.003% trainable parameters and zero extra inference cost. The trained LoRA matrices merge into the base weights, so deployment is identical to deploying the original model.

Adapters match LoRA on accuracy but add inference latency because the adapter modules remain as separate sequential computations during inference. If latency matters, LoRA wins.

Prompt tuning is the lightest touch. Only a few thousand parameters, trained in minutes. But it has the lowest accuracy ceiling and can't fix fundamental capabilities the model lacks. Best for tasks the model already almost knows how to do.

Memory Breakdown: Why PEFT Saves So Much

The memory savings aren't just about having fewer parameters. During training with Adam, every trainable parameter requires storing:

For a 175B parameter model with full fine-tuning, that's 2 + 2 + 4 + 4 = 12 bytes per parameter = 2.1 TB total. With LoRA (4.7M trainable params), only those 4.7M parameters need optimizer states: 4.7M × 12 bytes = 56 MB. The frozen 175B parameters are stored in fp16 (350 GB) but need no gradients or optimizer states.

This is why LoRA and adapters are called "parameter-efficient" — the savings cascade through the entire training pipeline: fewer trainable params → fewer gradients → fewer optimizer states → less memory → fewer GPUs → lower cost.

Activation Memory

One subtlety: even with PEFT, you still need to store activations for the backward pass. Activations scale with batch size × sequence length × hidden dimension × number of layers. For a 175B model with batch size 1 and sequence length 2048, activations alone can consume 200+ GB. This is why techniques like gradient checkpointing (recomputing activations instead of storing them) are essential even with LoRA.

Notice that activations are the same size regardless of PEFT method — because the forward pass through the frozen model is identical. PEFT saves on weights, gradients, and optimizer states, but not on activations. This is why you still need significant GPU memory even with LoRA.

Chapter 8: When to Use What

The choice of adaptation method isn't just about accuracy — it's about the intersection of your data, compute, deployment constraints, and how many tasks you need to serve. This chapter gives you a decision framework.

The Decision Tree

The interactive tree below walks you through the key questions. Click each node to explore a decision path. Your answers determine which method is recommended.

Detailed Decision Matrix

QLoRA: The Practitioner's Default

In practice, most people use QLoRA (Dettmers et al., 2023), which combines 4-bit quantization of the base model with LoRA adapters. This lets you fine-tune a 65B parameter model on a single 48 GB GPU — something that would normally require 8+ GPUs.

The trick: quantize the base model weights to 4-bit NormalFloat (NF4) format, reducing memory by 4x. Then add LoRA adapters in fp16 or bf16 on top. The LoRA gradients are computed in 16-bit, so training precision is maintained. Only the frozen base weights are quantized.

python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load 65B model in 4-bit (~16 GB)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    quantization_config=bnb_config,
)

# Add LoRA adapters (~18M trainable params)
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 18,874,368 || all params: 68,977,307,648
# trainable%: 0.0274

With QLoRA, you can fine-tune a 70B model on a single A100 (80 GB) or even a consumer RTX 4090 (24 GB, with careful gradient checkpointing). This democratized fine-tuning: what previously required a compute cluster now runs on a single GPU.

Deployment Considerations

The choice of PEFT method affects not just training but deployment architecture:

A common production pattern: fine-tune with QLoRA on your data, merge the adapter, quantize the merged model to 4-bit, and serve with vLLM or TGI. Total workflow from raw data to deployed model can be done in a single afternoon on one GPU.

The End-to-End QLoRA Workflow

This workflow has become the de facto standard for fine-tuning open-source models. Tools like Hugging Face's PEFT library, Axolotl, and LLaMA-Factory wrap all five steps into a single configuration file.

Chapter 9: Connections

Efficient adaptation is the bridge between pre-training and deployment. Pre-training gives the model broad knowledge; PEFT methods let you specialize that knowledge for your task without the cost of a full fine-tune. The field is evolving rapidly — LoRA itself has spawned dozens of variants (DoRA, AdaLoRA, QLoRA, LoRA+) in just three years.

The Adaptation Spectrum

Key Papers

Related Lessons

Method Summary

The Frontier

LoRA variants. DoRA (Weight-Decomposed Low-Rank Adaptation) separates magnitude and direction of weight updates, closing the gap to full FT. AdaLoRA allocates rank adaptively per layer based on importance scores. LoRA+ uses different learning rates for A and B matrices, improving convergence speed by 2x.

Mixture of LoRA experts. Instead of one LoRA adapter per task, train multiple small LoRA "experts" and route inputs to the right expert at inference. This combines multi-task serving with LoRA's efficiency. Each expert specializes in a subtask (code, math, creative writing), and a lightweight router selects the expert based on the input.

Merging without retraining. Model soups and TIES-Merging combine multiple LoRA adapters into a single set of weights — a multi-task model without multi-task training. TIES-Merging resolves sign conflicts between adapters, which naive averaging misses.

1-bit and sub-4-bit fine-tuning. As quantization methods improve (GPTQ, AWQ, GGML), the base model gets smaller and LoRA adapters become a proportionally larger fraction of the total. Eventually the adapter might be bigger than the quantized base.

Scaling LoRA. As models grow past 1 trillion parameters, even LoRA's 0.1% becomes millions of parameters. Research is exploring structured pruning of LoRA matrices, tying LoRA weights across layers, and learning which layers even need adaptation (many don't).

LoRA for non-language modalities. LoRA has proven effective far beyond NLP. It's now the standard approach for fine-tuning Stable Diffusion (LoRA for image generation), Whisper (LoRA for speech recognition), and multimodal models like LLaVA (LoRA for vision-language tasks). The low-rank hypothesis holds across modalities.

Historical Timeline

The field of efficient adaptation has transformed how practitioners deploy large language models. What once required million-dollar compute budgets and dedicated infrastructure teams can now be done on a single GPU in an afternoon. The key insight — that fine-tuning updates live in a low-dimensional subspace — is both theoretically elegant and practically revolutionary.

As models continue to grow (GPT-4 is estimated at 1.8T parameters, and future models will be larger), efficient adaptation becomes not just convenient but mandatory. Full fine-tuning of a 10T parameter model is beyond the reach of any organization. LoRA and its successors will be the only practical path to customization.