Adapter Modules for PEFT

Chapter 1: Adapter Architecture

An adapter module is a small neural network inserted into each Transformer layer. Its design is brilliantly simple: a bottleneck — project down to a small dimension, apply a nonlinearity, project back up — wrapped in a residual connection.

The residual connection is crucial. If the adapter is initialized near zero (so W_up · ReLU(W_down · h) ≈ 0), the output is just h — the identity function. This means at initialization, the adapter does nothing, and the model behaves exactly like the pre-trained model. The adapter only deviates from identity as needed during training.

python
import torch.nn as nn

class Adapter(nn.Module):
    def __init__(self, d_model=768, bottleneck=64):
        super().__init__()
        self.down = nn.Linear(d_model, bottleneck)  # 768 → 64
        self.relu = nn.ReLU()
        self.up   = nn.Linear(bottleneck, d_model)  # 64 → 768
        # Initialize up-projection near zero
        nn.init.zeros_(self.up.weight)
        nn.init.zeros_(self.up.bias)

    def forward(self, h):
        # h: [batch, seq_len, 768]
        return h + self.up(self.relu(self.down(h)))
        # Output: [batch, seq_len, 768] — same shape

# Parameter count per adapter:
# down: 768 × 64 + 64 = 49,216
# up:   64 × 768 + 768 = 49,920
# Total: 99,136 per adapter ≈ 99K

Chapter 2: Bottleneck Design

The bottleneck dimension m is the adapter's primary hyperparameter. It controls the trade-off between parameter efficiency and task-specific capacity.

Parameter count analysis

Each adapter has 2 × d × m + d + m parameters (weights + biases). With two adapters per layer (after attention and after FFN) and L layers:

python
# Adapter parameter counts for BERT-Base (d=768, L=12)
d, L = 768, 12
for m in [8, 16, 32, 64, 128, 256]:
    params = 2 * L * (2 * d * m + d + m)
    pct = params / 110e6 * 100
    print(f"m={m:>3d}: {params:>10,d} params ({pct:.2f}%)")
# m=  8:      313,344 params (0.28%)
# m= 16:      608,256 params (0.55%)
# m= 32:    1,198,080 params (1.09%)
# m= 64:    2,377,728 params (2.16%)
# m=128:    4,737,024 params (4.31%)
# m=256:    9,455,616 params (8.60%)

Houlsby et al. found that m = 64 provides the best accuracy-efficiency trade-off for BERT-Base. At m = 64, adapters add only 2.16% extra parameters while achieving 98.4% of full fine-tuning performance on the GLUE benchmark.

Information bottleneck perspective

The adapter bottleneck implements a form of the information bottleneck principle: compress the input (down-projection), keep only task-relevant information (the nonlinearity selects which dimensions matter), then expand back (up-projection). The bottleneck acts as a filter, discarding pre-trained features that aren't useful for the target task while amplifying useful ones.

Chapter 3: Where to Insert

Where in the Transformer should adapters go? Houlsby et al. tested two configurations:

Configuration 1: Two adapters per layer (recommended)

Configuration 2: One adapter per layer (more efficient)

Place only one adapter after the FFN sub-layer. This halves the adapter parameters with a small accuracy drop (~0.3 GLUE points). Later papers (AdapterFusion, MAD-X) adopted this single-adapter configuration as the default.

LayerNorm parameters

In addition to the adapter modules, Houlsby et al. also train the LayerNorm parameters (scale γ and bias β) in each layer. LayerNorms have very few parameters (2 × d per layer = 1,536 for BERT-Base), but they control the scale and shift of representations at each layer, providing a coarse but powerful adaptation signal.

python
# Complete training setup for adapter-based PEFT
def setup_adapter_training(model, bottleneck=64):
    # 1. Insert adapters into each Transformer layer
    for layer in model.encoder.layers:
        layer.adapter_attn = Adapter(d_model=768, bottleneck=bottleneck)
        layer.adapter_ffn  = Adapter(d_model=768, bottleneck=bottleneck)

    # 2. Freeze all pre-trained parameters
    for p in model.parameters():
        p.requires_grad = False

    # 3. Unfreeze adapters and LayerNorms
    for name, p in model.named_parameters():
        if 'adapter' in name or 'LayerNorm' in name:
            p.requires_grad = True

    # 4. Add task-specific classification head (also trained)
    model.classifier = nn.Linear(768, num_classes)

    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"Trainable: {trainable:,d} / {sum(p.numel() for p in model.parameters()):,d}")
    # Trainable: 2,414,848 / 112,414,848 (2.15%)

Chapter 4: Training Procedure

Training adapters is almost identical to standard fine-tuning, with one key difference: most parameters are frozen.

What's trained vs frozen

Training details

Houlsby et al. used the same hyperparameters as standard BERT fine-tuning:

1. Optimizer: Adam with learning rate 3e-4 (same as fine-tuning BERT).

2. Epochs: 3-10 depending on dataset size (same as fine-tuning).

3. Batch size: 32 (same as fine-tuning).

4. No special adapter-specific hyperparameters needed beyond the bottleneck dimension m.

The training speed is slightly faster than full fine-tuning because fewer parameters need gradients, but the difference isn't dramatic — the forward pass through the full model still dominates compute time.

Gradient flow through frozen layers

During backpropagation, gradients flow through the frozen layers (they need to, to compute gradients for the adapters). The frozen parameters aren't updated, but their activations and gradients are computed normally. This means the adapter at layer L receives gradient information from all subsequent layers — it can learn from the entire network's behavior, not just its local context.

Chapter 5: Results & Analysis

Houlsby et al. evaluated adapters on the GLUE benchmark (8 NLU tasks) and SQuAD (question answering), comparing against full fine-tuning and feature extraction.

GLUE results

Adapters achieve within 0.7 GLUE points of full fine-tuning while training only 2.2% of the parameters. That's a 46x reduction in trainable parameters for a 0.8% accuracy trade-off.

Which layers matter most?

Houlsby et al. found that adapters in higher layers (closer to the output) contribute more to task-specific performance than adapters in lower layers. This makes sense: lower layers learn general linguistic features (syntax, grammar) that transfer across tasks, while upper layers learn more task-specific features that need adaptation.

However, removing ALL lower-layer adapters hurts performance. Even general features benefit from slight task-specific adjustments.

Comparison with other efficient methods

At the time of publication, the main alternative to full fine-tuning was training only specific layers:

Adapters outperform training specific layers despite having fewer parameters. The distributed nature of adapters (one per layer) is key — they can make fine-grained adjustments throughout the network rather than large adjustments in a few layers.

Chapter 6: Adapter Explorer

Experiment with the adapter architecture yourself. This explorer lets you configure the bottleneck size, the number of adapters, and see the resulting parameter counts and accuracy estimates.

Chapter 7: Connections

Adapter modules launched the field of Parameter-Efficient Fine-Tuning (PEFT). Understanding their connections shows how the field evolved.

What came before

What came after

The PEFT family tree

All PEFT methods share the core insight from this paper: task-specific adaptation is low-dimensional. The methods differ in WHERE they inject the adaptation (new modules vs weight perturbations vs input perturbations) and HOW they parameterize it (bottleneck vs low-rank vs continuous prompts).

"The art of being wise is the art of knowing what to overlook." — William James (adapters know which parameters to overlook)