BERT: Bidirectional Transformers

Chapter 1: Masked Language Model (MLM)

BERT's primary pre-training objective is the Masked Language Model (MLM). The idea is deceptively simple: randomly mask some percentage of the input tokens, then train the model to predict the original tokens from the corrupted input.

Specifically, for each training sequence, BERT randomly selects 15% of the token positions for prediction. But there's a subtle twist in how these selected tokens are treated:

Why this 80/10/10 split? It addresses a critical problem: the mismatch between pre-training and fine-tuning.

During fine-tuning, the model never sees the [MASK] token — real inputs don't have masks. If the model learned to rely on the presence of [MASK] as a signal ("oh, there's a mask, I should predict something"), that skill wouldn't transfer to fine-tuning. The 10% random replacement forces the model to stay uncertain about which tokens are corrupted, so it must maintain a good representation of every token. The 10% unchanged tokens ensure the model learns that even "correct-looking" tokens may need prediction.

The MLM loss is simply the cross-entropy loss between the model's predicted distribution and the true token, summed only over the masked positions:

Where x_i is the true token at position i, x_\masked is the input with masked tokens replaced, and p(x_i | x_\masked) is the model's predicted probability for the correct token at position i.

Let's walk through a concrete example. Take the sentence "The cat sat on the mat" with 6 tokens. With 15% masking, we'd mask about 1 token (0.15 × 6 ≈ 1). Say we select "cat" (position 2):

python
# MLM: one training step
import torch
import torch.nn.functional as F

# Original tokens: [CLS] The cat sat on the mat [SEP]
input_ids = [101, 1996, 4937, 2938, 2006, 1996, 13523, 102]

# Masking "cat" (position 2) — with 80% prob, replace with [MASK]=103
masked_input = [101, 1996, 103, 2938, 2006, 1996, 13523, 102]
#                                ^^^^ [MASK] replaces "cat"

# labels: -100 means "don't compute loss here"
labels = [-100, -100, 4937, -100, -100, -100, -100, -100]
#                  ^^^^ only compute loss at masked position

# Forward pass: model outputs logits [8, vocab_size=30522]
logits = model(torch.tensor([masked_input]))  # [1, 8, 30522]

# Loss at position 2: cross-entropy between logits[2] and true id 4937
loss = F.cross_entropy(
    logits[0, 2, :],           # predicted distribution over 30522 vocab
    torch.tensor(4937)        # true token id for "cat"
)
# loss ≈ 5.2 (early training) → ≈ 0.3 (late training)

Notice that the model receives the full bidirectional context ("The [MASK] sat on the mat") and must predict "cat" from that context. It sees "sat" and "on the mat" to the right, and "The" to the left. Both directions contribute to the prediction. This is exactly the deep bidirectional conditioning that GPT-style models can't do.

Connection to denoising autoencoders

MLM is, mathematically, a denoising autoencoder. The input is corrupted (masking), passed through the model, and the model must reconstruct the original. This connection to the autoencoder literature (Vincent et al., 2008) was not an accident — Devlin et al. explicitly drew on this tradition. The key difference is that BERT operates on discrete tokens (not continuous pixels) and uses cross-entropy loss (not reconstruction error).

The denoising perspective also explains why MLM works so well as a pre-training objective: by learning to reconstruct corrupted text, the model must build rich internal representations that capture syntax, semantics, and world knowledge. A model that can correctly predict "[MASK] sat on the mat" → "cat" must know that cats sit on mats, that "sat" is past tense, that articles precede nouns — all learned implicitly from the reconstruction task.

Computational cost of 15% masking

One downside of MLM compared to standard language modeling: only 15% of tokens contribute to the training loss per step. In a left-to-right language model like GPT, every token contributes — the model predicts the next token at every position. This means MLM needs roughly 6-7x more training steps to see the same number of prediction tasks. BERT compensates with a larger batch size (256 sequences) and longer training (1M steps), but this inefficiency is a real cost.

Chapter 2: Next Sentence Prediction (NSP)

Many important NLP tasks — question answering, natural language inference, paraphrase detection — require understanding the relationship between two sentences, not just individual sentence meaning. MLM trains the model to understand tokens in context, but it doesn't explicitly teach the model about inter-sentence relationships.

BERT's second pre-training objective addresses this: Next Sentence Prediction (NSP). The idea is straightforward. Given two sentences A and B, the model must predict whether B actually follows A in the original text (label: IsNext) or whether B is a random sentence from the corpus (label: NotNext).

The input format packs both sentences into a single sequence with special tokens:

[CLS] is a special classification token prepended to every input. Its final hidden state is used as the "aggregate sequence representation" for classification tasks. [SEP] is a separator token that marks the boundary between sentences.

The NSP prediction uses the final hidden state of the [CLS] token, fed through a linear layer with softmax:

Where h_[CLS] is the d-dimensional hidden state of the [CLS] token at the final layer, and W is a 2 × d weight matrix. The loss is binary cross-entropy.

python
# NSP training example
# Positive pair (IsNext):
sent_a = "The cat sat on the mat."
sent_b = "It purred softly in the sunlight."
label  = 1  # IsNext

# Negative pair (NotNext):
sent_a = "The cat sat on the mat."
sent_b = "Quantum entanglement enables teleportation."
label  = 0  # NotNext

# Input to BERT: [CLS] sent_a [SEP] sent_b [SEP]
# Token type IDs:  0    0...0   0    1...1   1
# The type IDs tell BERT which tokens belong to sentence A vs B

# Forward pass
cls_hidden = model(input_ids, token_type_ids)[0][:, 0, :]  # [batch, 768]
nsp_logits = nsp_head(cls_hidden)  # [batch, 2]
nsp_loss   = F.cross_entropy(nsp_logits, labels)

The NSP controversy

NSP was perhaps the most debated design choice in BERT. Later papers showed it may actually hurt performance:

The consensus that emerged: NSP is too easy. When the negative pairs are random sentences from different documents, they're so obviously unrelated that the model can solve NSP by detecting topic similarity alone, without learning genuine discourse coherence. SOP (same document, swapped order) forces the model to learn actual sentence ordering, which is a harder and more useful skill.

Total pre-training loss

BERT's total pre-training loss is simply the sum of the MLM and NSP losses:

Both losses are computed on every training example. The MLM loss provides the rich token-level understanding signal, while NSP provides a weaker sentence-level signal. In practice, MLM does most of the heavy lifting — which is why removing NSP doesn't hurt much.

Chapter 3: Pre-training Data

BERT's representations are only as good as the data they're learned from. Devlin et al. used two unlabeled text corpora totaling approximately 3.3 billion words:

By 2018 standards, 3.3 billion words was substantial but not extreme. For context, GPT-2 (released months later) trained on 8 billion words, and GPT-3 trained on 300 billion tokens. BERT showed that even with relatively modest data, the right training objective (MLM + bidirectionality) could produce remarkable results.

Input format and preprocessing

Each training input is a pair of "sentences" (actually text spans) sampled from the corpus. The preprocessing pipeline works as follows:

The maximum sequence length is 512 tokens (including [CLS] and [SEP] tokens). This was a GPU memory constraint in 2018 — longer sequences require quadratically more memory for the attention matrix (512² = 262K entries per head per layer). Modern models have pushed this to 2K, 4K, 8K, and beyond using techniques like FlashAttention and sparse attention.

Training efficiency trick

BERT uses a clever training efficiency trick: for the first 90% of training steps, the maximum sequence length is reduced to 128 tokens. Only the final 10% of training uses the full 512-token length. This works because:

1. Short sequences are much cheaper: 128² / 512² = 1/16 the attention cost.

2. Most of language understanding can be learned from local context (nearby words).

3. The final 10% at 512 tokens teaches long-range dependencies without paying full cost for the entire training run.

This trick alone reduced total training time by approximately 40% with negligible performance loss.

Training hyperparameters

Chapter 4: Architecture

BERT uses the Transformer encoder — specifically, just the encoder half of the original Transformer (Vaswani et al., 2017). No decoder, no cross-attention, no autoregressive masking. Pure bidirectional self-attention, stacked into deep layers.

Two model sizes were released:

BERT-Base was deliberately sized to match GPT-1 (also 12 layers, 768 hidden, 110M parameters) to enable a fair comparison: same capacity, different training objective (bidirectional MLM vs unidirectional LM). BERT-Base outperformed GPT-1 on every benchmark, demonstrating that the bidirectional objective was the key ingredient, not model size.

Each Transformer encoder layer

Every layer in BERT follows the same structure:

The residual connections and layer normalization follow the original Transformer design. Each sub-layer (attention and FFN) has a residual connection around it, followed by layer normalization:

This is the "post-norm" pattern from the original Transformer. (Later models like GPT-2 switched to "pre-norm": output = x + SubLayer(LayerNorm(x)), which is more stable for deep networks.)

BERT's input representation

BERT's input is constructed from three embedding types, summed together:

Unlike the original Transformer which used fixed sinusoidal position encodings, BERT uses learned position embeddings. Each of the 512 possible positions has its own learned 768-dimensional vector. This means BERT cannot handle sequences longer than 512 tokens — there's no position embedding for position 513. Later models (RoPE, ALiBi) solved this with relative or extrapolatable position encodings.

GELU activation

BERT was one of the first major models to use the GELU (Gaussian Error Linear Unit) activation function instead of ReLU in the FFN layers:

Where Φ(x) is the standard Gaussian CDF. GELU is smooth (no sharp kink at 0 like ReLU) and stochastically gates the input based on its magnitude. Larger positive values pass through almost unchanged; negative values are suppressed but not completely zeroed (unlike ReLU). This smoother behavior is believed to help optimization in deep Transformer networks.

python
import torch
import torch.nn as nn

class BertLayer(nn.Module):
    def __init__(self, H=768, A=12, intermediate=3072):
        super().__init__()
        # Multi-head self-attention
        self.attn = nn.MultiheadAttention(H, A, batch_first=True)
        self.ln1 = nn.LayerNorm(H)
        # Feed-forward network
        self.ffn = nn.Sequential(
            nn.Linear(H, intermediate),   # 768 → 3072
            nn.GELU(),                     # smooth activation
            nn.Linear(intermediate, H),   # 3072 → 768
        )
        self.ln2 = nn.LayerNorm(H)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x):
        # x: [batch, seq_len, 768]
        attn_out, _ = self.attn(x, x, x)   # self-attention
        x = self.ln1(x + self.dropout(attn_out))  # residual + norm
        ffn_out = self.ffn(x)
        x = self.ln2(x + self.dropout(ffn_out))  # residual + norm
        return x  # [batch, seq_len, 768]

Chapter 5: WordPiece Tokenization

How do you represent text as numbers for a neural network? The naive approach — one token per word — fails for two reasons. First, the vocabulary would be enormous (English has ~170,000 words in common use, plus names, technical terms, and neologisms). Second, any word not in the vocabulary (an "out-of-vocabulary" or OOV word) can't be processed at all.

BERT uses WordPiece tokenization (Schuster & Nakajima, 2012), a subword method that strikes a balance between character-level and word-level tokenization. The key idea: common words stay as single tokens, but rare words are split into smaller subword pieces.

The "##" prefix indicates a continuation piece — a subword that's not at the start of a word. "play" is a word-initial piece; "##ing" is a continuation. This lets the model distinguish between "playing" (["play", "##ing"]) and "play" + "ing" as separate words.

How WordPiece vocabulary is built

The WordPiece vocabulary is constructed through an iterative greedy algorithm similar to Byte Pair Encoding (BPE):

The key difference between WordPiece and BPE is the scoring function. BPE simply merges the most frequent pair. WordPiece merges the pair with the highest mutual information — the pair where combining them provides the most information beyond what each piece provides alone. This tends to produce linguistically more meaningful subwords.

BERT's vocabulary of 30,522 tokens covers English text efficiently. The average English word is split into 1.1-1.5 WordPiece tokens, so the effective sequence length (in words) is roughly 340-465 words for a 512-token input. This is plenty for most NLP tasks — GLUE benchmarks have average input lengths of 20-60 words.

Special tokens

Chapter 6: Fine-Tuning

This is where BERT's design pays off spectacularly. After pre-training on 3.3 billion words of unlabeled text, BERT can be fine-tuned for virtually any NLP task with minimal architectural changes — often just a single output layer on top of the pre-trained model.

The fine-tuning recipe is always the same:

The key insight: all parameters are fine-tuned, not just the task head. The pre-trained weights shift slightly to adapt to the task, but they retain the vast majority of their pre-trained knowledge. This is why fine-tuning works with so little data — the model doesn't need to learn English from scratch; it just needs to learn the task-specific mapping.

Task-specific architectures

BERT handles four major task types with minimal modifications:

1. Sentence classification (SST-2, CoLA)

Use the [CLS] token's final hidden state as the sentence representation. Add a linear layer: h_[CLS] → logits.

2. Sentence pair classification (MNLI, QQP, RTE)

Input: [CLS] sentence A [SEP] sentence B [SEP]. Same as above — use h_[CLS] for classification.

3. Question answering (SQuAD)

Input: [CLS] question [SEP] passage [SEP]. For each passage token, predict two things: "is this the start of the answer?" and "is this the end of the answer?"

The answer span is the highest-scoring (start, end) pair where start ≤ end.

4. Named entity recognition (CoNLL-2003)

For each token, predict its entity tag (Person, Organization, Location, or None). Use each token's final hidden state with a linear classifier.

python
# Fine-tuning BERT for sentence classification
from transformers import BertModel
import torch.nn as nn

class BertClassifier(nn.Module):
    def __init__(self, num_classes=2):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        self.classifier = nn.Linear(768, num_classes)
        self.dropout = nn.Dropout(0.1)

    def forward(self, input_ids, attention_mask, token_type_ids):
        outputs = self.bert(input_ids, attention_mask, token_type_ids)
        cls_output = outputs.last_hidden_state[:, 0, :]  # [batch, 768]
        cls_output = self.dropout(cls_output)
        logits = self.classifier(cls_output)  # [batch, num_classes]
        return logits

# Fine-tuning hyperparameters (from the paper)
# Learning rate: 2e-5, 3e-5, or 5e-5
# Batch size: 16 or 32
# Epochs: 2, 3, or 4
# That's it — BERT pre-training did the hard work

Fine-tuning results

Chapter 7: Ablation Studies

One of BERT's greatest contributions was its thorough ablation study. Instead of just presenting results, Devlin et al. systematically removed components to understand what actually matters. These ablations are a masterclass in scientific rigor for ML papers.

Effect of pre-training tasks

What happens when you remove each pre-training objective?

The critical comparison is BERT vs LTR (left-to-right). Replacing MLM with standard left-to-right language modeling — keeping everything else the same — drops performance dramatically, especially on tasks requiring token-level predictions (SQuAD: 86.5 → 77.8). This is the smoking gun: bidirectionality is the key ingredient.

Adding a BiLSTM on top of the LTR model helps on SQuAD (77.8 → 84.8) but hurts on MRPC (77.5 → 75.7) and doesn't help on MNLI. Shallow bidirectionality (BiLSTM) is a poor substitute for deep bidirectionality (MLM + Transformer encoder).

Effect of model size

Does bigger always mean better? Devlin et al. trained models at multiple sizes:

Clear scaling: more layers and wider hidden dimensions help across all tasks. The jump from Base to Large is substantial (MNLI: 84.4 → 86.6, SQuAD: 86.5 → 90.9). This was early evidence of what became the scaling laws paper (Kaplan et al., 2020): bigger models systematically perform better.

Effect of number of training steps

BERT was trained for 1M steps. What if you train for less?

Devlin et al. showed that BERT's MLM pre-training needs significantly more steps than LTR to converge. This makes sense: MLM only gets a training signal from 15% of tokens per step (the masked ones), while LTR gets a signal from 100% of tokens (predict next at every position). MLM is 6-7x less sample-efficient per step. But at convergence, MLM produces much better representations — the quality of the training signal matters more than the quantity.

Feature extraction vs fine-tuning

Must you fine-tune all of BERT, or can you freeze it and just use the representations? Devlin et al. tested using BERT as a fixed feature extractor for NER:

Surprisingly, feature extraction comes close to fine-tuning (96.1 vs 96.4). This means BERT's pre-trained representations are already excellent for NER — fine-tuning provides only marginal improvement. For tasks where fine-tuning is expensive or impossible (e.g., you want to use BERT features in a pipeline with other non-differentiable components), feature extraction is a viable alternative.

Chapter 8: BERT Explorer

Now let's bring everything together. This interactive simulation lets you explore BERT's full pipeline — from raw text through tokenization, embedding, multi-layer encoding, masking, and prediction. You'll see exactly how information flows through the network and how bidirectional context shapes each token's representation.

What each layer learns

Research into BERT's internal representations (Tenney et al. 2019, "BERT Rediscovers the Classical NLP Pipeline") revealed a remarkable finding: BERT's layers form an implicit processing pipeline that mirrors the traditional NLP stack:

This is striking because nobody taught BERT about POS tags, parse trees, or semantic roles. These representations emerge purely from the MLM objective — the model discovered that building these intermediate representations helps it predict masked words. It's a form of unsupervised feature learning where the features that emerge happen to align with what linguists identified manually over decades.

Attention head specialization

Individual attention heads in BERT learn specialized roles (Clark et al. 2019). Some notable patterns:

Contextualization across layers

BERT's representations become increasingly contextualized as you go deeper. In the first layer, the representation of "bank" is nearly identical regardless of context. By layer 12, the representation of "bank" in "river bank" is very different from "bank" in "bank account." This progressive contextualization is what makes BERT's representations so powerful — they capture not just what a word is but what it means in this specific context.

python
# Probing BERT's layers: measuring contextual similarity
from transformers import BertModel, BertTokenizer
import torch

model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sent1 = "I went to the bank to deposit money"
sent2 = "I sat on the bank of the river"

# Get all layer outputs
tok1 = tokenizer(sent1, return_tensors='pt')
tok2 = tokenizer(sent2, return_tensors='pt')
out1 = model(**tok1).hidden_states  # tuple of 13 tensors (emb + 12 layers)
out2 = model(**tok2).hidden_states

# "bank" is token index 5 in both
for layer in [0, 3, 6, 9, 12]:
    v1 = out1[layer][0, 5]  # "bank" in sentence 1
    v2 = out2[layer][0, 5]  # "bank" in sentence 2
    sim = torch.nn.functional.cosine_similarity(v1, v2, dim=0)
    print(f"Layer {layer}: cosine similarity = {sim:.3f}")
# Layer 0:  0.95  (nearly identical — not yet contextualized)
# Layer 3:  0.85  (starting to diverge)
# Layer 6:  0.72  (clearly different)
# Layer 9:  0.58  (very different)
# Layer 12: 0.42  (completely different — different meanings)

Chapter 9: Connections

BERT didn't emerge in isolation — it built on years of work in transfer learning and language modeling, and it spawned an entire family of successors that addressed its limitations.

What BERT built on

The BERT family

BERT vs GPT: two philosophies

BERT and GPT represent two fundamentally different approaches to language AI, and the field eventually chose GPT's path:

BERT's lasting legacy

Even though GPT-style models dominate today, BERT's contributions remain foundational:

BERT-based models remain the default choice for production NLP tasks that don't require generation: search ranking (Google used BERT in its search engine starting 2019), sentiment analysis, entity extraction, text classification, and semantic similarity. When you need fast, accurate understanding of fixed text — not open-ended generation — BERT is still hard to beat.

"BERT is deeply bidirectional, GPT is unidirectional. There are advantages to both, but the big advantage of BERT is that it can be used for understanding tasks more effectively."
— Jacob Devlin