The Complete Guide to Attention & Transformers

What You'll Master

The Bottleneck Problem

You want to translate "we see the sky" into Italian: "vediamo il cielo." The standard approach in 2014 was a sequence-to-sequence model: an encoder RNN reads the English tokens one by one and compresses the entire sentence into a single hidden vector c, then a decoder RNN unfolds that vector into the Italian output.

The encoder updates its hidden state at each step: h_t = f_W(x_t, h_t-1). After processing all T input tokens, the final hidden state h_T becomes the context vector c. The decoder generates output tokens conditioned on c: s_t = g_U(y_t-1, s_t-1, c).

This works for short sentences. But think about what happens when T = 1,000. You're asking a single fixed-size vector — maybe 512 or 1,024 dimensions — to memorize the meaning, word order, and nuance of a thousand tokens. It's like compressing an entire book into a single sentence.

What's the fix? Instead of compressing the entire input into one vector, let the decoder look back at the entire input sequence on every step. At each output position, the decoder decides which parts of the input are relevant right now. Translating "cielo" (sky)? Look at position 4 ("sky"). Translating "vediamo" (we see)? Look at positions 1 and 2.

The Broader Impact

The bottleneck problem appears everywhere, not just in translation. Image captioning (compress an entire image into a vector, then generate a sentence), text summarization (compress a document, then generate a summary), speech recognition (compress an audio signal, then generate text) — all suffered from the same limitation. Attention provided a universal fix: let the decoder selectively access the encoder's full representation.

The Attention Mechanism

Bahdanau et al. (2015) proposed a simple but powerful idea: at each decoder step t, compute a fresh context vector c_t as a weighted sum of all encoder hidden states. The weights are learned — the network figures out which encoder states matter for the current output.

Step 1: Alignment Scores

For each encoder hidden state h_i, compute a scalar alignment score that measures how relevant h_i is to the current decoder state s_t-1:

Step 2: Attention Weights

Normalize the alignment scores with softmax so they sum to 1. These are the attention weights:

Step 3: Context Vector

Compute the context vector as a weighted sum of encoder states:

This context vector c_t is different at every decoder step. When the decoder is generating "vediamo" (we see), the attention weights might concentrate on h₁ and h₂ (the encoder states for "we" and "see"). When generating "cielo" (sky), they shift to h₄.

Visualizing Attention Weights

When researchers visualize attention weights on real translation tasks, something beautiful emerges: the attention matrix looks roughly diagonal for languages with similar word order (English-French: "The agreement" → "L'accord") but shows cross-diagonal patterns where word order differs ("European Economic Area" → "zone économique européenne" reverses the adjective order).

Self-Attention

Bahdanau attention is a decoder-to-encoder mechanism: the decoder queries attend to encoder states. But there's a more general operator hiding inside. What if a sequence attended to itself?

In self-attention, every token in a sequence looks at every other token in the same sequence. The input is a set of vectors X = {x₁, x₂, ..., x_N}. Each vector produces one output that's a weighted combination of information from all inputs.

Think of it like a meeting. Each person (token) has a question (query), a nametag (key), and a message (value). To form your output, you compare your question against everyone's nametag, figure out who's relevant, and blend their messages proportionally.

QKV: Three Projections, Three Roles

Why do we need three separate projections? Can't we just use X directly for everything? The answer is role separation. What a token is looking for (query), what a token advertises (key), and what a token provides (value) are three different things. The word "bank" might have a key that says "I'm a noun related to money or rivers" (for other tokens to find it), a query that says "I need context to disambiguate" (to look at surrounding tokens), and a value that says "here's my semantic content" (what it contributes to other tokens' representations). Three separate linear projections let the model learn these three roles independently.

From Cross-Attention to Self-Attention

In cross-attention, queries come from the decoder and keys/values come from the encoder — two separate sequences. In self-attention, queries, keys, and values all come from the same sequence. The shapes simplify: if the input is X [N × D_in], then Q, K, V are all [N × D_out] and the attention matrix E is [N × N].

Permutation Equivariance

A remarkable property: self-attention is permutation equivariant. If you shuffle the input tokens, the outputs get shuffled in the same way. Formally: F(σ(X)) = σ(F(X)). This means self-attention operates on sets, not sequences — it has no notion of position or order.

Scaled Dot-Product Attention

Bahdanau used a learned neural network f_att to compute alignment scores. The modern approach is simpler: use the dot product between query and key vectors, with one crucial scaling factor.

The Full Computation

Given input vectors X [N × D_in], we first project them into queries, keys, and values using learned weight matrices:

Then compute the attention output in three steps:

Or more compactly:

Why Divide by √D_k?

This is the part most presentations skip. Consider two random vectors q and k of dimension D_k, where each component is drawn from a distribution with mean 0 and variance 1. Their dot product q · k = ∑_i q_i · k_i has mean 0 and variance D_k.

Complexity Analysis

The attention matrix E has shape [N × N], where N is the sequence length. Computing it requires N² dot products, each of dimension D_k.

The Whole Thing Is Four Matrix Multiplies

Self-attention looks intimidating in equations but is remarkably simple computationally:

Cross-Attention vs. Self-Attention

It's worth pausing to distinguish these two forms precisely:

Multi-Head Attention

A single attention head computes one set of attention weights — one "pattern" of which tokens attend to which. But language has many simultaneous relationships: syntactic (subject-verb), semantic (synonyms), positional (nearby words). A single head must blend all these into one attention matrix. Can we do better?

The key trick: each head operates in a lower-dimensional subspace. If the model dimension is D = 512 and we use H = 8 heads, each head has dimension D_H = D / H = 64. The total parameter count is the same as a single head with dimension D.

Why Not Just One Big Head?

A single head with D_k = 512 and a single head with D_k = 64 have different expressiveness. But 8 heads of D_k = 64 is strictly more expressive than 1 head of D_k = 512 — it can learn 8 different attention patterns simultaneously. The output projection W_O then learns how to combine them.

Parameter Count

For D = 512: 4 × 512² = 1,048,576 parameters per multi-head attention layer. The number of heads H doesn't affect the total — it just determines how the computation is divided.

Implementation: Batched Matmul

In practice, all H heads are computed in parallel using a single batched matrix multiply. The fused QKV projection produces a tensor of shape [N × 3HD_H], which is reshaped to [H × N × 3D_H] and split into Q, K, V. The attention computation runs independently per head using batched matmul (the H dimension acts as a batch dimension). This is extremely efficient on GPUs.

The Transformer Block

Self-attention alone is not enough. It lets tokens communicate, but it doesn't add nonlinearity per token. The Transformer block wraps multi-head self-attention with three additional components: residual connections, layer normalization, and a feedforward network.

Layer Normalization

Layer normalization normalizes each vector independently across its feature dimensions. Given a vector h of dimension D:

This stabilizes training by preventing activations from drifting to extreme values. Each token's representation is normalized independently — no dependence on batch statistics like batch normalization.

The Feedforward Network (MLP)

The MLP applies the same two-layer network to each token independently:

The standard expansion factor is 4×. If D = 512, the hidden layer has 2,048 dimensions. This gives each token a chance to do "private computation" — self-attention is the communication channel between tokens, but the MLP is where individual token representations get transformed.

Residual Connections

The X + Z in step 2 is a residual connection (a.k.a. skip connection). It adds the input directly to the output. This serves two purposes: (1) gradients flow directly through the addition, avoiding vanishing gradients in deep stacks, and (2) the model only needs to learn the change to the representation, not the entire representation from scratch.

Pre-Norm vs. Post-Norm

The original Transformer (Vaswani 2017) places LayerNorm after the residual add. Modern practice moves it before the sublayer — inside the residual path:

Stacking Blocks

A Transformer is simply a stack of identical blocks. The original paper used 6 blocks. The architecture has barely changed since 2017, but models have gotten much bigger:

SwiGLU: The Modern MLP

The original Transformer uses ReLU in the MLP: Y = W₂ · ReLU(W₁ · x). Modern transformers (LLaMA, PaLM) replace this with SwiGLU, a gated variant:

The ⊙ is element-wise multiplication — a gating mechanism. One branch (Swish(X · W₁)) decides how much information to let through, while the other branch (X · W₂) provides the information. Shazeer (2020) showed this outperforms plain ReLU MLPs across the board, and offered the wonderfully honest explanation: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence."

Mixture of Experts (MoE)

The biggest modern models (GPT-4, Gemini, Claude) almost certainly use Mixture of Experts. Instead of one MLP per block, learn E separate MLP "experts." Each token is routed to only A < E of them. This multiplies parameters by E but only multiplies compute by A. A model with 1T total parameters might only activate 100B per token.

Positional Encoding

Self-attention is permutation equivariant — it treats its input as a set, not a sequence. "The cat sat" and "sat cat the" produce the same attention patterns (just permuted). But language is ordered: "dog bites man" means something very different from "man bites dog." We need to inject position information.

Sinusoidal Positional Encoding

Vaswani et al. (2017) proposed adding a fixed vector to each input token that encodes its position. For position pos and dimension i:

Each dimension of the positional encoding oscillates at a different frequency. Low dimensions change rapidly (high frequency), high dimensions change slowly (low frequency). This creates a unique "fingerprint" for each position.

Learned Positional Embeddings

An alternative: learn a lookup table of D-dimensional vectors, one per position. The model has a learnable matrix PE [N_max × D] and adds PE[pos] to x_pos. This is what GPT-2 and BERT use.

Modern Alternatives: RoPE

A limitation of both sinusoidal and learned positional encodings: they encode absolute position. Token 5 always gets the same encoding regardless of context. Modern models like LLaMA use Rotary Position Embeddings (RoPE), which encode relative position by rotating query and key vectors. The dot product Q_i · K_j then naturally depends on the offset (i − j) rather than the absolute positions i and j. This allows better generalization to sequence lengths not seen during training.

Vision Transformer (ViT)

Transformers were designed for sequences of words. Images aren't sequences — they're 2D grids of pixels. But Dosovitskiy et al. (2021) showed that you can treat an image as a sequence of patches and apply a standard transformer. The result: Vision Transformer (ViT), which matches or beats CNNs on image classification when given enough data.

Patch Embedding

Take a 224 × 224 × 3 image. Divide it into a grid of non-overlapping patches, each 16 × 16 × 3 = 768 values. This gives (224/16)² = 196 patches. Flatten each patch and apply a linear projection from 768 to D dimensions:

Position and Classification

Add learned positional embeddings to each patch vector (a learnable [196 × D] matrix). Since we flattened the 2D grid, position embeddings must encode both row and column — the transformer has no built-in 2D structure.

For classification, ViT prepends a special [CLS] token (a learned D-dimensional vector) to the patch sequence. After the transformer, the [CLS] token's output vector is fed to a linear classifier. Alternatively, you can average-pool all patch outputs and classify from that — both work.

ViT vs. CNNs

Why ViT Works

The magic of ViT is its simplicity. No pooling layers, no convolutional kernels, no special spatial operations. Just patch embedding + standard transformer + classification head. This means any improvement to the transformer architecture (better attention, better normalization, longer context) automatically benefits vision models too. The same architecture handles text, images, audio, video — unifying AI around a single computational primitive.

Self-Attention Visualizer

This is the interactive payoff. Below, you can see the full self-attention computation unfold step by step: input tokens are projected to Q, K, V vectors, the attention matrix is computed, and weighted sums produce the outputs. Select different tokens and sentences to see how attention patterns change.

How to Read Attention Patterns

When you explore the visualizer, keep these patterns in mind:

Summary & Connections

Attention started as a fix for the encoder-decoder bottleneck. Self-attention generalized it into a primitive that operates on sets of vectors. The Transformer stacked self-attention with MLPs, residual connections, and layer normalization into the most successful neural network architecture in history.

Three Transformer Families

Masked Self-Attention for Language Modeling

Decoder-only models like GPT use masked (causal) self-attention. Before computing softmax, entries where j > i are set to −∞. After softmax, these become 0 — each token can only attend to previous tokens and itself. This enables autoregressive generation: predict the next token given all previous tokens.

GPT: Decoder-Only Language Model

GPT (Radford et al., 2018) showed that a decoder-only transformer with masked self-attention, pre-trained on a massive text corpus, can be fine-tuned for many NLP tasks. The training objective is simple: predict the next token. Given tokens [x₁, ..., x_t], predict x_t+1 using softmax over the vocabulary.

The architecture is straightforward: an embedding matrix [V × D] converts tokens to vectors, a stack of masked self-attention blocks processes them, and a projection matrix [D × V] converts each output vector into scores over the vocabulary. Training minimizes cross-entropy between predicted and actual next tokens.

BERT: Masked Language Modeling

BERT (Devlin et al., 2019) uses an encoder-only transformer with no causal mask. Every token sees every other token. Training objective: randomly mask 15% of input tokens, and the model predicts the masked tokens. This forces bidirectional understanding — unlike GPT, which only looks backward.

Modern Tweaks

The core architecture has barely changed since 2017, but several refinements are now standard:

Three Ways to Process Sequences

What Hasn't Changed

It's remarkable how little the core architecture has evolved. The original 2017 Transformer had: multi-head self-attention, feedforward networks, residual connections, layer normalization, and positional encoding. Every major model in 2025 still has all five. The changes are incremental refinements (better normalization, better activation functions, better position encoding) — not architectural revolutions. The transformer is not just a good architecture. It's a stable attractor in design space.

Timeline

References