Understand GPT
From Absolute Zero

Chapter 1: Numbers All The Way Down

Neural networks only understand numbers. A vector is just a list of numbers. A parameter is a single adjustable number the model learns during training.

Interactive Dot Product

Chapter 2: The Slope of a Hill

The derivative tells you: if I nudge the input a tiny bit, how much does the output change?

Chapter 3: Rolling Downhill

Gradient descent: compute the loss, compute gradients, take a small step downhill, repeat.

Chapter 4: The Autograd Engine

Every number in microGPT is wrapped in a Value object that tracks its gradient. When you multiply two Values, the result remembers how it was made.

python
class Value:
    def __init__(self, data):
        self.data = data   # the actual number
        self.grad = 0      # how the loss depends on this

The 6 Building Blocks

Chapter 5: From Characters to Vectors

The Problem: Numbers Aren't Rich Enough

We've built a gradient descent engine. It can adjust numbers to minimize a loss function. But here's the problem: how do we feed text into a system that only understands numbers?

The naive approach: assign each character a number. a=1, b=2, c=3... z=26. Now "cat" becomes [3, 1, 20]. But this creates a lie — it says "b" is halfway between "a" and "c," and "m" is close to "n." In reality, the letter "a" has no meaningful numerical relationship to "b." The number 1 vs. 2 is an arbitrary label, not a measurement.

A single number per character can only tell you which character it is. It can't capture anything about what that character means in context. Think of it like GPS: a single number (longitude) tells you east-west position, but you need two numbers (latitude + longitude) to pinpoint a location on Earth. With more numbers, you can describe richer things.

Step 1: Tokenize — Give Each Character a Name

Before anything else, we need a consistent way to refer to each character. We assign each one an integer ID — think of it as a name tag, not a measurement. The specific numbers don't matter; what matters is that each character gets a unique ID.

For microGPT: a=0, b=1, c=2, ..., z=25, and a special "beginning of sequence" token BOS=26. That's 27 possible tokens total.

Step 2: Embed — From Name Tags to Rich Descriptions

Now we convert each token ID into a vector. How? With a simple lookup table — a big grid of numbers called an embedding table.

Picture a spreadsheet with 27 rows (one per character) and 16 columns (one per dimension). Each row is that character's vector. To "embed" the character "e" (ID=4), you just grab row 4. No math, no computation — just look up the row.

python
# The embedding table: 27 characters, each gets a 16-number vector
wte = nn.Embedding(27, 16)   # 27 rows × 16 columns = 432 learnable numbers

# "Look up" a character: just index into the table
# token ID 4 (the letter 'e') → row 4 → a vector of 16 numbers
e_vector = wte[4]             # → [0.23, -0.81, 1.42, ..., 0.05] (16 numbers)

Before training, these 432 numbers are random. The letter "e" starts with a meaningless vector. But during training, gradient descent adjusts these numbers so that characters appearing in similar contexts end up with similar vectors. The model discovers structure on its own.

Step 3: Position — Where Are You in the Sentence?

There's a subtle problem. If we just embed each character, the model sees the word "cat" as three vectors: [vec_c, vec_a, vec_t]. But it also sees "act" as the same three vectors in a different order: [vec_a, vec_c, vec_t]. How does the model know which character came first?

The answer: we add a second set of vectors that encode position. Position 0 gets its own vector, position 1 gets a different vector, and so on. These are also stored in a lookup table and also learned during training.

python
# Position embedding table: 16 positions, each gets a 16-number vector
wpe = nn.Embedding(16, 16)   # 16 rows × 16 columns = 256 learnable numbers

# For the word "cat" at positions 0, 1, 2:
tok_emb = wte[[2, 0, 19]]     # look up c, a, t → three 16-number vectors
pos_emb = wpe[[0, 1, 2]]     # look up positions 0, 1, 2 → three 16-number vectors

# Combine: add them element-by-element
x = tok_emb + pos_emb         # "c at position 0", "a at position 1", "t at position 2"

Why add instead of sticking them side by side (concatenating)? If we concatenated, each vector would double from 16 numbers to 32 numbers, making every downstream computation more expensive. Addition keeps the size at 16. The model learns to pack both "which character" and "which position" into the same 16 numbers — they share the space.

Reading Shape Notation

From here on, we'll describe the size of data using shape notation like [3, 16]. This just means "3 rows, 16 columns" — a grid of numbers. You'll also see a third dimension: [B, T, 16], where B is the batch size (how many examples we process at once for efficiency) and T is the sequence length (how many characters). So [4, 8, 16] means "4 examples, each with 8 characters, each character described by 16 numbers."

Scaling Up

microGPT uses 27 characters and 16 dimensions. Real models are bigger, but the mechanism is identical — just larger tables:

GPT-2 and GPT-3 use subword tokens instead of individual characters — chunks like "the", "ing", "##tion" — which is more efficient. But the embedding mechanism is the same: one row per token, looked up by index.

Chapter 6: Attention — Tokens Talking

Each token creates three vectors: Query ("what am I looking for?"), Key ("what do I contain?"), Value ("what info do I offer?").

The Causal Mask: The Autoregressive Secret

GPT is autoregressive — it predicts the next token from only the previous tokens. This means token 3 must NOT see tokens 4, 5, 6... How do we enforce this? With a causal mask: a lower-triangular boolean matrix.

python
# For a 5-token sequence, the mask looks like:
mask = [
  [1, 0, 0, 0, 0],  # token 0 sees only itself
  [1, 1, 0, 0, 0],  # token 1 sees tokens 0,1
  [1, 1, 1, 0, 0],  # token 2 sees tokens 0,1,2
  [1, 1, 1, 1, 0],  # token 3 sees tokens 0,1,2,3
  [1, 1, 1, 1, 1],  # token 4 sees all tokens
]

# Where mask is 0, we set the attention score to -infinity
# softmax(-inf) = 0, so those tokens are completely invisible
scores = scores.masked_fill(mask == 0, float('-inf'))

This is the ENTIRE autoregressive property. No special architecture, no separate backward pass. Just a triangular matrix of zeros and ones applied before softmax. The implementation is one line of code, but it's what makes GPT a generative model.

Chapter 7: The Full Model

Model Configuration

Where Do the Parameters Live?

Let's account for every single parameter in microGPT:

microGPT parameter audit
# Embeddings
token_emb:   27 × 16  =   432
pos_emb:     16 × 16  =   256

# Attention (1 block, 4 heads)
W_Q:         16 × 16  =   256
W_K:         16 × 16  =   256
W_V:         16 × 16  =   256
W_O:         16 × 16  =   256

# MLP (4× expansion: 16 → 64 → 16)
W_up:        16 × 64  = 1,024
b_up:        64       =    64
W_down:      64 × 16  = 1,024
b_down:      16       =    16

# LayerNorm (2 per block, each has scale + shift)
ln1:         16 + 16   =    32
ln2:         16 + 16   =    32

# LM Head (shared with token_emb via weight tying)
lm_head:     16 × 27  =   432  # often tied with token_emb
ln_final:    16 + 16   =    32

# TOTAL:                    ~4,192

Now here's the same audit for GPT-2 Small (124M parameters):

python
import numpy as np

def layernorm(x, g, b, eps=1e-5):
    mean = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    return g * (x - mean) / np.sqrt(var + eps) + b

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def gpt_forward(token_ids, params):
    T = len(token_ids)
    d = 16

    # 1. Embed
    x = params['wte'][token_ids] + params['wpe'][np.arange(T)]

    # 2. Attention block
    x_norm = layernorm(x, params['ln1_g'], params['ln1_b'])
    Q = x_norm @ params['W_Q']  # [T, 16]
    K = x_norm @ params['W_K']  # [T, 16]
    V = x_norm @ params['W_V']  # [T, 16]

    # Scaled dot-product with causal mask
    scores = Q @ K.T / np.sqrt(d)  # [T, T]
    mask = np.tril(np.ones((T, T)))
    scores = np.where(mask == 0, -1e9, scores)
    attn = softmax(scores)  # [T, T]
    attn_out = attn @ V  # [T, 16]
    attn_out = attn_out @ params['W_O']

    # 3. Residual
    x = x + attn_out

    # 4. MLP block
    x_norm = layernorm(x, params['ln2_g'], params['ln2_b'])
    h = x_norm @ params['W_up'] + params['b_up']  # [T, 64]
    h = np.maximum(0, h)  # ReLU
    mlp_out = h @ params['W_down'] + params['b_down']  # [T, 16]

    # 5. Residual
    x = x + mlp_out

    # 6. Final norm + project to vocab
    x = layernorm(x, params['ln_f_g'], params['ln_f_b'])
    logits = x @ params['wte'].T  # [T, 27] — weight tying!

    return logits

Chapter 8: Training — Learning From Mistakes

The Shift Trick

Here's the crucial detail: the model's input and its target (ground truth) are the same sequence, shifted by one position. The target for position t is the token at position t+1.

python
# Training sequence: "emma"
# Input tokens:  [BOS, e, m, m]  (positions 0,1,2,3)
# Target tokens: [e, m, m, a]    (the NEXT token at each position)

logits = model(input_tokens)  # [B, 4, 27] — 27 probs per position
targets = input_tokens[1:]    # shift by 1

# Cross-entropy loss at EVERY position:
# Position 0: model predicts next after BOS → should be 'e'
# Position 1: model predicts next after 'e' → should be 'm'
# Position 2: model predicts next after 'm' → should be 'm'
# Position 3: model predicts next after 'm' → should be 'a'

loss = cross_entropy(logits.view(-1, 27), targets.view(-1))

The model outputs [batch, seq_len, vocab_size] logits. The loss is computed at every position simultaneously — one forward pass gives you seq_len−1 training signals. This is why language model training is so data-efficient compared to, say, image classification where each image gives you just one label.

Chapter 9: Generation — Creating Something New

The Actual Generation Loop

Here is the complete generation algorithm. It's surprisingly short:

python
def generate(model, prompt_tokens, max_new_tokens, temperature=1.0):
    tokens = prompt_tokens.clone()  # start with prompt

    for _ in range(max_new_tokens):
        # 1. Forward pass — only need logits for LAST position
        logits = model(tokens)          # [1, T, vocab_size]
        logits = logits[:, -1, :]       # [1, vocab_size] — last token only

        # 2. Apply temperature
        logits = logits / temperature   # higher T → flatter distribution

        # 3. Convert to probabilities
        probs = softmax(logits, dim=-1)

        # 4. Sample from the distribution
        next_token = torch.multinomial(probs, num_samples=1)

        # 5. Append and repeat
        tokens = torch.cat([tokens, next_token], dim=1)

    return tokens

That's it. Five lines in the loop. The model sees more and more context each iteration (or with KV cache, just the new token). Generation is inherently sequential — you can't parallelize it because each token depends on all previous ones.

Temperature

Temperature is division before softmax. It controls the "sharpness" of the distribution:

Top-k and Top-p Sampling

Temperature alone isn't enough. Even with T=0.8, the model might sometimes sample a very unlikely token (the "long tail"). Top-k fixes this by zeroing out everything except the k most likely tokens before sampling:

python
# Top-k: keep only the 5 highest logits, zero the rest
logits = [3.2, 2.5, -0.1, 1.8, -2.0, 0.5, -1.2, 1.5]
top_5  = [3.2, 2.5, -inf, 1.8, -inf, 0.5, -inf, 1.5]
# Now softmax only distributes probability among those 5

# Top-p (nucleus): keep smallest set of tokens
# whose cumulative probability ≥ p (e.g., 0.9)
# More adaptive — sometimes keeps 3 tokens, sometimes 20

Chapter 10: From Micro to Macro

The GPT Family

Here are the exact architectural parameters for every public GPT model. Notice how each dimension scales:

*GPT-4 specs are rumored (MoE with ~16 experts, ~110B active per forward pass). OpenAI has not confirmed.

The Three-Stage Pipeline

Pre-training is microGPT's algorithm at scale. SFT (Supervised Fine-Tuning) continues training on high-quality conversation data — same loss function, just better data. RLHF (Reinforcement Learning from Human Feedback) trains a reward model on human preferences, then optimizes the language model against it. The base algorithm never changes — what changes is what you train on.