A single straight line cannot tell two simple patterns apart. The fix — stack layers, bend them with a nonlinearity, and learn by tracing blame backwards — is the engine under every CNN, transformer, and diffusion model your robot will ever run.
Part 1 left you with a powerful tool: a linear model that draws a straight boundary and slides it around to minimize a loss. For a huge number of problems that is enough. For an embarrassingly simple one, it is hopeless — and that failure is the entire reason deep learning exists.
Here is the simple problem. XOR ("exclusive or"): four points on a square. The two diagonal corners (0,0) and (1,1) are class A; the other two corners (0,1) and (1,0) are class B. Try to separate A from B with one straight line. You can't. No line, at any angle, puts both A's on one side and both B's on the other. The pattern is not linearly separable.
w·x + b and thresholds it — that is literally a line (or a flat plane in higher dimensions). Stack two linear models and you get… another line: W₂(W₁x) = (W₂W₁)x. Depth alone buys you nothing. To bend the boundary you need a nonlinearity between the layers.
The fix is two ideas working together, and you should hold them as a pair for the whole lesson:
With one hidden layer of two units and a nonlinearity in between, XOR becomes trivially separable. The network reshapes the input space until a line can cut it. Watch it happen.
The four XOR points are colored by class. Drag the line angle slider with Linear selected: no angle separates the colors. Switch to Hidden layer + nonlinearity and the learned boundary curves to wrap the diagonal — the same data, now separable.
The multilayer perceptron (MLP), also called a fully connected network, is the plainest deep network: a stack of layers where each layer is a matrix multiply, a bias add, and an activation. Nothing about it is mysterious once you track the shapes.
One layer takes an input vector and produces an output vector. If the input has d_in numbers and the output has d_out numbers, the layer holds a weight matrix W of shape (d_out, d_in) and a bias b of shape (d_out,). The layer computes:
Here z is the pre-activation (the raw linear output) and a is the activation after the nonlinearity φ bends it. The next layer eats a as its input. That is the whole architecture — a relay race of vectors.
Say we feed a single image flattened to 784 numbers (a 28×28 digit) through a network with one hidden layer of 128 units and a 10-way output (the ten digits):
| Step | Operation | Shape in | Shape out |
|---|---|---|---|
| input | flatten image | 28×28 | (784,) |
| layer 1 | W₁ x + b₁, then ReLU | (784,) | (128,) |
| layer 2 | W₂ a₁ + b₂, then softmax | (128,) | (10,) |
So W₁ is (128, 784) and W₂ is (10, 128). The output is a 10-vector of class scores; softmax turns those scores into probabilities that sum to 1. With a whole batch of N images at once, you stack the inputs into a (N, 784) matrix and every layer just processes all rows in one matrix multiply — same math, one extra dimension.
Now build the forward pass yourself, the same two-layer net but on tiny numbers you can verify by hand.
The activation function is the small nonlinear function we put after each layer's linear step. It is what lets the network bend. But the choice of activation has a giant consequence for training — one that took the field two decades to fully appreciate.
Three classics:
max(0, z) — a hinge. Below zero it outputs 0; above zero it passes the value straight through.Why does this matter so much? Because of training. When we push gradients backwards (Chapter 3), each layer multiplies the incoming gradient by the slope of its activation. Look at the slopes:
| Activation | Output range | Max slope | Slope far from 0 |
|---|---|---|---|
| Sigmoid | (0, 1) | 0.25 | → 0 (saturates) |
| Tanh | (−1, 1) | 1.0 | → 0 (saturates) |
| ReLU | [0, ∞) | 1.0 | 1.0 (if active), 0 (if dead) |
0.2510 ≈ 0.000001. The gradient reaching the early layers is essentially zero, so those layers never learn. ReLU's slope is exactly 1 wherever the unit is active, so it passes the gradient through undiminished. This single fact is why deep networks trained well only after ReLU became standard.
ReLU's one flaw: a unit whose input is always negative outputs 0 forever and its slope is 0, so it stops learning — a dead neuron. Variants like Leaky ReLU (a tiny slope below zero) patch this.
Pick an activation. The teal curve is the function; the warm curve is its slope (the gradient multiplier). Notice how sigmoid's slope flattens to nearly zero away from the center — that is the gradient vanishing — while ReLU's slope is a flat 1 on the right.
We can run a forward pass and get a loss. Now the hard question: how should each weight change to make the loss smaller? The answer is the gradient — how much the loss moves when you nudge each weight. Backpropagation is the efficient algorithm for computing every one of those gradients in a single backward sweep, using the chain rule.
The chain rule says: if the loss depends on a, and a depends on z, and z depends on w, then the loss's sensitivity to w is the product of the local sensitivities along the path:
Think of it as tracing blame backwards through the computation: the loss blames its input, which blames its input, all the way back to the weights. Each step multiplies in one local slope. A computational graph is just the picture of this — nodes for operations, edges for "depends on" — and backprop walks that graph in reverse.
Let us do a real one, all numbers, no hand-waving. A net with one input, one hidden unit (ReLU), and one output, trained to match a target with squared-error loss. The numbers:
Forward pass — push the input through:
| Quantity | Formula | Value |
|---|---|---|
| z₁ (hidden pre-act) | w₁ · x | 0.5 · 2 = 1.0 |
| a (hidden, ReLU) | max(0, z₁) | max(0, 1.0) = 1.0 |
| ŷ (output) | w₂ · a | −1 · 1.0 = −1.0 |
| L (loss) | ½(ŷ − y)² | ½(−1 − 1)² = 2.0 |
Backward pass — now trace blame from L back to each weight. Start at the loss and multiply in local slopes:
| Gradient | Chain | Value |
|---|---|---|
| ∂L/∂ŷ | ŷ − y | −1 − 1 = −2.0 |
| ∂L/∂w₂ | (∂L/∂ŷ) · a | −2.0 · 1.0 = −2.0 |
| ∂L/∂a | (∂L/∂ŷ) · w₂ | −2.0 · (−1) = 2.0 |
| ∂L/∂z₁ | (∂L/∂a) · ReLU′(z₁) | 2.0 · 1 = 2.0 |
| ∂L/∂w₁ | (∂L/∂z₁) · x | 2.0 · 2 = 4.0 |
Note ReLU′(z₁) = 1 here because z₁ = 1.0 > 0 (the unit is active). Had z₁ been negative, that slope would be 0 and the gradient would stop dead at that unit — the dead-neuron problem made concrete. With these gradients, a step of gradient descent nudges w₁ down by 4·(learning rate) and w₂ down by −2·(learning rate) (i.e. it grows), shrinking the loss.
The same tiny net. Press Forward to flow values left→right (teal). Press Backward to flow gradients right→left (warm). Drag w₁ negative and re-run: the ReLU dies, its slope becomes 0, and the gradient to w₁ is killed at the hidden node.
Trust nothing on faith. The cleanest way to check a hand-derived gradient is to compare it against a finite difference: nudge the weight by a tiny h, see how much the loss changes, divide. Let's do exactly that — you compute one gradient by the chain rule and the lab confirms it matches the numerical slope.
Backprop hands us a gradient. An optimizer decides what to do with it — how big a step to take and in what direction. The recipe behind GPT and every robot policy uses one word for this: SGD, stochastic gradient descent. But the modern default is Adam, and understanding the ladder from one to the other is worth ten minutes.
Stochastic gradient descent takes a step downhill, scaled by the learning rate η (eta):
"Stochastic" means we compute the gradient on a small random mini-batch rather than the whole dataset — cheaper, and the noise actually helps escape bad spots. The catch: in a long narrow valley, plain SGD zig-zags across the walls and crawls down the floor.
Momentum fixes the zig-zag by accumulating a running average of past gradients — a velocity. Like a ball rolling downhill, it builds speed in the consistent direction and damps the side-to-side wobble:
The β (around 0.9) controls how much memory the velocity has. Down the valley floor the gradients agree, so velocity accumulates and you accelerate; across the walls they cancel, so the wobble dies.
Adam adds a second trick: per-parameter learning rates. It tracks not just the average gradient (like momentum) but also the average squared gradient, and divides the step by the square root of that. Parameters with consistently large gradients get smaller steps; quiet ones get larger steps. The effect is that Adam mostly "just works" across very different scales without hand-tuning — which is why it is the lazy-but-correct default for deep nets.
Three optimizers descend the same elongated loss valley from the same start. SGD zig-zags; Momentum rolls through; Adam rescales each axis. Push the learning rate up until plain SGD diverges (flies off the bowl) — Adam usually survives.
Numbers, not pictures, settle arguments. The lab below runs one optimizer step on a stretched quadratic and lets you compare a momentum update against a plain SGD update from the same point. You write the momentum velocity rule.
An MLP can classify an image: flatten it to a vector and go. But for a 200×200 color image that flattened vector has 120,000 numbers, and the first layer alone would need billions of weights — and it would have to learn from scratch that a cat in the top-left looks the same as a cat in the bottom-right. The convolutional neural network (CNN) fixes both problems by baking in two inductive biases — assumptions about the data built into the architecture.
Two knobs control the slide: stride (how far the kernel jumps each step) and padding (zeros added around the border so the output stays a convenient size). Larger stride → smaller output. After convolutions, a pooling layer downsamples (e.g. take the max of each 2×2 block) to shrink the map and add a little position-invariance. The classic early example, LeNet (1998), stacked exactly this: conv → pool → conv → pool → fully connected.
f(shift(x)) = shift(f(x))). Pooling and the final readout make the classification invariant: move the cat, still "cat" (f(shift(x)) = f(x)). You can get these properties two ways — design them into the architecture (the CNN way) or teach them with data augmentation (show the model shifted/rotated copies).
A 3×3 edge-detector kernel slides across a small image. The highlighted patch on the left is multiplied element-wise by the kernel and summed into one output pixel on the right (the feature map). Step through it, and change the stride to see the output shrink.
Images have spatial structure; sequences (text, sensor streams, robot trajectories) have temporal structure. The job is to summarize "everything relevant so far" so you can predict what comes next.
A recurrent neural network (RNN) processes a sequence one step at a time, carrying a hidden state — a running summary of the past — and updating it at each new input. Elegant, but it has the same disease we met in Chapter 2: to learn a dependency between step 1 and step 100, the gradient must propagate back through 100 multiplications, so it vanishes (or, with large weights, explodes). RNNs forget the distant past.
The LSTM (long short-term memory) patches this by adding a separate cell state — a memory highway — with learnable gates that decide what to write, keep, and read. Information can flow far down the highway without being repeatedly squashed, so gradients survive longer.
The transformer throws out recurrence entirely. Instead of passing a summary step by step, it lets every position look directly at every other position through attention. The mechanism, self-attention: from each token's embedding, three vectors are made by separate linear layers — a query (Q), a key (K), and a value (V). The dot product of one token's query with another's key is a score: "how much should I attend to you?" Those scores are scaled, passed through softmax so they sum to 1, and used to take a weighted average of the values.
Because attention has no built-in notion of order, transformers add a positional embedding to each token. Each encoder layer is then: multi-head self-attention → a small fully connected network, both wrapped with residual connections (add the input back to the output) and layer normalization. The decoder is similar but also runs cross-attention to the encoder's output and generates one token at a time — it is autoregressive.
A short sentence. Click a token to make it the query; the bars show its softmax attention weights over all tokens (the keys). Notice the weights always sum to 1 — attention is a weighted average, and it can reach any position directly, near or far.
For the full story of each piece, this site has dedicated lessons: the Transformer and the Vision Transformer (attention applied to image patches — the bridge between this chapter and Chapter 5).
So far every model had labels. Unsupervised learning finds structure in unlabeled data. The lecture lists three flavors: clustering (group similar points — e.g. K-means, fit by the EM algorithm: assign points to the nearest center, then move each center to its points' mean, repeat), dimension reduction (compress to fewer numbers while keeping the variance — PCA, or nonlinear t-SNE for visualization), and generative models.
A generative model learns the distribution of the data so it can sample new examples. The lecture frames it with Feynman's motto, inverted: "What I understand, I can create." This matters enormously for robots — imitation learning is literally a conditional generative model of expert actions, and generative models can manufacture training simulations.
Three dominant paradigms, each trading off the same "impossible triangle" of coverage, speed, and quality:
| Model | Core idea | Known weakness |
|---|---|---|
| VAE | Encoder maps data to a latent distribution; decoder reconstructs. Trained to maximize a lower bound (the ELBO) on the data likelihood, with a KL term regularizing the latent toward a Gaussian. Uses the reparameterization trick to backprop through sampling. | Samples tend to be blurry. |
| GAN | A two-player game: a generator makes fakes, a discriminator tries to spot them. They co-improve. | Unstable training — mode collapse, vanishing gradients, hard to reach equilibrium. |
| Diffusion | Gradually corrupt data into noise (forward), then learn to reverse it step by step. Unlike VAE/GAN the latent has the same dimension as the data. | Slow to sample (many denoising steps). |
mean + std × ε where ε is fixed external noise. Now the randomness sits outside the network, the mean and std are smooth functions of the weights, and gradients flow right through. The same idea reappears in the forward diffusion process.
A toy 1-D latent. Drag the latent z slider; the decoder maps it to a generated shape. Because the VAE regularizes the latent toward a smooth Gaussian, nearby z values produce nearby outputs — that smoothness is what lets you interpolate and sample novel data.
The site expands each of these: VAE / VQ-VAE, GANs, Diffusion, and Flow Matching.
A robot that confidently grasps thin air is worse than one that says "I'm not sure, let me look again." So the lecture closes with a topic that is life-or-death for embodied systems: uncertainty quantification (UQ) — getting a network to report how confident it is, not just what it predicts.
Two flavors of uncertainty, and the difference matters:
The lecture's toolbox for estimating it:
| Method | Idea |
|---|---|
| Bayesian neural nets / Gaussian processes | Put a distribution over the weights/functions; predict a distribution, not a point. |
| Dropout at test time | Randomly drop units and predict many times; the spread of predictions is the uncertainty. |
| Ensembles | Train several networks; where they disagree, you are uncertain. |
| Conformal prediction | For any black-box predictor, calibrate a score so the output is a set guaranteed to contain the truth with probability 1−α. |
Five networks (faint lines) fit the same training points (warm dots). Where data is dense they agree — low epistemic uncertainty (narrow teal band). Drag the data gap slider to remove a region: out there the models fan out, and the band balloons — the ensemble knows it doesn't know.
You now own the deep-learning toolkit the rest of 16-831 assumes. Notice the single thread running through all of it: every model is differentiable layers + a nonlinearity + a loss, fit by backprop and an optimizer. CNNs, transformers, VAEs, diffusion — each just changes which layers and which loss. The machinery never changes.
This lesson is Lecture 4 of the 16-831 series. Continue with Lecture 5 — MDPs & Imitation Learning, where the discounted return and policy from Lecture 1 meet the deep networks you just built. To review the supervised-learning half, see Lecture 3 — ML / DL Refresher, Part 1. The earlier lectures: Lecture 1 — What Is Robot Learning? and Lecture 2 — Robot Learning Overview.
Companion Gleams that go deeper on each architecture in this lecture:
"What I understand, I can create." — the generative-modeling inversion of Feynman