A good description makes a hard problem easy. Two rings of dots cannot be split by one straight cut, until you describe each dot by its distance from the centre.
Learn the review that gave deep learning its map: what makes a representation good, and how auto-encoders, sparse codes and Boltzmann machines learn one without labels.
Pick a way to describe each dot, then try the third option: press Teach the hidden layer and watch a small network find its own description. Then we build, piece by piece, the paper’s whole argument: the priors, the features, the auto-encoders and the probabilistic models.
You need what a vector is (a list of numbers) and a little probability (a number from 0 to 1 that says how likely something is). We build the rest from zero.
Based on “Representation Learning: A Review and New Perspectives” by Yoshua Bengio, Aaron Courville and Pascal Vincent (University of Montreal), arXiv:1206.5538, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013. It is an assigned reading of MIT MMAI, Multimodal AI, and goes with Lecture 1, Introduction. Every idea is re-taught here in our own words and devices.
Toy data, seeded so it is the same on every visit: 60 dots on a ring of radius 1 and 60 on a ring of radius 2, each pushed in or out by random fuzz. “Raw” and “distance” show the best cut found by brute force (every direction in 3-degree steps for a line; every threshold for the circle). “Learned” is a 2-input, 4-unit, 1-output network trained on the labels by gradient descent in your browser. Illustrative: the paper has no such experiment; it argues the same point in Section 1.
Chapter 0
See why the same data can be hopeless in one description and easy in another, and what the review sets out to do about it
Picture a robot arm over a table. Coins lie on the table, seen from above. One kind of coin sits on a small circle around the middle. The other kind sits on a bigger circle around the same middle. The arm has a camera that reports each coin as two numbers, its x and its y position. Your job is to build a fence that sends one kind of coin to the left bin and the other kind to the right bin.
You try a straight fence. It fails. Whatever direction you pick, the big circle wraps all the way around the small one, so the fence always cuts through both kinds. Now change one thing. Stop giving the arm x and y. Give it one number per coin: the distance from the centre. Every coin on the small circle reports about 1. Every coin on the big circle reports about 2. A fence at 1.5 now works, and it is a single cut.
The coins did not move. The arm did not get smarter. Only the description of each coin changed. That is the whole subject of this lesson. A representation is the list of numbers you use to describe one example. Each number in it is a feature. The hero at the top of this page lets you try three descriptions of the same dots. Go back and try them once more, then read on.
The paper opens with a plain observation. How well a learning method works depends heavily on the representation it is given. For that reason, most of the effort in a real project goes into feature engineering: people designing the preprocessing steps and data transformations that produce good features. The authors call this important but labor-intensive. They also say it shows a weakness of today’s learners, which cannot pull out and organize the useful information from raw data by themselves. Feature engineering is how human ingenuity covers for that weakness.
The distance trick above is feature engineering. A person looked at the problem, noticed the rings, and wrote the formula. That worked because the problem was simple and the person was clever. The paper asks for something bigger: learners that find such descriptions on their own, so that new applications are built faster, and so that machine learning moves toward artificial intelligence.
The authors give a reason why the description matters so much. Their hypothesis is that different representations entangle and hide, to different degrees, the different explanatory factors of variation behind the data. A factor of variation is one cause that makes examples differ, such as the direction of the light in a photo, the shape of an object, or the speaker in a recording. A good representation pulls those causes apart. A poor one smears them together. Try it on a toy photo.
A bright blob sits on a strip of 12 pixels. Two things can change: where the blob is, and how bright it is. The dashed outline is the starting picture. Move one slider at a time, then flip the view to the two factors.
Illustrative toy, not from the paper. Pixel i holds brightness × exp(−(i − position)2 / 2.88), a blob about 1.2 pixels wide. A value counts as “changed” when it moves by more than 0.03 from the starting picture (position 5, brightness 0.7).
In the pixel view, one cause touches many numbers. Make the blob brighter and most of the pixels near it change. Slide the blob and a different set of pixels changes. No single pixel belongs to one cause. In the factor view, the two causes each own one number. If a learner had to find “is the blob in the left half?” from pixels alone, it would have to untangle the mix first. From the factors, it only reads one number.
The paper was written when this idea was starting to win in practice, and it opens with a list of wins. It reports a speech system from Microsoft that cut the word error rate on four major benchmarks by about 30%, for example from 27.4% to 18.5% on one of them (RT03S), with the same 309 hours of training speech. It reports that deep networks broke the lead of support vector machines on MNIST handwritten digits (1.4% error) and reached 0.27% error. And it reports that on the 1000-class ImageNet benchmark the best error fell from 26.1% to 15.3%. Chapter 9 collects these in a table. Representation learning also won two transfer learning challenges held in 2011.
The authors ask three questions, and the rest of the paper is organized around them. What makes one representation better than another? Given an example, how do we compute its representation? And what objective should we train on to get good representations, when the real goal is somewhere downstream, such as a classifier we have not built yet?
They sort the answers into three families that grew up apart. Probabilistic models treat the features as hidden causes of the data. Auto-encoders train a network to rebuild its input through a narrow code. Manifold learning asks about the geometry of where the data lives. The conclusion says the three look disconnected but are being tied together, and a large part of this lesson is following those ties.
Here is the road. It follows the paper’s order, with its sections noted.
Chapter 1
Count why smoothness alone fails in many dimensions, and meet the list of priors the paper offers instead
You run a bakery and want to predict tomorrow’s sales. You know three things about each day: the temperature, whether it rains, and whether it is a holiday. After a few hundred days of records, a simple habit works well. To guess tomorrow, find the past days that look most like tomorrow and average their sales. This habit has a name, nearest neighbours, and it rests on a belief about the world.
The belief is that the world is smooth: if two situations are close, their outcomes are close. The paper lists this as the first and most basic prior. A prior is an assumption you build in before you see the data. Smoothness is so basic that almost every learner uses it. Kernel machines, nearest neighbours, decision trees and Gaussian support vector machines all lean on it. The paper calls this local generalization: the answer at a new point comes mostly from training examples near it.
Local generalization has a hidden price. To predict well in a region, you need examples inside that region. The learner has to see the wrinkles of the target function, the ups and downs, one neighbourhood at a time. The paper’s point is that the number of wrinkles can grow exponentially with the number of relevant factors that interact. This is the curse of dimensionality.
Work it through with the bakery. Cut each factor into 3 levels: low, medium, high. With 1 factor there are 3 cells to cover. With 2 factors there are 3 × 3 = 9. With 3 factors there are 27. With 10 factors there are 310 = 59,049 cells. A local learner needs about one example per cell to know what happens there. Ten factors is not many; one small image already has far more.
Now ask what a different learner would need. Suppose the learner knew the effect of each factor can be learned on its own and then combined. Then it needs about 3 numbers per factor, so 3 × 10 = 30 numbers for ten factors. The gap between 59,049 and 30 is the whole argument. Move the slider and watch the two counts part ways.
Each factor is cut into levels. The warm bar counts cells a local learner must visit. The blue bar counts numbers a learner needs if it handles each factor on its own. The scale is logarithmic: each gridline is ten times the one before.
Illustrative counting, not an experiment: cells = levelsfactors, and “separate” = levels × factors. The paper makes the same argument in words in Section 3.2; real functions are not this extreme, but the growth rate is the point.
This is why the paper does not rely on smoothness alone. It says that smoothness-based learners and linear models are still useful, but on top of a learned representation. It also makes a neat remark: learning a representation and then fitting a kernel machine on it is the same as learning the kernel, which is the similarity measure. Kernel machines need someone to choose that similarity in advance. The authors would rather find it from data, with very general priors as a guide.
So which general priors? Section 3.1 gives a list. None is about one task. Each is a belief about the world that tends to be true for the hard problems the authors call AI-tasks, such as vision and language. Read the table once now. Later chapters cash each one out.
| Prior | In plain words | Cashed out in |
|---|---|---|
| Smoothness | Close inputs give close outputs. | This chapter: necessary, not enough |
| Multiple explanatory factors | The data comes from several causes. What you learn about one cause carries over to many settings of the others. | Chapters 2 and 3 |
| Hierarchy of factors | Big concepts are built from smaller ones. | Chapter 2 (depth) |
| Semi-supervised learning | The causes that explain the inputs also explain most of the labels. | Chapter 8 |
| Shared factors across tasks | Different tasks use overlapping causes. | Chapter 2 |
| Manifolds | The data fills only a thin, low-dimensional sheet of the space. | Chapters 4 and 6 |
| Natural clustering | Each class sits on its own sheet, with empty space between classes. | Chapter 6 |
| Temporal and spatial coherence | Neighbours in time or space usually share the same category. | Chapter 8 |
| Sparsity | Only a few causes matter for any one example. | Chapters 6 and 7 |
| Simple factor dependencies | At a high level, causes relate in simple, often linear, ways. | Chapter 3 |
One worked example of the last row: if the learned features were good, then a plain linear classifier on top of them should do well, because the factors relate to the label in a simple way. This is exactly how the paper later judges features: by training a cheap linear model on them. It is also the reason why the paper’s own test is not “which method is cleverest” but “which method leaves the easiest problem for the model on top”.
Section 3.6 makes a point that is easy to skip. In classification the goal is obvious: make few mistakes. In representation learning the goal is far from the thing you care about, which is some classifier you have not trained yet. The authors compare it to the credit assignment problem in reinforcement learning: a reward arrives late, and it is hard to say which earlier choice earned it. They suggest the answer is open. Is it enough to maximize the likelihood of the data under a good model? Or should we add priors such as the ones above, possibly ones that depend on the data? The rest of the paper is, in effect, a tour of attempts to answer that.
Chapter 2
Compare one label per region with many features that switch on together, and see why depth multiplies the reuse
Imagine you must describe a stranger to a police artist. One way is to hold a book with 1,000 numbered faces and say “face 417”. If the stranger is not in the book, you are stuck. Another way is to answer short questions: tall or short, beard or none, glasses or none, dark hair or light. Ten yes or no answers give 210 = 1,024 different descriptions, and each answer is reused by thousands of faces.
That is the difference between a one-hot code and a distributed one. In a one-hot code, exactly one of N slots lights up: “face 417”. In a distributed code, many features can be on at once, each one is a yes or no about some property, and a concept is a pattern of features. The paper adds a word: a sparse representation is a distributed one where only a few of the features may be on at the same time.
Section 3.3 turns this into a count. Ask: how many parameters does a model need, compared with how many different regions of input it can tell apart? Clustering methods, Gaussian mixtures, nearest neighbours, decision trees and Gaussian SVMs all give a one-hot answer. They need on the order of N parameters, or N examples, to separate N regions. That seems natural. But a restricted Boltzmann machine (Chapter 7), sparse coding, an auto-encoder or a multi-layer network can separate on the order of 2k regions with only about N parameters, where k is the number of features that can be on at once.
The reason is reuse. Each feature in a distributed code is one parameter set, for example one direction in input space with a threshold: a line, or a flat sheet in higher dimensions. That one feature takes part in telling apart a huge number of regions, because it can combine with every other feature. In a one-hot code, each region owns its own private parameters and nobody shares. Try it in two dimensions.
Each unit costs about the same: a centre has two numbers, a line has two numbers and an offset. Add units one at a time. Each colour is one distinct code. Count the colours in each view.
Illustrative toy: the lines and centres are fixed by hand, not learned. The region count for N lines in general position in a plane is 1 + N + N(N − 1)/2 (exact, from the geometry of lines); for centres it is N. In a space with more dimensions than lines, N lines give up to 2N regions, which is the paper’s O(2k) claim. A plane is too small to show that growth, so it shows only the start.
With 4 units the one-hot code has 4 regions and the line code has 1 + 4 + 6 = 11. With 8 units it is 8 against 1 + 8 + 28 = 37. In a plane, the gain is a square, not an exponential, because two dimensions leave little room for lines to cross in new ways. In a space with many more dimensions than lines, each new line can double the number of regions, and that is where 2k comes from.
The generalization of clustering to distributed codes is what the paper calls multi-clustering. It either runs several clusterings in parallel or applies one clustering to different parts of the input. A well-known case in the paper is object recognition that clusters small image patches and then counts how many patches fall in each cluster. It is a distributed code made of one-hot pieces.
The same idea of reuse explains why the paper cares about depth. The depth of a circuit is the length of its longest path from an input node to an output node. A circuit can have exponentially more paths as depth grows. Take a chain of layers where every unit feeds both units of the next layer. From one given input unit to one given output unit there are 2L−1 routes for L layers: 1 route for one layer, 2 for two, 4 for three, 8 for four. Each route is a different way to reuse the parts in between.
The paper states the formal result carefully. If you change what each node may compute, the depth changes only by a constant factor. Within that fixed menu of node types, which includes weighted sums, products, artificial neurons, kernels and logic gates, there are families of functions that a deep circuit computes with exponentially fewer parts than a circuit that is not deep enough. Learning theory then says that fewer parameters need fewer examples. So depth gives two gains: it saves computation, with fewer nodes to visit, and it saves data, with fewer parameters to learn and each one reused on many kinds of input.
Depth also builds abstraction. More abstract concepts can be built from less abstract ones: strokes make letters, letters make words. A more abstract feature is also more invariant, which means it does not change under most small changes of the input. “This is the digit 3” stays true when the 3 is shifted a pixel or drawn thicker. Because of this, abstract features are usually highly non-linear functions of the raw input. A convolutional network builds in some of this explicitly with pooling (Chapter 8).
The paper’s Figure 1 makes the reuse concrete for several tasks. The middle layer of a network holds features that are partly shared. Some features explain the input. Different subsets of them explain the target of task A, task B and task C. Because those subsets overlap, what is learned for one task helps the others. The paper cites the 2011 transfer learning challenges, both won by representation learning methods, as support for this picture. It is a hypothesis backed by results, not a theorem.
Chapter 3
Separate the hidden causes of the data, and learn why dropping information is not the same as untangling it
Look at a photo of a coffee mug on a desk. Many separate causes made those pixels. A lamp lit the scene from the left. The mug has a shape and a glossy surface. A pen lies beside it and throws a shadow that falls across the mug’s handle. The shadow edge looks like an object boundary, although no object is there. Each cause is simple on its own. Together they interact in a tangled web.
The paper asks a pointed question about this scene: how do we disentangle the objects and their shadows? Its answer is a belief, not a proof. It must use the data itself, in huge amounts and mostly unlabeled, to learn representations that separate the explanatory sources. If that works, the representation should be far more robust to the rich variation in real data.
Section 3.5 separates two goals that are easy to mix up. The first is invariance: build features that do not react to changes that do not matter. A digit detector should not care whether the 3 is shifted or thick. The second is disentangling: build features so that each underlying factor lives in its own place, and keep all of them.
The paper says the central difference is the preservation of information. An invariant feature, by definition, is less sensitive in the direction it is invariant to. That throws information away. It is a bet that the discarded direction will never matter. The trouble is that you often cannot know in advance. The same learned features may later serve several tasks, and each task may need a different subset of factors. A feature that ignored the pose of the mug is useless for a task that asks where the mug points.
So the authors conclude that the most robust path is to disentangle as many factors as possible and discard as little information about the data as is practical. If you must reduce dimensions, they suggest dropping first the local directions of variation that the training data shows least, as PCA does, though PCA does it for the whole data set at once instead of around each example.
Here is the smallest version of the idea. Two hidden factors, call them size and tilt, are independent: knowing one tells you nothing about the other. A representation is two numbers, h1 and h2. We build them by turning the axes by an angle θ:
At θ = 0, each number is one factor: disentangled. At θ = 30°, h1 = 0.87 · size + 0.5 · tilt. It is mostly size but carries a piece of tilt. At θ = 45°, both numbers are equal blends, 0.71 of each factor. Nothing is lost at any angle, because turning axes keeps all the information. But at 45° you cannot read size or tilt from one number. Drag the angle, then switch on the invariance test.
The cloud is 150 examples, each with a size and a tilt. The warm axes are your representation. The bars on the right show how strongly each axis follows each factor. Then press Keep only axis 1 to throw one number away.
Illustrative toy: 150 points with size and tilt drawn independently and uniformly, seeded. The bar for an axis and a factor is the absolute sample correlation. “Recover” is the share of a factor’s variance a straight-line fit on the kept axis explains (the squared correlation).
Set the angle to 0 and keep only axis 1. You recover all of the size and none of the tilt. That is invariance to tilt, and it is perfect, as long as tilt never matters. Now set 45° and keep one axis: you recover half of each factor and have a feature that is neither invariant nor clean. Finally, keep both axes at 0°: nothing is lost, and each factor is readable alone. That is the disentangled representation the paper wants.
Here the factors were simple and independent, so a rotation was enough. Real data is not like that. The paper says it is doubtful that independent underlying factors relate to high-dimensional observations by a linear map, which is why it moves on to non-linear learners. It also notes that independent component analysis, which finds independent linear features, is related to the goal but is not sufficient for complex real data.
Section 11.4 admits how much is open. Pooling, as in convolutional networks, builds invariance in two steps: recover low-level features, then pool subsets of them together. But pooling loses details, and we have little control over what is lost. What we want is a feature set that is invariant to the irrelevant factors and also separates the relevant ones. The paper lists this as a research direction, not a solved problem. The training criteria that follow in this lesson do not state disentangling as a goal. They push toward it indirectly through the priors of Chapter 1.
Chapter 4
Fit the best straight line through the data, see why that is a representation, and see where a line gives out
Measure the height and the arm span of 1,000 adults. Plot each person as a dot, with height across and span up. The dots form a thin diagonal cloud, because tall people have long arms. You do not need two numbers for each person. One number, the position along the long direction of the cloud, says almost everything. The second number, the distance off the cloud, is small and mostly noise.
That is principal components analysis, PCA, the oldest feature extraction method in the paper. The authors use it as a thread through the whole review. They show that PCA has a probabilistic reading (Chapter 7), an auto-encoder reading (Chapter 5) and a manifold reading (Chapter 6). If you hold PCA in mind, the later methods look like its non-linear descendants.
PCA learns a linear map from the input x, a vector of dx numbers, to the representation h, a vector of dh numbers. The paper writes it as h = f(x) = WTx + b. The columns of the dx × dh matrix W are an orthogonal basis for the dh directions along which the training data varies most. The features in h come out decorrelated, meaning that none of them predicts the others linearly.
Where do the directions come from? Centre the data so its average is zero. Build the covariance matrix C, which holds the variance of each coordinate on the diagonal and the covariance between coordinates off it. Its eigenvectors with the largest eigenvalues are the principal components. The paper notes this link: the leading eigenvectors of the covariance matrix are the principal components of the data.
Do it for the two-number case. Suppose the covariance matrix is C = [[2, 1.2], [1.2, 1]]. The trace is 2 + 1 = 3 and the determinant is 2 × 1 − 1.2 × 1.2 = 0.56. The eigenvalues solve λ2 − 3λ + 0.56 = 0, so λ = (3 ± 2.6) / 2, which gives 2.8 and 0.2. For the larger one, solve (C − 2.8 I)v = 0. The first row says −0.8 v1 + 1.2 v2 = 0, so v is proportional to (1.2, 0.8). Scale it to length 1 and get (0.832, 0.555). With one component kept, h is one number, 0.832 x1 + 0.555 x2. It carries 2.8 / 3 = 93% of the variance. The average squared miss when you rebuild x from h is the dropped eigenvalue, 0.2.
The problem is in the name: linear. The paper states it plainly. The expressive power of linear features is very limited, and they cannot be stacked into deeper, more abstract representations, because the composition of linear operations is another linear operation. Check with numbers. Let the first layer be W1 = [[1, 1], [0, 1]] and the second W2 = [[2, 0], [1, 1]]. Applying W1 then W2 to x = (1, 2): W1x = (3, 2), then W2(3, 2) = (6, 5). The single matrix W2W1 = [[2, 2], [1, 2]] applied to (1, 2) gives (6, 5) too. Two layers equal one. Depth buys nothing unless something non-linear sits between the layers.
There is a second limit, about shape. Data that lies along a curve is not well described by a straight line. Bend the data below and watch.
Sixty points sit on an arc of fixed length, with a little noise. The warm line is PCA’s first component. Thin lines show each point’s miss. Drag the bend, then switch to the position along the curve.
Illustrative toy, computed live: arc length is fixed at 2, so the arc has radius 2 divided by the bend in radians; each point gets radial noise with standard deviation 0.04. PCA is the exact eigen-decomposition of the 2 × 2 covariance matrix. “Along the curve” uses the known arc, which a learner would have to discover.
At 0° the data is a straight line and PCA is perfect. At 150° the first component still carries most of the variance, yet it misses each point by a clear margin, and points at the two ends of the arc land on top of points from the middle. At 300° the arc almost closes into a circle and the line is a poor summary. But one number, the position along the arc, still describes every point with only the noise left over. The data is one-dimensional and bent. A straight line cannot see that.
This gap is what the non-linear methods in the paper exist to close. They want features that can follow a curved sheet of data and can also be stacked. In the paper’s words, the aim is non-linear features that can be composed in deep networks. PCA remains the reference point, and it will return in Chapter 5 in the form of a tiny network.
Chapter 5
Train a network to rebuild its own input through a narrow code, and watch the code become a representation
A photocopier makes a copy of a page. If it works, the copy matches the page. Now put a tiny gap in the middle of the machine, so that the page must be squeezed through a note of ten numbers before the copy is printed. If the copy still looks right, those ten numbers must hold what matters about the page. Nobody told the machine what matters. The demand to copy well forced it to find out.
That is an auto-encoder. It has two halves. The encoder turns an input into a short code. The decoder turns the code back into a reconstruction of the input. Training makes the reconstruction as close to the input as possible. No labels are involved, so it learns from unlabeled data, and the code is the representation.
Section 7.1 writes the encoder and decoder as an affine map followed by a non-linearity. An affine map is a matrix multiply plus a bias, a list of offsets.
Take the loss L from the data type. For unbounded inputs the paper says a linear decoder with squared error is natural: L(x, r) = ‖x − r‖2. For inputs between 0 and 1, use a sigmoid decoder. For binary inputs, a binary cross-entropy loss is common. The aim is that L is the negative log-likelihood of the observed x.
Follow the shapes for a batch of B examples. The input X has shape B × dx. The code H = sf(XWᵀ + b) has shape B × dh. The reconstruction R = sg(HW′ᵀ + d) has shape B × dx. The loss compares R with X. After training, the decoder is usually thrown away, and H is the representation fed to a classifier.
Run one step by hand. Let the input be x = (1, 0.5) and use no non-linearity. Let the encoder be W = [0.8, 0.6] and the decoder W′ = (0.8, 0.6) as a column. The code is h = 0.8 × 1 + 0.6 × 0.5 = 1.1. The reconstruction is r = 1.1 × (0.8, 0.6) = (0.88, 0.66). The error is r − x = (−0.12, 0.16), and the loss is 0.0144 + 0.0256 = 0.04. The gradient of the loss with respect to r is 2(r − x) = (−0.24, 0.32). The gradient for the decoder is that vector times h: (−0.264, 0.352). With a step size of 0.1 the decoder becomes (0.8264, 0.5648). The new reconstruction is (0.909, 0.621), and the loss falls to 0.0229. That is all that training does, repeated.
There is a trap. If the code has as many numbers as the input, or more, the machine can learn to copy. The identity function has zero reconstruction error everywhere, so a perfect score teaches nothing. The paper puts it directly: something in the training criterion or the parametrization must prevent the identity. Good generalization means a low error on test examples, and a high error for most other configurations of x. A model that rebuilds everything has learned nothing about which inputs are likely.
The classic fix is a bottleneck: a code with fewer numbers than the input, dh < dx. The paper proves one link here. A linear auto-encoder with squared error learns the same subspace as PCA. The weight vectors it learns are not forced to be orthonormal or ordered, but they span the same space. Watch it happen below with real training in your browser.
Eighty points lie along a tilted cloud. The warm line is the direction the decoder writes in. The dashed line is PCA. The pink dot is a test point off the cloud. Train with a one-number code, then with a two-number code, and compare how each treats the pink dot.
Real training, toy data: a linear encoder and decoder with no biases, squared error, plain gradient descent (step 0.08, 10 steps per frame, up to 1,000 steps) on 80 seeded points with standard deviation 1.0 along the cloud and 0.25 across it. The paper’s claim (Section 7.1) is the equivalence with PCA for linear auto-encoders; the data and sizes are ours.
With a code of one number, the decoder line swings into the PCA line and the loss settles at the small leftover variance across the cloud. The pink test point is pulled onto the line, so its reconstruction error is large. That is the good behaviour: the machine knows the pink point is unlikely. With a code of two numbers, the loss keeps sinking toward zero, and the pink point is rebuilt almost exactly. The machine has learned to copy and cannot tell a likely input from an unlikely one.
First, a practical advantage: auto-encoders define a simple, tractable objective, and you can watch it fall to monitor progress. This is much harder for the probabilistic models of Chapter 7. Second, an echo: if the encoder and decoder both use a sigmoid, then f(x) and g(h) have the same form as the conditional distributions P(h | x) and P(x | h) of a binary RBM. That similarity led to a study of replacing RBMs with auto-encoders to pre-train deep networks (Chapter 8).
Chapter 6
Block the copy-the-input shortcut with sparsity, noise or a Jacobian penalty, and see the manifold appear
Chapter 5 left a puzzle. A bottleneck stops the copy shortcut, but it also caps the code at fewer numbers than the input. The paper notes that sparse coding and Boltzmann machines do well with overcomplete codes, which have more features than inputs. So how do you keep a wide code from simply copying? The answer is to add a pressure that makes copying costly. The paper reviews three such pressures. Each has the same spirit: the code should be as insensitive as possible to the input, while still carrying enough to rebuild the training examples.
A sparse auto-encoder adds a penalty on the hidden activations h, so most of them sit at zero for any one input. The simplest penalty is the L1 norm, the sum of the absolute values. A close cousin is the Student-t penalty, log(1 + hj2). Some papers instead push the average activation of each unit, over a mini-batch, toward a small target such as ρ = 0.05, using a Kullback-Leibler divergence: −ρ log h̄j − (1 − ρ) log(1 − h̄j) plus a constant. The paper also warns that penalizing the bias is risky, because the weights can compensate.
A denoising auto-encoder, DAE, changes the task. Corrupt the input on purpose, then ask the machine to rebuild the clean original from the damaged copy. Copying no longer helps, because the copy would contain the damage. The paper’s cost averages over the corruptions:
Why does this teach the shape of the data? Suppose the data sits near a thin sheet, a manifold, in a large space. Random noise mostly pushes a point off the sheet, almost at right angles to it. To undo the damage, the machine must map low-probability points back toward high-probability ones, which means toward the sheet. The reconstruction function becomes a vector field pointing at the high-density region. The paper links this to the score, the derivative of the log of the data density: the vector r(x̃) − x̃ points along the estimated score. A later result generalizes this to auto-encoders of any shape with small Gaussian noise. Try the ideal case on a ring.
Clean data lies on a ring of radius 1. Pink dots are the corrupted copies. Purple dots are what the best possible denoiser outputs. Faint arrows show the whole field. Raise the noise and watch how far the field can still reach.
Exact toy: data uniform on a unit ring plus Gaussian noise of the chosen standard deviation. The best reconstruction under squared error is the mean of the clean point given the damaged one, which has the closed form A(k) times the damaged point’s direction, with A = I1(k)/I0(k) (Bessel functions) and k = distance / noise2. A trained DAE would only approximate this field. The paper’s Figure 3 draws the same idea.
At low noise, the denoised points land almost exactly on the ring, and the field is a clean pull toward it. At high noise, the damaged points no longer carry a clear memory of where the ring was, and the best guess shrinks toward the middle. The denoiser can only undo damage that is small compared with the structure it learned.
A contractive auto-encoder, CAE, attacks sensitivity directly. It adds a penalty on how fast the code changes when the input changes: the squared Frobenius norm, the sum of squared entries, of the encoder’s Jacobian J(x), the matrix of derivatives of h with respect to x. The cost is JCAE = Σt [ L( x(t), g(f(x(t))) ) + λ ‖J(x(t))‖F2 ], where λ sets the strength. For a sigmoid encoder it has a short closed form: the penalty is Σj (hj(1 − hj))2 ‖Wj‖2, the sum over hidden units j of the squared slope of the sigmoid times the squared length of that unit’s weight row.
Check it on one unit with Wj = (3, 4), so ‖Wj‖2 = 25. If the unit sits at h = 0.5, the slope term is (0.5 × 0.5)2 = 0.0625, and the penalty is 25 × 0.0625 = 1.56. If the unit is saturated at h = 0.99, the slope term is (0.99 × 0.01)2 = 0.000098, and the penalty is 0.0025. The penalty is about 640 times smaller. This is why the paper observes that CAE codes tend to be saturated, with most units near 0 or 1 and a few unsaturated, sensitive ones.
The reconstruction term fights back. It needs different training points, especially neighbours, to get different codes so they can be rebuilt. So the machine becomes insensitive in every direction except along the data. The few sensitive directions are the tangent space of the manifold at that point, the directions in which you can move and still stay on the data. The paper’s Figure 4 shows tangent vectors learned on MNIST digits: each is a plausible deformation of the digit, and unlike PCA they differ from one input to another.
The authors list three differences from the DAE: the CAE penalizes the sensitivity of the features, not of the reconstruction; its penalty is an exact formula, not an average over random noise; and a separate λ trades off reconstruction and robustness, which a DAE mixes together. They also show that a DAE with small noise is a contractive auto-encoder in disguise, with the penalty on the whole reconstruction function.
Chapter 7
Treat features as hidden causes, watch causes compete, and learn a Boltzmann machine without ever computing its normaliser
You are on holiday when your phone buzzes: the alarm at home went off. Your first thought is a burglar. Then the radio says a small earthquake hit your town. You relax at once. The alarm has not changed, but a second cause has appeared that explains it, and the burglar now seems far less likely. Your mind just did something clever: it let two possible causes compete to explain one observation.
This is the heart of the probabilistic view in Section 6. A feature is a hidden cause. We cannot see it, so we call it a latent variable, written h. The data we can see is x. A model of both is a joint distribution p(x, h), the probability of every pair. A feature value is then the answer to an inference question: given this x, what is the distribution of the causes, p(h | x)? That distribution is the posterior.
A directed model builds the joint from a prior on the causes and a recipe for the data: p(x, h) = p(x | h) p(h). Examples in the paper are probabilistic PCA, sparse coding and sigmoid belief networks. The paper uses the alarm story to explain a property of such models, called explaining away: two causes that were independent before become dependent once you see their shared effect. Work the numbers with the device. They are illustrative, but the arithmetic is exact.
Two rare causes can set off the alarm. Choose what you know. The bars show the chance of each cause given that evidence. The grey ticks mark the chance given the alarm alone.
Illustrative numbers, not from the paper: chance of a burglary 1%, of an earthquake 2%; the alarm rings with chance 0.1% if neither happens, 95% for a burglary alone, 30% for a quake alone and 96% for both. The posterior chances come from exact enumeration of the four cases, done live.
With no evidence, a burglary has a 1% chance. After the alarm rings, the number leaps to about 58%. Then add the radio report of a quake, and it falls to about 3%. The earthquake explained the alarm, so the burglar was no longer needed. The paper says this effect promises a parsimonious posterior, one that uses few causes to explain the input. It is useful for features, because features then compete and collaborate instead of all firing at once. The price is that inference becomes hard, and can be intractable, especially when the causes are discrete.
The best-known directed feature model is sparse coding. It writes the input as a mix of a few columns of a matrix W, called the dictionary. The code of a new input x is found by solving a small optimization problem (Equation 2):
The probabilistic reading puts a Laplace prior, which is peaked at zero, on h. Then h* is the most probable code, the MAP estimate. Sparse coding shows explaining away at its best: even when the dictionary is very overcomplete and its columns are strongly correlated with the input, the search can pick the right few columns and zero out the rest. The paper cites a result on the CIFAR-10 image set: with fewer than 1,000 labeled examples per class, sparse coding features beat other strong encodings.
The other family is undirected. It does not say which variable causes which. It scores each joint configuration with an energy, where low energy means likely: p(x, h) = exp(−E(x, h)) / Z. The constant Z is the partition function, the sum of exp(−E) over every configuration, which makes the probabilities add up to 1. The restricted Boltzmann machine, RBM, allows connections only between visible and hidden units, never within a layer. Its energy is E(x, h) = −bᵀx − dᵀh − xᵀWh. The restriction has a pay-off: given x, the hidden units are independent, so inference factorizes (Equation 12):
P(hi = 1 | x) = sigmoid( Σj Wji xj + di )
Try numbers. Let three visible units connect to two hidden units with W = [[1, −1], [2, 0], [−1, 1]], all biases zero, and x = (1, 1, 0). Hidden unit 1 gets 1 + 2 + 0 = 3, so P(h1 = 1 | x) = sigmoid(3) = 0.953. Hidden unit 2 gets −1 + 0 + 0 = −1, so P = sigmoid(−1) = 0.269. The RBM feature vector is (0.953, 0.269), computed in one pass. This is the contrast the paper draws with directed models, where the same posterior is intractable.
Training maximizes the log-likelihood, and its gradient has two parts (Equation 15): lower the energy of data configurations, the positive phase, and raise the energy of the model’s own samples, the negative phase. The positive phase is easy. The negative phase needs samples from the model, which needs an endless Gibbs chain: alternately draw h from x, then x from h. Waiting for it to settle at every step is hopeless.
Contrastive divergence, CD, starts the chain at the training example and runs just one step or a few. Persistent contrastive divergence (also called stochastic maximum likelihood) keeps the chain from the last update alive between updates. With our numbers, one CD step draws h from (0.953, 0.269), then draws a new x̃ from sigmoid(W h), and the weight change is the data product xᵀh minus the reconstruction product x̃ᵀh̃. The paper warns that persistent chains fail to mix as weights grow, and that no fully convincing fix exists yet. It also notes how much the RBM has done: fMRI classification, motion, collaborative filtering and image modelling.
Chapter 8
Build a deep model one layer at a time, then see why joint training is hard and what finally made it work
Teaching a child to read does not start with novels. First come letters, then words, then sentences. Each stage builds on the one before. The same idea sat behind the 2006 breakthrough the paper describes in Section 4. Train a deep network not all at once, but one layer at a time, each layer learning from the output of the layer below.
The recipe, started by Geoff Hinton and soon followed by others, is called greedy layer-wise unsupervised pre-training. Follow the data. Take unlabeled inputs x, for example images of 784 pixels. Train a single-layer model on them, an RBM or an auto-encoder, to get a code h(1), say 500 numbers. Freeze that layer. Now treat the codes h(1) as the new data, and train a second single-layer model on them to get h(2). Repeat. Each round adds one layer of weights. “Greedy” means that each layer is trained for itself, without looking ahead to the layers above.
When the stack is done, the paper lists what you can do with it. You can feed the top code to a standard predictor such as an SVM. You can add a logistic regression layer and fine-tune the whole stack with labels, so the pre-trained weights become an initialization. Or you can combine the layers into a generative model. The paper describes three ways to do the last: stack RBMs into a deep belief network, halve the weights into a deep Boltzmann machine, or chain encoders and decoders into one deep auto-encoder and train it on the global reconstruction error. It says honestly that no clear winner emerges.
The same recipe works with labels. In greedy layer-wise supervised pre-training, you train a one-hidden-layer network, discard its output layer, and stack another on top. The paper reports that this was worse than the unsupervised version but still better than no pre-training at all. One variant that kept pre-training every added layer in a supervised way did better still in the speech experiments it cites.
The authors split the answer in two, following earlier experiments. There is a regularization effect: starting from unsupervised features biases training toward a basin of attraction around a minimum that generalizes well. The working hypothesis is that features good for explaining the input are often good for explaining the labels, which is the semi-supervised prior of Chapter 1. And there is an optimization effect, harder to isolate, because the top two layers can overfit whether or not the lower layers learned anything. They link it to curriculum learning: simple concepts first, higher ones on top.
Why did anyone need pre-training? Because training all layers of a deep network together with plain gradient descent mostly failed. Section 10.1 gives a main suspect. Look at the transformation from one layer’s features to the next. Its Jacobian has singular values, which say how much the map stretches or shrinks in each direction. If all are below 1, the map is contractive in every direction. The learning signal must travel backward through many such maps, and it shrinks at each one. This is the vanishing gradient. The paper traces it to the work on recurrent networks, which are very deep networks with shared weights when unrolled in time.
Put numbers on it. If each layer shrinks the signal to 0.8 of its size, then after 20 layers the signal is 0.820 = 0.0115, about 1% of what it was. The first layer learns about 87 times slower than the last. If each layer instead grows the signal to 1.3, it explodes: 1.320 is about 190. Only a gain very near 1 keeps the signal alive. Try it.
The signal starts at full size at the output on the right and passes backward through each layer. Each layer multiplies it by the same gain. Pick a gain, then change the number of layers.
Illustrative simplification: one gain stands for the typical singular value of every layer’s Jacobian, and the signal is its size relative to the output. Real layers have a spread of singular values. The paper’s Section 10.1 gives the argument and points to careful initialization, which aims to make the Jacobian near 1 in all directions.
The paper lists the fixes it knew in 2013. Choose the initialization so the Jacobian of each layer is close to 1. Choose non-linearities with care, in particular sparse rectifying units. Use second-order methods or better first-order update rules. The paper also stresses a change in the setting: with large amounts of labeled data and the right initialization, very deep supervised networks can be trained without any layer-wise pre-training. Reports said pre-training then brought little or no gain when training ran long enough. This supports the view that pre-training acts as a prior, one that matters less when labeled data is plentiful.
The key example is ImageNet. The network that cut the best error on the 1000-class task from 26.1% to 15.3% used GPU training for more than 100 million example visits, a large labeled set, artificially transformed examples, a convolutional architecture with max-pooling, rectifying units, careful initialization, adaptive learning rates, feature normalization, and dropout, a trick that injects strong binary multiplicative noise on hidden units. The paper notes that dropout resembles the binary noise of a stack of denoising auto-encoders. It adds that future work must find which of these elements matter most, and how to use unlabeled data again.
Section 11 turns to domain knowledge. If you know some changes never alter the label, use them. Transformed examples add small shifts, rotations or elastic deformations to training digits. With this and a large network on a GPU, one team reached a record 0.32% error on MNIST. Convolution and pooling use only the layout of the input: each feature looks at a small neighbourhood, and the same weights are reused at every position. Temporal coherence says that categories change slowly in time. The simplest penalty on a feature h is the squared change from one time step to the next, (ht+1 − ht)2. Penalizing the absolute change instead would say that most of the time the change should be exactly zero.
Chapter 9
Keep the numbers, the terms, the checklist and the core code on one page, and find where to go next
The paper argues one thing from many angles. The success of a learner depends on the representation of its data, because representations can entangle or untangle the factors that cause the data. Generic priors about the world, listed in Chapter 1, can guide a learner toward good representations without labels: many features used together, depth, sparsity, thin manifolds, slow change over time. Three families of methods put these priors to work. Probabilistic models treat features as latent causes. Auto-encoders train a machine to rebuild its input under a pressure. Manifold methods ask about the geometry of the data. The conclusion says the three are tied together more tightly than they first seem.
All of these are results the paper reports from other work. It is a review: it has no experiments of its own.
| Task | Before | With representation learning | Where in the paper |
|---|---|---|---|
| Speech, word error rate (RT03S, 309 hours) | 27.4% | 18.5% | Section 2; about 30% lower on four benchmarks |
| Speech, mobile search (40 hours) | baseline | 16% to 23% lower | Section 2 |
| MNIST digits, error | 1.4% (SVM) | 0.27% with convolution; 0.81% with no image prior | Section 2 |
| MNIST with transformed examples | n/a | 0.32% | Section 11.1 |
| ImageNet 1000 classes, best error | 26.1% | 15.3% | Section 2 |
| Language model perplexity | 140 | 102 | Section 2 |
| Speech recognition with a neural language model | 17.2% | 14.4% | Section 2 |
| Word sense accuracy | 67.8% | 70.2% | Section 2 |
| CIFAR-10 with under 1,000 labels per class | other encoders | sparse coding wins | Section 6.1.1 |
| Sparse auto-encoder target activation | n/a | ρ = 0.05 | Section 7.2.1 |
| Term | Meaning in one line |
|---|---|
| Representation, feature | The numbers that describe one example, and each single number in them. |
| Factor of variation | One hidden cause of differences between examples, such as light or shape. |
| Distributed code | Many features on at once; a concept is a pattern, and each feature serves many concepts. |
| Sparse code | A distributed code with only a few features on at a time. |
| Invariant vs disentangled | Invariant drops a direction. Disentangled keeps every direction in its own coordinate. |
| Manifold | The thin sheet of the input space where the data really lives. |
| Auto-encoder | An encoder and a decoder trained to rebuild the input through a code. |
| Denoising, contractive | Two pressures: rebuild the clean input from a damaged one; make the code flat in every direction except along the data. |
| Explaining away | Causes that were independent compete once their shared effect is seen. |
| RBM, partition function | A bipartite energy model with easy inference, and its hard normalizing sum Z. |
| Contrastive divergence | Train an RBM using a very short Gibbs chain that starts at the data. |
| Greedy layer-wise pre-training | Learn a deep stack one layer at a time, each layer on the codes of the one below. |
Three small functions carry most of the paper’s machinery. Every shape is in a comment. The weight matrix of the auto-encoder is dh × dx; the RBM’s is dx × dh, as in the paper’s Equation 12.
pythonimport numpy as np rng = np.random.default_rng(0) sigmoid = lambda z: 1 / (1 + np.exp(-z)) def dae_step(X, W, b, d, lr=0.1, drop=0.3): # X (B, dx) in [0,1]; W (dh, dx); b (dh,); d (dx,) Xc = X * (rng.random(X.shape) > drop) # masking noise: corrupted copy, (B, dx) H = sigmoid(Xc @ W.T + b) # encoder: the code, (B, dh) R = sigmoid(H @ W + d) # decoder with tied weights W' = W.T, (B, dx) dR = R - X # cross-entropy gradient at the decoder, vs the CLEAN X dZ = (dR @ W.T) * H * (1 - H) # back through the encoder sigmoid, (B, dh) W -= lr * (H.T @ dR + dZ.T @ Xc) / len(X) # tied weights collect both paths b -= lr * dZ.mean(0); d -= lr * dR.mean(0) def cae_penalty(X, W, b): # contractive term, added to the loss with weight lambda H = sigmoid(X @ W.T + b) # (B, dh) return ((H * (1 - H)) ** 2 @ (W ** 2).sum(1)).mean() # sum_j (h_j(1-h_j))^2 * ||W_j||^2 def cd1_step(V, W, b, d, lr=0.05): # V (B, dx) binary; W (dx, dh); b (dx,); d (dh,) ph = sigmoid(V @ W + d) # P(h=1 | x), (B, dh): factorizes h = (rng.random(ph.shape) < ph) * 1.0 # sample the hidden units pv = sigmoid(h @ W.T + b) # P(x=1 | h), (B, dx) v2 = (rng.random(pv.shape) < pv) * 1.0 # one Gibbs step, started AT the data ph2 = sigmoid(v2 @ W + d) W += lr * (V.T @ ph - v2.T @ ph2) / len(V) # positive phase minus negative phase b += lr * (V - v2).mean(0); d += lr * (ph - ph2).mean(0)
Read it line by line and name the chapter behind each one. The mask is Chapter 6’s noise. The clean X in the gradient is the whole point of denoising. The tied weights come from Chapter 5. The factorized ph is Chapter 7’s easy inference. The two products in the last update are the positive and negative phases. Stack any of these layer after layer, each on the codes of the one below, and you have Chapter 8’s recipe.
This paper belongs to the first lecture of MIT MMAI, Introduction, because multimodal learning begins with the question this paper asks: what description of text, images and sound makes the useful structure easy to read? The review already shows the seed. In its Section 2 it describes learning a word embedding for each word, and learning image representations so that images and text can be linked: Google’s image search mapped images and queries into the same space, and later work built deeper multi-modal representations. The later lectures of the course, on fusion and alignment, pick up exactly that thread.
The paper is honest about its limits. It lists open questions: the right training objective for a representation, how best to compute a representation (inference), and the geometry that links representation learning, density estimation and manifold learning. It admits there is no clear winner among ways of stacking layers, that training deep Boltzmann machines is delicate, and that the many hyper-parameters make the field look like an art, though automatic search was starting to help.
Now press Present or Teach and explain the paper from memory: why the description matters, what makes one good, how an auto-encoder learns one, why noise teaches the shape of the data, and what made deep training work. Then go back to the rings, set the fuzz high, and teach the hidden layer again.