Data and Heterogeneity

Scramble five items and the right model gives the same answer. A plain model must see all 120 orderings to learn that. The wiring you choose decides how much data you need.

Learn what makes pictures, words, sound, tables, sets and graphs different kinds of data, and how each kind earns a model shaped to fit it.

Pick a kind of data, then press Scramble a few times. Watch which model keeps its answer and which one changes its mind. Then we build, piece by piece, the idea underneath: every kind of data has a symmetry, and a good architecture builds that symmetry in.

You need a little algebra (adding and multiplying numbers) and a feel for what a vector is (a list of numbers). We build the rest from zero.

Based on Lecture 2.2 of MIT Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group, and its six readings. Course site and slides. Every idea is re-taught here in our own words and devices.

Scramble it, compare the answers

Toy models with fixed made-up weights, so the numbers are illustrative. The symmetric model shares one function across items and pools them (a sum, a sliding filter with a max, or neighbour messages with a sum). The plain model gives each position its own weight. The count 120 is 5 factorial, the lecture’s own figure for five set elements.

Chapter 0

One Patient, Many Shapes

See why a single hospital stay produces data that no single model design can swallow whole

A patient arrives at a hospital with chest pain. Within an hour the record holds five very different things. There is a chest X-ray, which is a grid of brightness values. There are the nurse’s notes, which are sentences. There is a heart monitor trace, which is a long wiggly line. There is a table of blood tests, one row per test. And there is a web of links between this patient, their drugs and their doctors.

You want one system to read all of it and help a doctor decide what to do next. Can one model design do that? Try to feed the X-ray into a model built for sentences, and it fails at once. Words come in a line. Pixels sit on a grid. A drug web has no first item and no last item.

This difference has a name. Heterogeneity means that the data come in different kinds, and each kind has its own shape and its own habits. This lesson, the second half of Lecture 2 of MIT’s Multimodal AI course, teaches you to see those shapes. It then shows how each shape earns a model built to fit it.

What a modality is

The lecture starts with one definition. A modality is the way in which something is expressed or perceived. Seeing and hearing are two modalities of a person. A photograph and a recording are two modalities of a dog barking at a door.

A sensor is the device that turns the world into numbers: a camera, a microphone, a thermometer. The slide draws a line that starts at the sensor. Close to it sit the raw modalities: a speech signal and an image, almost exactly what the sensor wrote down. Far from it sit the abstract modalities: a sentiment score, a list of detected objects, a set of object categories. Someone, or some program, had to compute those from the raw signal.

Language sits in the middle of the line. Written words were already shaped by a human mind, so they are further from a sensor than a photograph is. They are still closer than a sentiment score.

The slide ends with a sentence worth keeping: most of AI is about learning abstractions, or representations, from data. A representation is a list of numbers that keeps what matters about a thing and drops the rest. Learning to climb from the raw end of the line to the abstract end is the whole job.

Walk away from the sensor

Each dot is a kind of data from the lecture’s slide. Pick one. The chain below the line shows what must happen between the sensor and that data.

The order of the six on the line follows the lecture’s slide. The exact spacing and the chains of steps are our own illustration, to show the idea: each step is one computation after the sensor.

Notice what happens as you move right. Each step throws information away on purpose. A sentiment score is easy to use and says almost nothing about the voice it came from. A raw speech signal says everything and is hard to use. Every modality is a choice about where on this line to stand.

Why one design cannot swallow it all

Count what each item in the hospital record looks like to a computer. These sizes are illustrative, but they are the right order of size. A 224 × 224 colour X-ray is 224 × 224 × 3 = 150,528 numbers. A nurse’s note of 200 words, as a bag of words over a 50,000-word dictionary, is one vector of 50,000 numbers. A heart monitor at 250 samples a second records 900,000 numbers every hour. The blood table might hold 40 numbers.

The sizes already differ by a factor of more than 20,000. More important, the neighbours differ. In the X-ray, a pixel’s neighbours are the pixels above, below and beside it. In the note, a word’s neighbours are the words before and after it. In the blood table, row 7 has no special link to row 8. A model that assumes one kind of neighbour will use the wrong clues on the others.

Here is the usual failure, as symptom, measurement and fix. The symptom: someone glues all the numbers into one giant vector of about 1.1 million numbers and feeds it to a plain network. It trains slowly and learns shortcuts, such as the table values, while ignoring the image. The measurement: switch off the image input and watch the accuracy. If it barely drops, the model never learned to read the image. The fix is the plan of this lesson: give each kind of data a model that fits its shape, then combine the results. Combining is the topic of later lectures.

The plan, and a word on logistics

The lecture has two halves. The first half is about data: the main kinds of data, the properties that set them apart, the kinds of labels, and what it means for a model to learn something that holds up on new data. The second half is about models: a single recipe for building them, and then the special designs for sequences, images, sets and graphs. This lesson follows that order.

The slides also carry a few course notes. The first homework is a reading assignment plus a problem set, due on Tuesday, February 17. Project proposals are due on Tuesday, February 24, and groups have two to four students. Those notes belong to the class, so we leave them here.

The road

  1. Six kinds of data: what a model actually reads for each one.
  2. The modality profile: six properties that set kinds of data apart.
  3. Learning paradigms: supervised, unsupervised, reinforcement and mixtures.
  4. Generalising: fitting the training data versus new data.
  5. One recipe: representation blocks, aggregation blocks, a loss and gradients.
  6. Sets: shared function plus a sum. This is the hero you played at the top.
  7. Recurrence: a hidden state reused through time.
  8. Attention: dynamic weights, all at once.
  9. Space: convolution and image patches.
  10. Graphs: messages along edges.
  11. Field guide: numbers, terms, checklist, code.
Which statement about the line from sensor to abstract idea is right?

Chapter 1

Six Kinds of Data, Six Kinds of Input

Turn a photo, a sentence, a sound, a robot log, a table and a graph into the numbers a model actually reads

Hand a photo of a dog to a model. What does the model receive? Not a dog. Not even a picture. It receives a long list of numbers, and every kind of data turns into such a list in its own way. This chapter walks through the six ways the lecture shows, then two more.

Pictures: a number for every colour at every pixel

A digital photo is a grid of tiny squares called pixels. Each pixel stores how much red, green and blue light it holds. So each pixel is a short vector with d = 3 numbers (a vector is just an ordered list of numbers). Lay all the pixels end to end and you get one long vector, the input xi.

The task attached to the picture is the label, the answer we want. In the lecture’s first example the label yi is 0 or 1: is there a dog? That is binary classification. In the second example the label can be a duck, a cat, a dog, a pig or a bird, numbered 0, 1, 2, 3 and on. That is multi-class classification.

Words: a dictionary with one lit cell

A computer has no use for the letters in “lazy”. It needs a number. So we first fix a dictionary of all words the model knows, and give each word a position. The word “lazy” becomes a vector as long as the dictionary, with a 0 in every cell except a 1 in the cell for “lazy”. This is a one-hot vector: exactly one cell is hot.

One-hot vectors suit questions about a single word, such as: is it a noun or a verb, is its sentiment positive, is it the name of a person? The lecture calls these word-level tasks.

Now ask about a whole review: is it positive? Add up the one-hot vectors of all its words. The result counts how often each dictionary word appears. This is the bag-of-words vector, named for a bag you shake: the words are in, their order is not. The slide asks the sharp question: what happens with word ordering? The answer is that it is thrown away. “Dog bites man” and “man bites dog” give the same bag.

Sound: samples, windows and a spectrogram

A microphone measures air pressure many thousands of times a second. Each measurement is a sample. The slide gives the typical settings: 8 to 96 kHz for the sampling rate (how many samples per second), and 8, 16 or 24 bits for the bit depth (how finely each sample is rounded).

One sample says almost nothing, so we cut the signal into short windows of 20 ms that start every 10 ms, so neighbours overlap. Each window is turned into a column of numbers that say which pitches are loud. Stacked side by side, the columns form a spectrogram, a picture with time across and pitch up. Tasks on audio include recognising the spoken word, the emotion and the voice quality.

Build the model’s input

Choose a kind of data, then drag the size. The canvas shows the object the model reads, and the read-out counts its numbers.

200

Sizes are illustrative sliders. The rules are the lecture’s: 3 numbers per colour pixel, a one-hot vector as long as the dictionary, 20 ms windows every 10 ms, and T by 6 numbers for a force-torque log. The picture, the word ids and the traces are made-up toy data, seeded so they look the same on every visit. Sound uses 16 kHz, a rate inside the lecture’s 8 to 96 kHz range.

Two worked examples make the sizes concrete. Take the sentence “Fox jumps over the lazy dog” with a 50,000-word dictionary. As a bag of words it is one vector of 50,000 numbers, six of them equal to 1. As an ordered list of one-hot vectors it is six vectors, so 6 × 50,000 = 300,000 numbers. The ordered form keeps the sequence. The bag is six times smaller and forgets it.

Now take one second of speech at 16 kHz. That is 16,000 samples. Windows of 20 ms hold 320 samples each, and a new window starts every 10 ms, so the second yields 99 windows. After the spectrogram step, each window becomes one column. The model then reads a table of 99 columns, not 16,000 separate samples.

The failure to watch for is the mismatch of rates. If a robot’s touch log ticks 1,000 times a second and its camera gives 30 frames a second, a model that pairs them one to one will pair every frame with the wrong 33 ticks. The measurement is simple: divide the two rates and check that your pairing respects the ratio. The fix is to resample one stream or to pool the fast one over each frame’s time span.

Robot logs: a few numbers at every tick

A robot arm carries sensors that measure the arm itself. A force-torque sensor reports six numbers at each tick: pushes along three directions and twists around three. That is a time series of T ticks by 6 channels. Proprioception, the robot’s sense of its own body, adds the position and velocity of the hand: T by 2 more. The lecture’s example, from a paper on contact-rich tasks (Lee et al., ICRA 2019), predicts an object property and the next action from touch.

Tables, graphs and sets

Three kinds of data do not come as a grid or a line. A table has rows and named columns of mixed types: an age, a heart-rate trace, a yes or no for a ventilator. The lecture shows MIMIC-IV, a freely available set of hospital intensive-care records (Johnson et al., 2023), where the goal is to predict mortality and diagnosis codes.

A graph is a set of nodes joined by edges: people and friendships, atoms and bonds. Typical tasks are node classification (label each node) and link prediction (guess which edges are missing). Graphs also help other tasks: knowledge graphs for question answering, and social networks for sentiment analysis.

A set is a bag of items with no order and no links, such as the points of a 3D scan, called a point cloud. Tasks include set anomaly detection (which item does not belong), set expansion, set completion, and point cloud classification and generation.

A model reads a review as a bag-of-words vector. Which two reviews give it the same input?

Chapter 2

The Modality Profile

Measure any kind of data on six properties, and see which ones noise attacks first

Take one scene: a teacup on the right of a laptop in a clean room. You can photograph it. You can describe it in a sentence: “A teacup on the right of a laptop in a clean room.” Both say the same thing. Yet a model that handles the photo well may fail on the sentence, and the reverse. Why?

The lecture answers with a checklist. A modality profile is the set of qualities and structures that are unique to a kind of data. It has six parts. We take them one at a time, using the teacup scene for all six.

1. Element distribution: what the pieces are

Every kind of data is built from small pieces, its elements. In the sentence the elements are words. In the photo they are small patches of the picture. The first question is which values an element can take.

Words are discrete: each one comes from a fixed list, such as {teacup, right, laptop, clean, room}, and nothing lies between “laptop” and “room”. A patch of image is continuous: its brightness can be any value in a range. The lecture calls the set of possible values the support. A model for one kind of support does not simply work for the other.

2. Granularity: how often pieces appear

Granularity is the rate at which elements are sampled or appear. A photo might hold a few objects. Speech might carry about 150 words a minute (a common figure, used here only as an illustration). The lecture’s own examples are objects per image and words per minute. A fine-grained signal gives a model many pieces to relate. A coarse one gives few.

3. Structure: how pieces fit together

Structure is the way elements compose into the whole. The lecture lists four types: static, temporal, spatial and hierarchical. A sentence has a tree of phrases inside it (the slide draws a parse tree): “the right laptop” groups into smaller parts. A photo has a spatial layout, with neighbours next to neighbours. A recording has time order. Chapters 6 to 10 show that each structure asks for its own model design.

4. Information: how much is in there

How many surprises does the data hold? The lecture uses entropy, written H, as the measure. Entropy is the average surprise of an element. A coin that always lands heads gives no surprise, so H = 0. A fair coin is the most surprising a coin can be, so H = 1 bit.

H(X)
The entropy of X, in bits: the average surprise of one element.
∑x
Add up one term for every value x that an element can take.
p(x)
How likely value x is, a number from 0 to 1.
log2 p(x)
Zero for a sure thing, and a larger negative number the rarer x is. The minus sign out front makes each term positive.

Biased coin, heads 90% of the time: H = −(0.9 × log2 0.9 + 0.1 × log2 0.1) = 0.137 + 0.332 = 0.47 bits

The slide draws a big circle for the image’s entropy and a small one for the sentence’s. A photo holds many details a sentence leaves out. Density is the other half: how much of that is useful per element. Keep this in mind: entropy counts surprise, not meaning.

5. Noise: how the data goes wrong

Noise covers the natural imperfections of a modality: uncertainty, a poor signal-to-noise ratio (how loud the wanted part is compared with the unwanted part), and missing data. The slide shows a blurred photo, and a sentence with “teacup” typed as “teacip” and “right” as “rihj”. Noise looks different in each kind of data. Try it on the sentence.

Add noise to the caption

Drag the noise. Each letter is swapped for a random one with that chance. Red letters are changed ones. The bars count the words that survive and the entropy of the letters.

8%

The sentence is the lecture’s own. The swap pattern is a fixed random draw, so the same letters change at the same slider value on every visit. Entropy is computed live on the letter and space counts of the corrupted line, in bits per character. It measures spread, not meaning.

As the noise rises, fewer words survive. A word of length L survives with chance (1 − p)L, so long words fall first. Meanwhile the entropy drifts up, from 3.70 bits at 0% to 3.92 at 30%, though not at every step because this line is short. Random letters are surprising, but the surprise is useless. This is why the profile separates information from noise.

6. Relevance: useful for what?

The last part is relevance: how much of the data matters depends on the task and the context. For the question “is there a laptop?” the clean room is irrelevant. For “is the room tidy?” the laptop is irrelevant. The same photo holds two different answers, y1 and y2, on the slide. A model must learn which parts to ignore.

Relevance also explains a common failure. A team builds a model to flag a tidy room, trains it on photos where the tidy rooms happen to have bright light, and ships it. The symptom: it flags every bright room. The measurement: test it on dark tidy rooms and bright messy ones. The fix: collect data where the irrelevant property, brightness, varies independently of the label.

Comparing two modalities

The lecture closes the profile with a side-by-side of two made-up modalities, A and B. Row by row it compares element representations (discrete or continuous, and granularity), element distributions (density, frequency), structure (temporal, spatial, latent, explicit), information (abstraction, entropy), noise (uncertainty, missing data) and relevance (task and context dependence). When two modalities differ on many rows, joining them in one model is hard. Later lectures in the course take that on.

You add random typos to a sentence. What happens to the letter entropy and to the meaning a reader can recover?

Chapter 3

What the Data Is For: Learning Paradigms

Match the shape of your examples to the kind of learning they allow

You have collected a folder of data. Before you pick a model, one question decides almost everything: what kind of answer comes with each example? If every photo has a name tag, you can learn to read name tags. If nothing has a tag, you must learn something else. If the only feedback is a score after a long game, you must learn from the score.

The lecture draws three overlapping circles for this: supervised, unsupervised and reinforcement learning. It then adds a longer list. We use one notation for all of them. X is an input, such as a photo. y is a label, the answer we want for that input. A model is a function that takes X and tries to produce y.

Supervised learning: a tag on every example

In supervised learning each example comes with its label, and the model learns the map from X to y. The lecture gives three cases. Classify far-away, unseen images correctly. Predict how strongly a drug candidate binds to its target protein, learning from molecules that were already screened against that target. Learn machine translation from pairs of sentences, such as Spanish “Aquí tienes un bolígrafo” and English “Here’s a pen”.

The cost is the labels. Someone must write them, and that takes time and money.

Unsupervised and self-supervised: the data labels itself

Unsupervised learning uses no labels at all. It finds structure in X alone. The slide shows two uses: squeezing data into fewer dimensions or an embedding (a short vector for each item, as in the word embeddings of Mikolov et al., 2013), and studying 3D protein structures.

Self-supervised learning is the clever cousin. It makes its own label by hiding part of the input. The slide’s example is denoising diffusion over images: add noise to a photo, then train a model to remove it. The clean photo is the label, and it came free with the data. The same idea gives the arrow X to X′ on a later slide, where X′ is a changed version of X.

Reinforcement learning: only a score at the end

In reinforcement learning a model acts, sees the world change, and eventually receives a reward r, a number that says how well it did. There is no label for each step. The slide shows ChatGPT (trained in part from human feedback), a robot, and game players. The model must work out for itself which of its many steps earned the reward.

More ways to combine inputs and answers

The next two slides draw a family of small diagrams. Read each as: what goes in, and what the model must produce.

Read the paradigm

Pick a paradigm. The diagram shows its boxes: inputs on the left, the model in the middle, outputs on the right. Then drag how many examples you collected.

20,000

The diagrams follow the lecture’s slides 26 and 27. The label counts are illustrative: we assume 5% of examples get a hand label for the fine-tuning step, and that reinforcement learning gets one reward per run of 100 steps.

Transfer learning trains on one task (y1), then reuses the model for a second task (y2). Multitask learning trains one model for several outputs at once. Multimodal learning feeds two inputs, X1 and X2, into one model for one answer. Cross-modal learning learns with one input kind and carries what it learned over to another. Curriculum learning orders the labelled examples from easy to hard. Human-in-the-loop learning uses a person’s evaluation as the label, and LLM adaptation adds language to the label (an instruction).

The slide is honest about its limits: the list can be refined further, with active, semi-supervised, selective, contrastive, few-shot and inverse reinforcement learning. The main point is that the paradigm follows from what your data contains.

A worked choice

Say a clinic holds 20,000 chest X-rays. Radiologists have labelled only 400 of them as normal or abnormal. Which paradigm fits? Plain supervised learning would use just the 400 labelled images, and a model trained on so few overfits quickly (Chapter 4 shows how).

A better plan uses all three ideas above. First, train a model by self-supervised learning on all 20,000 images with no labels: hide patches of each X-ray, and ask the model to fill them in. The model learns what lungs and ribs look like. Second, take that model and fine-tune it on the 400 labelled images. This is transfer learning: the first task (fill in patches) is a stepping stone to the second (find disease). The counts in this plan are illustrative, but the shape matches the lecture’s slide 26.

The failure mode is label noise. The symptom: the model hits 99% on the labelled set but only 80% in the clinic. The measurement: have two radiologists label the same 100 images and count how often they disagree. If they disagree on 15 of them, no model should be expected to beat about 85% against one radiologist’s labels. The fix: write clearer labelling rules, use several labellers, and judge the model against their agreement.

You have a million photos and no tags, and you add noise to each one and train a model to restore it. Which paradigm is this?

Chapter 4

Fitting, Overfitting and Generalising

Watch a model memorise its training data, then fail on the first new example

A student memorises last year’s exam, answer by answer. On last year’s exam she scores 100%. On this year’s exam she scores far lower, because she learned the answers and not the subject. A model can fail in exactly the same way.

The learning process

The lecture frames learning as labelling. We want a good way to label our data. Data goes in as input, a label comes out, and in between sits a model that we learn. We never search all possible functions. We pick a model class, a family of functions such as straight lines, or small neural networks, and search inside it.

One sentence on the slide is boxed in red, and it is the key: how well our model labels new data depends largely on how good the chosen model class is. A class that is too small cannot bend to the real pattern. A class that is too big can bend to everything, including the accidents.

Underfitting and overfitting

What we really want is to generalise: to label future data well, not just the data we already hold. The lecture names two ways to fail. If the model does not capture the relationship between input and output, it is underfitting. If it is too specialised to its training data, it is overfitting.

The cure is a discipline, not a trick. Split the data you collected into three parts. The training set is what the model learns from. The validation set is what you use to choose between models and settings. The test set is locked away until the end, and tells you how the final model will do on new data.

One more line from the slide matters in practice: make sure the test data conditions match the real-time deployment conditions. A test set from a quiet lab says little about a noisy factory.

Fit it, then test it

Twelve blue points are the training data. The warm curve is a polynomial of the degree you choose. Purple points are new data the model never saw. Drag the degree from 0 to 9, then try New conditions.

3

Toy data: the truth is sin(2.4 x) with noise of 0.15, 12 training points and 60 test points, drawn from a fixed seed. The curve is a least-squares polynomial fit computed live in your browser. New conditions moves the test points from the training range (x from −1 to 1) to a range just outside it (x from 0.5 to 1.5). Errors are mean squared error.

Slide the degree up. At degree 0 the curve is a flat line: it underfits, and both errors are large (0.557 on training, 0.693 on new data). At degree 3 the curve follows the wave, and the two errors are close (0.018 and 0.023). That is good generalisation.

At degree 9 the curve passes almost through every training point, so the training error is tiny (0.0014). But it swings between the points, and the error on new data is 0.145, more than a hundred times larger. The model has memorised the noise. The gap between the two errors is the sign of overfitting.

Now press New conditions. Degree 3 still copes (0.363). Degree 9 explodes (about 20,692), because the test points lie where the curve was never pinned down. The gap is not only about the model. It is also about whether the future looks like the past.

Cleaning, with numbers

Step 2 of the checklist says to normalise or standardise. Here is why. Suppose three patients have heart rates of 60, 80 and 100 beats a minute, and their blood sugar is 5.0, 5.2 and 5.4. A model that adds these numbers up is dominated by the heart rate, simply because its numbers are bigger.

Standardising fixes that. For each column, subtract the mean and divide by the standard deviation. The heart rates have mean 80 and standard deviation 16.3, so they become −1.22, 0 and 1.22. The sugars have mean 5.2 and standard deviation 0.16, so they become the same −1.22, 0, 1.22. Now both columns speak in the same units.

An outlier is a value far from the rest, such as a heart rate of 600 from a loose sensor. After standardising, a value many standard deviations out is a flag to inspect. The lecture’s step 3 then asks you to look: plot the data, shrink it to two dimensions with PCA or t-SNE, and see whether clusters appear.

One failure here is subtle: leakage. The symptom: your test score is great, and the deployed score is poor. The cause is often that you standardised or chose a degree using the whole data set, test points included, so information from the test set leaked into training. The fix: split first. For 1,000 examples that might be 700 for training, 150 for validation and 150 for test. Compute the means and the choice of degree on the first two parts only, and touch the test part once at the end.

How to data: five steps

The lecture ends its data half with a checklist. We will extend it in the last chapter.

  1. Decide how much data to collect, and how much to label. Both cost time and money.
  2. Clean it. Normalise or standardise values, find noisy data, and look for anomalies and outliers.
  3. Look at it. Plot it, reduce its dimensions (PCA, t-SNE), and run a cluster analysis.
  4. Decide the evaluation metric. Use a proxy and a real one, quantitative and qualitative.
  5. Choose a model class and a learning algorithm. The second half of the lecture covers this.
A model scores 99% on its training set and 62% on its test set. What is the most likely cause, and what is the first thing to try?

Chapter 5

One Recipe for Every Model

Build any deep model from two kinds of block, then train it with the same four steps

You now know your data: its kind, its profile, its labels. A friend asks which model you will use. A cook faces the same choice: bake from a family recipe, or buy a ready meal and add salt? Each is right in some kitchens. This chapter gives the lecture’s answer in two parts: where to stand on the choice, and the one recipe that nearly every deep model follows.

From custom to off the shelf

The lecture draws a line with two ends. On the left is the domain-specific end: custom neural nets built for your data, then custom training objectives. In the middle sit models taken off the shelf and then fine-tuned or adapted. On the right is the general-purpose end: an off-the-shelf large language model used as it is. The slide adds one sentence: your decision will depend on many factors.

What are those factors? The next slide gives a list for “What is a good model?” A good model captures the right semantic information, at the right granularity, using an appropriate amount of data and labels, within the right resource constraints, and with the right level of usability, such as explainability and accessibility. The list says “and more”. No model wins on every line.

A small team with 500 labelled examples and a laptop will usually start toward the right of the line. A team with a unique sensor and millions of examples may build toward the left. Either way, the rest of this lesson tells you what is inside both ends, because the parts are the same.

Two jobs: learn representations, combine them

The lecture’s unified view says every deep model does two jobs. The first is learning representations. A model takes an input, such as a photo of a plane, and maps it to a vector of numbers that keeps what matters. The second is combining representations, also called information aggregation: when the data has many elements, the model must merge their vectors into one answer.

Think of a committee. Each member first forms a private opinion about one document (job one). Then the chair collects the opinions into one decision (job two). Different kinds of data change how members share notes and how the chair collects them. They do not change the two jobs.

The recipe in four steps

Slide 35 turns the view into a recipe: compose differentiable functions and training objectives. A function is differentiable if we can ask how its output changes when we nudge its inputs. That is what lets a computer learn.

  1. Pick representation blocks for each element. The lecture lists fully-connected layers, ReLU, layer norm, convolution and self-attention.
  2. Pick aggregation blocks. The list: concatenate, cross-attention, add, max.
  3. Compute a loss function. A loss is one number that says how wrong the model is.
  4. Take gradients and update with stochastic gradient descent (SGD). The gradient says which way to nudge each weight to lower the loss, and SGD takes a small step that way, again and again.

The slide’s picture stacks these like blocks: a fully-connected layer feeding a convolution and a self-attention block, joined by concatenation and add, with a final add producing y. The rest of the lecture is a tour of which blocks suit which data.

Step 4, by hand

Use the smallest model: a line, ŷ = w · x + b, on four points x = 1, 2, 3, 4 with labels 2.1, 3.9, 6.2, 7.8. Start at w = 0 and b = 0. The loss is the mean of the squared errors. Every prediction is 0, so the loss is (4.41 + 15.21 + 38.44 + 60.84) / 4 = 29.725.

The gradient with respect to w is the mean of 2 × (error) × x. The errors are −2.1, −3.9, −6.2, −7.8, so it is 0.5 × (−2.1 − 7.8 − 18.6 − 31.2) = −29.85. For b it is 0.5 × (−2.1 − 3.9 − 6.2 − 7.8) = −10.0. A learning rate of 0.05 moves each weight by 0.05 times minus its gradient: w becomes 1.4925 and b becomes 0.5. Press the button to see it.

Take the gradient step

Press Step to run one round of the recipe: predict, compute the loss, take the gradient, update. The top plot shows the line. The bottom plot shows the loss on a log scale. Then raise the learning rate.

0.05

Toy data we chose: x = 1, 2, 3, 4 and y = 2.1, 3.9, 6.2, 7.8. The arithmetic is exact, with full-batch gradient descent (the same update that stochastic gradient descent makes on one random batch). For this data a learning rate above about 0.12 makes the loss grow.

Now the failure. Set the rate near 0.15 and run ten steps. The symptom: the line swings wildly and the loss climbs instead of falling. The measurement: the loss after step 10 is larger than the loss at the start. The fix: lower the learning rate until each step shrinks the loss. Too big a step overshoots the valley and lands higher on the other side.

In the lecture’s recipe, what does the aggregation block do?

Chapter 6

Sets: When Order Must Not Matter

Share one function across items and pool with a sum, and the order stops mattering

Five photos of faces lie on a table. Four are of one person, and one is of someone else. Your job: find the odd one. You do not care which photo you picked up first. Swap the order of the pile and the odd face is still the odd face. A model for this job must have the same habit. The data is a set: a collection of items with no first and no last.

The lecture uses this as its simple classification example. Sets also appear as point clouds, the thousands of 3D points a scanner records on a chair, and the tasks are set anomaly detection, set expansion, set completion, point cloud classification and point cloud generation.

The two rules for set models

The slide states the rule: models for set-based data must be invariant to element order. Invariant means the output does not change when the input changes in a certain way, here when the items are reordered. Two design choices give that for free.

  1. Parameter sharing for each set element. Every item goes through the same function φ with the same weights.
  2. A permutation-invariant aggregation function. A permutation is a reordering. The aggregation, such as a sum, gives the same result for every reordering.

A function ρ then reads the pooled vector and produces the answer y. So the whole model is y = ρ(φ(A) + φ(B) + φ(C) + φ(D) + φ(E)). The lecture’s remark: giving ABDCE also gives ECBDA, BCAED, and every other order.

f(X)
The model’s answer for the whole set X.
ρ
A small network that reads the pooled vector and writes the answer.
∑x ∈ X
Add up one vector for every item x in the set. Addition does not care about order.
φ(x)
A small network that turns one item into a short vector. The same weights are used for every item.

What breaks each rule

The lecture shows both failures. In the first, the model has no parameter sharing: slot 1 has its own φ1, slot 2 has φ2, and so on. The model now treats “item in slot 1” and “item in slot 5” differently. To learn that ABDCE and ECBDA mean the same, it must see both. The slide’s count: 5! more samples are needed, which is 5 × 4 × 3 × 2 × 1 = 120.

In the second failure, the model shares φ but joins the vectors by concatenation (gluing them end to end). The glued vector has slots, so order matters again, and again 120 times the data are needed. The device tests every wiring against all 120 orderings.

Try every ordering

Five items A to E go through a small network. Pick how they are pooled, and whether every item shares one φ. Press Shuffle for one new order, or Test all 120 to sweep through every order. The dots show every ordering’s answer; the teal line marks the current one.

Toy network with fixed random weights: each item is two numbers, φ maps them to three with a tanh, and ρ is a weighted sum of the pooled numbers followed by tanh. Two answers count as the same if they agree to nine decimal places. Parameter counts are for φ only.

Run it. Sum, mean and max with a shared φ give 1 distinct answer over 120 orderings. Switch to Concat, or turn off sharing, and the number of distinct answers jumps to 120 (one case is odd: max without sharing gives 12, because the max throws away most slots). That is the failure in numbers: a model that does not respect the symmetry sees 120 different problems where there is one. Sharing also helps in a second way: one φ has 9 numbers, five separate ones have 45.

Shapes and a failure mode

Follow the data through the model in the lecture’s terms. A set of n items, each with d numbers, is a table X of shape n × d. The function φ maps every row to h numbers, giving n × h. The sum collapses the n rows into one vector of h numbers. Then ρ maps those h numbers to the answer y. A point cloud with n = 1,024 points and d = 3 (x, y, z) works the same way as one with 10,000 points, because n disappears at the sum.

That last fact hides a trap. A sum grows with n: a set of 10,000 items gives a pooled vector about ten times larger than a set of 1,000, even if the items look alike. The symptom: the model is accurate on sets near the training size and wrong on much larger ones. The measurement: feed the same items twice (n doubles) and check whether the answer should have changed. The fix is to use a mean or a max in place of a sum. PointNet (Qi et al., CVPR 2017), the other reading cited on the slides, pools with a max.

Invariant versus equivariant

The next two slides, from the survey on geometric deep learning by Bronstein and colleagues (2021), give a precise word for the idea. A symmetry is a change that leaves what matters unchanged. A function f is invariant to a change if applying it to the input leaves the output untouched: shift a sunset photo to the left and the label “sunset” stays. The slide’s image classification example is invariant.

A function is equivariant if the change on the input shows up as the same change on the output. For image segmentation, which colours every pixel by what it shows, shifting the photo shifts the coloured map by the same amount. Choose invariance when you want one answer for the whole thing. Choose equivariance when you want one answer for every part.

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (Bronstein, Bruna, Cohen and Veličković, 2021) is the reading behind this idea. It organises most architectures in this lecture as one blueprint: find the symmetry of the data, then build layers that respect it. Deep Sets (Zaheer et al., 2017) gives the set case its proof. It shows that a function over sets that ignores order can be written as ρ of a sum of φ, under the conditions the paper states. That is the exact shape you just tested.

A set model shares φ across items but joins the item vectors by concatenation. What goes wrong?

Chapter 7

Sequences: One Brain Reused Through Time

Read words one at a time with a shared hidden state, and measure how fast the first word fades

You read a sentence one word at a time: “Fox jumps over the lazy…” What comes next? You do not look at “lazy” alone. You also remember “fox” and “jumps”, and your guess is “dog”. Your memory carries the story forward. This chapter builds that memory into a model, and then measures where it fails.

From one element to a sequence

The lecture starts small. One element, the word “lazy” (Xi), can be classified on its own: is its sentiment positive? That is a single word, and the earlier tools are enough.

Now the task is to predict the next word given the whole phrase. The slides keep asking one question: how do we aggregate information? We must combine “Fox”, “jumps”, “over”, “the” and “lazy” into one answer, and the answer depends on the order. The lecture gives two rules for sequence models. The model must be invariant to time: the rule it uses at step 3 is the rule it uses at step 300, so the weights are shared across time steps. And it must be equivariant to word order: shuffle the words and the output shuffles with them, so order is respected and never lost.

The recurrent network

A recurrent neural network (RNN) reads one word at a time and keeps a short vector of numbers called the hidden state, written h. The hidden state is the model’s memory of everything read so far. At every step, the model combines the new word with the old memory to make the new memory.

x(t)
The word read at step t, as a vector.
U
A weight matrix that turns the new word into a contribution to memory.
h(t−1)
The memory from the step before.
W
A weight matrix that decides how much of the old memory carries over.
h(t)
The new memory. The tanh squashes every number into the range −1 to 1.
z(t)
The scores for the answer at step t, computed from the memory by a third matrix V.

The slides build this in three pictures. First a plain feedforward network: x to h to z to the loss L(t) = −log P(Y = y(t) | z(t)). Second the same network with the loop W added, so that h feeds itself. Third the loop unrolled in time: one copy of the network per word, joined by W, and a total loss L = ∑t L(t). The key line sits under the last picture: the same model parameters are used for all time steps.

The data flow is simple. Take words with vectors of length 300, a memory of length 128, and 10,000 possible answers. U is 128 × 300, W is 128 × 128 and V is 10,000 × 128. Those matrices are the same at step 1 and at step 1,000. So the model is not bigger for a longer sentence.

Generation and sequence to sequence

The same loop can write. In sequence generation the model feeds its own last output y(t−1) back in as the next input: h(t) = tanh(U y(t−1) + W h(t−1)). Start with “Fox”, and out comes a sentence, or a tune. The slide lists the modern versions of the same idea: the RNN, then the LSTM, the temporal convolutional network (TCN) and state space models.

A sequence-to-sequence model uses two such loops. The first reads “Fox jumps over” and ends in one memory vector. The second starts from that memory and writes the output words Y1, Y2, Y3. The slide’s example is machine translation, and it calls the next step the birth of attention-based models.

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho and Bengio, 2014) is that step. Squeezing a whole sentence into one memory vector was a bottleneck, so the paper lets the writer look back over all the reader’s memories and choose which to use at each output word. The picture on the slide is the result: a grid that shows which input word each output word looked at. Chapter 8 does the same with a few lines of arithmetic.

Measuring the fading memory

There is a catch. To train the loop, the gradient must travel back through time, one step at a time. At each step it is multiplied by a factor. If the factors are below 1, a word seen long ago has almost no influence on the answer now. Try it with the smallest RNN: one number of memory.

Hear the first word

A one-number RNN reads T random words. Each bar shows how much a nudge to that word changes the final memory, on a log scale. Drag the length and the carry-over weight W.

12
0.80

Toy model: h = tanh(W · hprevious + x) with U = 1, a fixed random list of inputs between −1 and 1, and h starting at 0. The influence of word t is the exact derivative of the final h with respect to xt, computed by the chain rule. The same two weights are used at every step, so the model size stays 2 numbers whatever T is.

Set T to 12 and W to 0.8. The first word influences the final memory only 0.0024 times as much as the last word. Now push T to 30. The ratio falls to about 4 × 10−7. That is the vanishing gradient: the symptom is a model that forgets the start of long inputs. The measurement is the ratio of the first bar to the last. The fixes, as the lecture’s list of modern versions hints, are better memory cells (the LSTM), convolutions over time (the TCN) and state space models. Chapter 8 shows a different fix, which skips the chain completely.

Why can a recurrent network read a sentence of any length without growing?

Chapter 8

Attention: Dynamic Weights for Every Word

Let each word choose how much to listen to every other word, all at once

In a meeting of five people, you do not give everyone equal weight at every moment. When the budget comes up, you listen mostly to the finance lead. When the schedule comes up, you listen to the planner. Your weights change with the topic. That is the whole idea of attention: each word decides, for itself, how much to listen to every other word.

A recurrent network (Chapter 7) must pass everything through one memory, one step at a time. Attention drops the chain. Slide 52 calls this the birth of attention-based models and the idea of dynamic weights for different elements.

One word, five weights

The slide shows the sentence “Fox jumps over the lazy” and focuses on the last word, number 5. The new representation of “lazy” is a weighted mix of the vectors of all five words.

y = a15 v1 + a25 v2 + a35 v3 + a45 v4 + a55 v5

Each vj is a value vector, the content word j offers. Each aj5 is an attention weight, a number between 0 and 1, and the five of them add up to 1. A big a25 says “word 5 listens closely to word 2”. The weights are not fixed in the model. They are computed fresh for every sentence from the words themselves.

Where the weights come from

Every word is turned into three vectors by three learned matrices. The query is what the word is looking for. The key is what the word advertises about itself. The value is what it hands over if chosen. Word 5 scores every word by matching its query against that word’s key, using a dot product (multiply matching numbers, then add). A high score means a good match.

Then the scores are turned into weights with the softmax: make every score positive with the exponential function, then divide by the total so the five weights add up to 1. The lecture writes the whole thing for all words at once:

XWq
All queries at once. X holds the T words as rows (a T × d table), and Wq is a learned d × d matrix.
WkTXT
All keys, turned on their side (d × T). Multiplying queries by keys gives a T × T table of match scores.
√d
A scale factor. It keeps the scores from growing with the vector width d, which would make the softmax too sharp.
XWv
All values (T × d). The softmax weights, a T × T table with each row adding to 1, mix them into the new representation h.
h
The new T × d representation: every word now carries a blend of the others.

Shapes make it concrete: with T = 5 words and d = 512, X is 5 × 512, the score table is 5 × 5, and h is 5 × 512. The slide’s small picture draws the same table for “fox” and “jumps”, with weights 0.7 and 0.3 in one row and 1.0 in the next. Each row adds up to 1.

Work one row by hand. Say the word “lazy” scores the five words 0.72, 0.89, 0.28, −0.11 and −0.21. Take e to the power of each score: 2.05, 2.44, 1.32, 0.90 and 0.81. They total 7.52. Divide each by 7.52, and the weights are 0.27, 0.32, 0.18, 0.12 and 0.11. They add up to 1. The largest score gets the largest weight, but no word gets zero.

Attention Is All You Need (Vaswani et al., 2017) is the paper behind this slide. It drops the recurrent loop entirely and builds the whole model from attention. It also runs several attention computations side by side, called heads, so that a word can listen for different things at once. The lecture keeps only the single-head formula above.

Who does each word listen to?

Pick the word that is asking. The arcs show its five weights; thicker means more attention. The grid shows every word’s weights at once, one row per word.

Toy numbers we chose: every word has a 2-number query, key and value, so the scores are computed for real (dot product over √2, then softmax) but the learned meaning is not. A trained model learns these vectors from data.

Choose “lazy”. It gives about 0.32 to “jumps” and 0.27 to “Fox”, and 0.11 to itself. Its new vector is that mix of five values. Then look at the grid: all five rows were computed together, in one matrix multiplication. Nothing waits for the word before it.

Why this beat the loop

The slide gives the two rules again. Parameter sharing across time steps: the same Wq, Wk and Wv serve every word. And information aggregation over time in parallel: all T words are processed in one go. In Chapter 7 the first word reached the last through 29 multiplications when T = 30, and faded. Here every word reaches every word in one step.

There is a price. The score table has T × T cells, so doubling the sentence length quadruples the table. Real models pay it for the speed of running in parallel and the short path between words.

In self-attention, where do the weights a15, a25, … a55 for word 5 come from?

Chapter 9

Space: Sliding Filters and Patches

Reuse one small filter at every position, and count the weights you save

Someone shows you a photo and asks: is there a fox? The fox might sit in the top left, or the bottom right, or fill the whole frame. You find it anyway, because you look for the same shape wherever you look. A model for pictures needs that same habit. Move the fox, and the answer should not change.

The lecture states the rule in one line: models for spatial data need to be invariant to spatial translations. A translation is a shift of the picture left, right, up or down. The data is a grid, and the grid has a symmetry: sliding it keeps the content.

The expensive way: connect every pixel to every output

The most direct model connects every input pixel to every output. The slide counts the cost. A 200 × 200 image has 40,000 pixels. Each output needs a weight for each of them, so the layer needs 40,000 × n parameters, where n is the size of the output. That is not efficient.

It is also wrong in a second way. Each position has its own weights. A detector that learned to see a fox at the top left has learned nothing about a fox at the bottom right. The slide says it plainly: it may learn different outputs for different pixel positions, so it is not spatially invariant.

Two modifications give the convolution

The lecture fixes both problems in two moves. Modification 1 is to apply the filter only to a small sliding window, for efficiency and locality. A filter is a tiny list of weights, such as w1, w2, w3. An output looks only at the few pixels near it, which is where the useful clues are.

Modification 2 is to apply the same filter in every window. That is parameter sharing again, as in the set model of Chapter 6, now across positions instead of across items. The slide draws the three weights in red, blue and green, repeated at every window. The result is the convolution. Its output, the response map, says how strongly the filter matched at each place.

In two dimensions the filter is a k × k grid, say 3 × 3 = 9 weights. For the 200 × 200 image with an output map of the same size, the dense layer needs 40,000 × 40,000 = 1.6 billion weights. The convolution needs 9.

Then comes the second rule on the slide: information aggregation over a k × k pooling region. Pooling replaces each little region of the response map with one number, such as the largest. The fox, wherever it was, leaves a strong response, and the maximum still sees it.

Move the pattern

The left grid is an 8 × 8 image holding a small plus. The right grid is the response map of one shared 3 × 3 filter shaped like that plus. Move the plus with the sliders. A fixed template that learned the plus at one position is compared too.

2
2

Toy 8 × 8 image of ones and zeros. The filter has the plus shape, with 5 ones, so a perfect match scores 5. The fixed template is the whole image as it looked with the plus at position (2, 2), 64 weights per output. Both are real dot products computed live; neither is trained.

Drag the plus to (4, 1). The response map’s bright cell moves with it: the map is equivariant, shifted as the input was. The pooled value, the largest cell, stays at 5: that is invariance. The fixed template, which had learned only position (2, 2), falls from 5 to near 0. The shared filter bought invariance with 9 numbers, and the template needed 64 per output.

Here is the data flow with shapes. An input of H × W pixels and one filter of k × k, sliding one pixel at a time with no padding, gives a response map of (H − k + 1) × (W − k + 1). A real layer has several input channels and several filters. With 3 input channels (red, green, blue), 64 filters and k = 3, the layer has 3 × 3 × 3 × 64 = 1,728 weights, whatever the image size, and it makes 64 response maps. Each map answers one question: “where is this pattern?”

Why do small windows work for images? Because nearby pixels are linked and far-away pixels mostly are not. An edge, a corner or a patch of fur lives inside a few pixels. The window captures the local clue, and the shared weights apply it everywhere. This is a fact about pictures, not about all data. For a table of unrelated columns, a window would link the wrong things.

Stacking layers: from edges to objects

One filter finds one pattern. The slide stacks many convolution layers and many pooling layers. Multiple convolutional layers let the network learn combinations of sub-parts and so grow in complexity: pixels combine into edges and blobs, edges into parts, parts into objects. Multiple pooling layers let it learn increasingly abstract and summarised information. Each layer reads a bigger region of the original image than the layer before it.

Pooling has a cost, and you should know it. It throws away exact position. After several poolings the network knows a fox is somewhere, not where. For tasks that need place, such as drawing a mask around the fox, use the equivariant map before pooling (as Chapter 6 said for segmentation).

The Vision Transformer

The same two rules can be met with attention. Slide 65 states them for the Vision Transformer: parameter sharing across a k × k self-attention region, and information aggregation over a k × k patch region. The image is cut into square patches, and each patch is treated like a word in Chapter 8.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Dosovitskiy et al., ICLR 2021) is the reading. It cuts a picture into 16 × 16 pixel patches, turns each patch into a vector with one linear layer, adds a number for each patch’s position, and feeds the sequence to a standard transformer. A 224 × 224 image gives 14 × 14 = 196 patches. The convolution gave position-sharing for free. Here the model must learn more from data, and the paper shows it matches or beats strong convolutional networks when pretrained on enough images.

Why does applying the same filter at every window of an image give invariance to translation?

Chapter 10

Graphs: Neighbours Talk to Neighbours

Pass messages along edges, and see sets, sequences and grids as special graphs

A chemist hands you a drawing: six atoms in a ring, one more atom hanging off the side, and lines for the bonds. What molecule is this? You answer by looking at how the atoms connect. An atom that touches a nitrogen means something different from one that touches only carbon. A model for this needs to read who touches whom.

That drawing is a graph: dots called nodes, and lines between them called edges. The lecture shows three kinds of task on graphs. Node classification asks what each node is. Link prediction asks which edges are missing. Graph classification asks one question about the whole graph, such as the molecule’s identity.

The same two rules, again

The lecture answers the aggregation question once more. For graphs the two rules are:

  1. Parameter sharing across nodes. Every node runs the same small function φ on its own features.
  2. Information aggregation over neighbours. A node gathers the results of the nodes it touches along edges.

That is a round of message passing. Every node sends its vector to each neighbour, and each node adds up what it receives. The slide draws it for the node A, which has neighbours B, C, D and E. Two “Aggregate” boxes sit side by side, because the model repeats the round.

Why repeat? After one round, a node knows about its direct neighbours. After two rounds, it has heard from its neighbours’ neighbours, because they heard from theirs in round one. After r rounds, a node knows its r-hop neighbourhood. This is the graph version of the growing region seen by stacked convolution layers in Chapter 9.

Sets, sequences and grids are graphs too

Slides 69 and 70 close the circle. A set is a graph with only nodes and no edges: sharing φ across nodes is the set rule of Chapter 6, and aggregation over neighbours has nothing to do, so the model pools all nodes at the end, with a sum. The lecture cites Deep Sets and the Set Transformer for this.

A sequence is a chain graph: each word touches the one before and the one after. A picture is a grid graph: each pixel touches its four neighbours. Both are graphs where the connection pattern is fixed and regular, and the lecture credits the geometric deep learning survey for this view. That is why convolutions and recurrences are special cases of message passing, and why the one blueprint of “share the function, aggregate over the right neighbours” covers the whole lecture.

Pass the messages

Choose a structure. Tap a node to pick it. Then drag the rounds. Teal nodes are those whose information has reached your node after that many rounds.

2

Toy graphs: a six-atom ring with one side atom (7 nodes, 7 edges), a six-word chain (6 nodes, 5 edges), a 3 × 3 pixel grid (9 nodes, 12 edges) and five set items with no edges. A round sends one message each way along every edge. The reach is a breadth-first search, computed live.

Pick the middle of the sequence and use 2 rounds: the node has heard from 5 of 6 words, two on each side. Switch to the grid, tap the centre and use 1 round: it has heard from its 4 neighbours and itself. Switch to the set. Whatever you do, the reach stays at 1: there are no edges, so there is nothing to pass. A set model must pool at the end, as Chapter 6 did.

The symptom of too many rounds is the opposite: in a small, dense graph every node ends up hearing from every node, so all the node vectors blur into one. The measurement is the reach count in the read-out; when it equals the node count, extra rounds add no new information. The fix is fewer rounds, or attention (below) to choose whom to listen to.

Work a tiny case by hand to see the blur. Take three nodes in a line, A, B and C, with starting values 1, 2 and 4. Let each node replace its value by the average of itself and its neighbours. After round 1, A is (1 + 2) / 2 = 1.5, B is (1 + 2 + 4) / 3 = 2.33 and C is (2 + 4) / 2 = 3. After round 2, A is 1.92, B is 2.28 and C is 2.67. After round 3 they are 2.10, 2.29 and 2.48.

Look at the gap between the biggest and the smallest value: 3 at the start, then 1.5, 0.75 and 0.375. It halves every round. Run enough rounds and all three nodes hold nearly the same number, and the model can no longer tell them apart. This is the blur, usually called oversmoothing.

Attention on a graph

Plain message passing adds neighbours up with equal weight. Graph Attention Networks (Veličković et al., ICLR 2018) borrow the idea of Chapter 8: each node computes a score for each neighbour from the two feature vectors, turns the scores into weights with a softmax over its neighbours only, and takes the weighted mix. Because attention is limited to the edges, the graph’s structure is kept. The slide also cites Graph Transformer Networks (Yun et al., NeurIPS 2019) for the same family of models.

Which statement matches the lecture’s view of sets and graphs?

Chapter 11

Field Guide

Keep the numbers, the terms, the checklist and the core code on one page

The whole lecture fits in one idea. Different data come in different shapes: sets, sequences, grids, graphs. Each shape has a symmetry, a change that must not change the answer. A good model builds that symmetry in with two moves: share the same function across the elements, and aggregate over the right neighbours. Do that, and you need far less data. Go back to the hero at the top and scramble a set, shift a signal and relabel a graph to see it again.

The numbers to remember

QuantityValueWhere from
Numbers per colour pixel3 (R, G, B)Slide 5, Chapter 1
A 200 × 200 colour image120,000 numbers3 × 200 × 200, Chapter 1
Audio sampling rate and bit depth8 to 96 kHz; 8, 16 or 24 bitsSlide 9, Chapter 1
Audio window and offset20 ms and 10 msSlide 9, Chapter 1
Force-torque and proprioception logsT × 6 and T × 2Slide 11, Chapter 1
Parts of the modality profile6Slides 16 to 21, Chapter 2
Orderings of five set items5! = 120Slides 38 and 39, Chapter 6
Dense layer on a 200 × 200 image40,000 × n weightsSlide 57, Chapter 9
A 3 × 3 convolution filter9 weights, any image sizeSlides 58 to 60, Chapter 9
Steps run one after anotherRNN: T; attention: 1Slides 49 and 54, Chapters 7 and 8

The terms, in one line each

TermPlain meaningChapter
ModalityThe way something is expressed or perceived. Raw is near the sensor; abstract is far from it.0
Modality profileElement distribution, granularity, structure, information, noise, relevance.2
Self-supervisedThe data supplies its own label by hiding part of itself.3
OverfittingLow error on training data, high error on new data.4
InvariantThe output stays the same when the input changes in a set way.6
EquivariantThe output changes in the same way as the input.6
Hidden stateThe fixed-size memory a recurrent network carries from step to step.7
Attention weightHow much one word listens to another, computed from query and key, adding to 1.8
Filter and poolingA small shared pattern slid over the data, then a summary of each region.9
Message passingEach node gathers the vectors of its neighbours along edges.10

The checklist: how to model

Slide 71 extends the data checklist from Chapter 4 into eight steps. Use it as a working list.

  1. Decide how much data to collect and how much to label, given cost and time.
  2. Clean the data. Normalise or standardise, find noisy data, detect anomalies and outliers.
  3. Visualise it. Plot it, reduce dimensions (PCA, t-SNE), run a cluster analysis.
  4. Decide the evaluation metric. Proxy and real, quantitative and qualitative.
  5. Choose the modelling paradigm: domain-specific or general-purpose.
  6. Work out the base elements and how to represent them.
  7. Work out the invariances and equivariances, and the other parts of the modality profile.
  8. Iterate between data collection, model design, training and tuning until satisfied.

The core as code

Five functions hold the lecture’s five architectures. Each one is the pair of rules from this lesson: share the function, aggregate over the right neighbours. The assertions at the end check the symmetries.

pythonimport numpy as np

def deep_set(X, phi, rho):               # X: (n, d). Share phi, pool with a sum.
    return rho(phi(X).sum(axis=0))        # same answer for every row order

def rnn(xs, U, W, V):                     # xs: (T, d). The same U, W, V at every step.
    h, zs = np.zeros(W.shape[0]), []
    for x in xs:
        h = np.tanh(U @ x + W @ h)           # new memory from new word + old memory
        zs.append(V @ h)
    return np.array(zs)

def attention(X, Wq, Wk, Wv):             # X: (T, d). All words at once.
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(K.shape[1])     # (T, T) match scores
    A = np.exp(S - S.max(1, keepdims=True))
    A /= A.sum(1, keepdims=True)              # softmax: each row adds to 1
    return A @ V, A                        # (T, d_v) new representation

def conv2d(img, w):                       # img: (H, W), w: (k, k). One shared filter.
    k = w.shape[0]; H, W_ = img.shape
    out = np.zeros((H - k + 1, W_ - k + 1))
    for i in range(H - k + 1):
        for j in range(W_ - k + 1):
            out[i, j] = (img[i:i+k, j:j+k] * w).sum()
    return out                             # response map; its .max() is the pooled value

def message_pass(H, adj, W):              # H: (n, d), adj: (n, n) of 0 and 1
    return np.tanh((H + adj @ H) @ W)       # self + sum over neighbours, shared W

# the symmetry checks
r = np.random.default_rng(0)
X = r.normal(size=(5, 2)); Wp = r.normal(size=(2, 3))
phi, rho = lambda X: np.tanh(X @ Wp), lambda v: np.tanh(v.sum())
p = r.permutation(5)
assert np.isclose(deep_set(X, phi, rho), deep_set(X[p], phi, rho))   # invariant
H, adj, W = r.normal(size=(3, 4)), np.array([[0,1,0],[1,0,1],[0,1,0]]), r.normal(size=(4, 4))
Pm = np.array([[0,0,1],[0,1,0],[1,0,0]])               # relabel the nodes
assert np.allclose(message_pass(Pm @ H, Pm @ adj @ Pm.T, W), Pm @ message_pass(H, adj, W))  # equivariant

Where it sits in the field

The lecture opens the course’s study of data. Next comes the question of how to combine modalities: Lecture 5, on fusion, picks up from here, and Lecture 3, the AI tutorial comes just before. The six readings of this session are the primary sources for the blueprint above.

Each reading has a home in the series or the catalogue:

For more depth on each architecture, see recurrent networks and GRUs, the transformer, convolutional networks, the Vision Transformer and graph neural networks. For the generalisation story, see bias and variance. For a set model in practice, see the PointNet paper lesson.

The takeaway. Look at your data before you pick a model. Find its kind, its profile and its symmetry. Then build a model that shares one function across the elements and aggregates over the right neighbours. That choice, not a bigger network, is what saves you from needing 120 times the data.
You must classify a point cloud from a 3D scanner. Which design follows the lecture’s two rules?

Now press Teach and explain one architecture out loud, from memory: the symmetry it respects, what it shares, and what it aggregates over. Then return to the hero and scramble the data once more.

MIT Multimodal AI, Lecture 2.2: Data and Heterogeneity
The course site and slides · Back to Gleams