Most failed AI projects fail before the model: in the data, then in the loss curve. Learn the order to look, and the one fix each symptom points to.
Learn how to think about a multimodal AI project, where its data comes from, how PyTorch trains a model on it, and how to debug the run when it breaks.
Pick a broken run, then choose the fix you think cures it. A right fix bends the curves back to health. A wrong fix tells you why it does not apply. Then we build, piece by piece, the data, the code and the checks underneath.
You need a feel for what a model is (a function with numbers inside that we adjust until its answers improve) and a little Python. We build the rest from zero.
Based on Lecture 2.1 of MIT MAS.S60 / 6.S985, Modeling Multimodal AI (Spring 2026), from the MIT Multisensory Intelligence group led by Paul Pu Liang. The thinking-about-projects talk is by Valdemar Danry; the data and PyTorch parts are by teaching assistant Edgar Morfin. Lecture slides. Every idea is re-taught here in our own words and devices.
The teal line is the loss on training data. The warm line is the loss on validation data, which the model never trains on. Choose a symptom, then a fix.
The curves are illustrative shapes drawn from the tutorial's debugging rules (slides 67 and 68), not logs of a real run. Chapters 7 to 9 run real training in your browser. The five symptoms and their fixes are the tutorial's.
Chapter 0
See why this class asks for projects instead of problem sets, and what it means to think about a project
Picture a laptop with a camera and a microphone, and one free weekend. You could build a program that watches a face and says how the person feels. Or one that listens to a voice and writes the words. Or one that reads a chart and answers a question about it.
None of these is a homework problem with one right answer. Each is a project: you choose a goal, gather data, pick a model, and find out whether it works. The lecture opens with a warning about exactly this: this is not your typical machine learning class.
A typical class gives you a dataset and a model and asks you to reproduce a number. This class gives you a technique each week and asks you to point it at something you care about. The first half of this lecture, by Valdemar Danry, is about how to think about that choice. The second half, by teaching assistant Edgar Morfin, gives you the data and PyTorch tools to carry it out.
The course has five homeworks and a final project. Each homework teaches one technique.
A modality is one kind of signal: text, sound, an image, a video, a touch reading, a brain recording. Multimodal AI is a model that uses more than one kind at once. We define each technique properly in the later lectures. For now you only need the order: data first, then models that combine signals, then models that reason, then models that act.
Then comes the part that makes the class unusual. The course gives you complete freedom over what you apply the required techniques to. Nobody assigns the topic. In homework 2 you fuse whatever signals your project has. In homework 5 you build whatever agent your project needs.
The lecture states the goal in one line: to teach you how to use multimodal AI to make (almost) anything. After each assignment there is a show and tell, where everyone presents what they made. You see what your classmates tried before you start the next one.
The final project may include new techniques, datasets, applications or empirical findings. An empirical finding is a measured result, such as "this signal helps and that one does not". The course also links reports from earlier students, so you can see what a finished project looks like. The last slides of this part cover documentation: you set up a class page where you record each project as you go. Keep that page honest and current. It is the record you will use to write the final report.
Here is one small project carried through all five homeworks: read the emotion in short video clips. The device below shows what each homework would add. The course does not assign this project. We chose it because the next chapters use it. Press each homework in turn and read what it adds to the pipeline.
The boxes are the five homeworks, left to right. The teal box is the one you are on. Below it is what that homework adds to a clip-emotion project. Press each box in order.
The five homework names are the course's (slide 5). The clip-emotion project and every number on the cards are illustrative: a toy project we chose to show what each technique contributes.
Notice the shape. Each homework takes the output of the one before and makes it more useful. That is the real skill this lecture teaches: seeing the project as a pipeline whose pieces you can swap, test and improve one at a time.
Chapter 1
Map a gallery of real projects by what goes in and what comes out, then pick a shape for yours
Imagine you walk into a gallery with thirty exhibits. A glove that turns sign language into speech. A model that draws what you type. A robot arm that folds a towel when you tell it to. A brain recording that becomes a sentence. What do they share?
Each one takes some kind of signal in and gives some kind of signal out. The signal in is the input. The signal out is the output. A model is the function in the middle that learns the mapping from many examples. Once you see a project as an input type and an output type, a huge gallery shrinks to a small map.
The lecture spends more than thirty slides on such a gallery. We will not copy those slides. Instead we sort their projects into a grid, so you can see which cells are crowded and which are empty.
In the grid below, each row is a kind of input and each column is a kind of output. Each dot is one project from the lecture. Pick an output kind, and the device lists every project that makes it, with its input. We assigned the cells ourselves from each project's name and slide caption.
Rows are inputs, columns are outputs. Each dot is one project from slides 8 to 40. Choose an output to light its column and list the projects.
The projects and their order are the lecture's. Placing each one in an input row and an output column is our own reading of its slide name and caption, so a project could reasonably sit in a neighbouring cell.
Two things stand out. First, the Words column is the most crowded: text is a good output because a language model can write it, and you can read it to check the result. Second, the input rows are varied: sound, body sensors, images, radio signals, and plain text. Real projects rarely use only one row. The emotion project from Chapter 0 uses three at once.
Writing the project as a function signature is a quick way to check you have thought it through. Here are three from the gallery, written as code. The shapes are illustrative: they show the kind of tensor each project handles, not the exact sizes the authors used.
python# illustrative signatures: input type -> output type def brain_to_text(recording): # float array [time, channels] return sentence # str def chart_qa(chart_image, question): # uint8 [H, W, 3] and a str return answer # str def robot_policy(camera_image, instruction): # [H, W, 3] and a str return action # float vector, e.g. 7 numbers
If you cannot write the signature, you do not yet know what data to collect. If you can write it, you already know which row and column your project lives in, and the next chapters tell you how to feed it.
The lecture gives the final project a clear bar. It "can include new techniques, datasets, applications or empirical findings". Read that as four ways to add something new.
You need only one. A good project often has a surprise built in: an unusual input, an odd place to deploy, or a question nobody checked. The show and tell after each assignment is where you find out whether yours is surprising.
Try it with a project of your own. Say you want a tool that tells a speaker, in real time, how a talk is landing. Write the input: a video of the audience (image row) and the room's sound (sound row). Write the output: a number from 1 to 5 for the mood (labels column). Your project sits in the image and sound rows, in the labels column. Now look at the dots there. Four lecture projects already live in that column, three of them with image inputs and none with sound. That tells you the output is well trodden, and the sound input is the part that could be new.
This simple exercise also tells you what to ask for in Chapter 2. A project that needs an audience video and a room recording, both labelled with a mood score, needs paired data. Paired data is rare, so you should check early whether it exists.
Slide 20 shows a model page on a public model hub with a notice: access to this model has been disabled. The notice lists harms such as hate speech, spam and harassment. It is a quiet reminder that a project has people on the other side of it. Before you build, ask who could be hurt if it works, and who if it fails. Write the answer on your class page. It belongs there with your goal and your data.
Documentation is the other habit these slides push. Your class page is where each project gets a goal, a data description, what you tried and what happened. A project you cannot explain from your own notes is hard to improve, and hard to grade.
Chapter 2
Decide the goal and the modalities first, then find a dataset that already holds them
You decide to build a program that reads the emotion in a spoken sentence. You open your laptop and start choosing a model. Stop. The model is the second decision. The first is the data, and the lecture's TA, Edgar Morfin, states the stakes in one line: bad data gives bad results.
A model learns only from the examples it sees. Show it mislabelled clips and it learns the mistakes. Show it clips with no sound and it can never learn to use sound. So before any code, you answer two questions.
These two answers shorten the search. Without them you could browse datasets forever. With them you can reject most candidates in seconds.
The lecture's advice is short. Find a dataset that suits the goal. Ideally it holds all the modalities you plan to use. If it does not, find one from which you can extract (pull out with a program) the missing modalities. And do not limit yourself to the datasets that already exist: sometimes you must be creative to solve the problem.
The slide's example is MELD, the Multimodal EmotionLines Dataset. It holds about 13,000 spoken lines from 1,433 conversations in the TV series Friends. Each line has a video clip, an audio track and a text transcript. Each is labelled with an emotion and a sentiment (positive, neutral or negative), according to the dataset's paper (Poria et al., arXiv 1810.02508). It already has three modalities and labels, so a project on emotion in speech can start the same day.
Now try the search yourself. The device holds six datasets. One is MELD. The other five are toy examples we made up, to show the reasoning. Tell the device which modalities your project needs, and it sorts the datasets into three groups: fits as is, fits after extraction, and does not fit.
Toggle the modalities and labels your project needs. Each row is a dataset. Filled chips are what it has. A teal ring means you need it and it is missing.
MELD's modalities and labels come from its paper (arXiv 1810.02508). The other five datasets and the extraction rules are illustrative: audio can come out of a video file that has a soundtrack, and text can come out of audio with speech recognition. A label cannot be extracted: someone or something has to annotate it.
Turn on Video, Audio and Emotion label, and MELD is the only perfect match. Add Text and MELD still fits. Now turn off Emotion label and several sets become useful, because text can be extracted from the audio. The silent clips never fit once you need audio, because a missing soundtrack cannot be invented. Labels are the hard part: if no dataset has the labels you need, you must pay for annotation, or find a model that can label for you (Chapter 4).
Two more questions belong at this stage, and neither is technical. First, may you use this data? Read the dataset's licence, and ask whether the people in the clips agreed to be recorded. A dataset of faces or voices carries real people, and the answer decides whether you can publish what you build. Second, can you make the data yourself? The slide's advice to be creative covers this. A few hundred clips you record and label with care can beat a large set that does not match your goal. Many of the gallery projects in Chapter 1 began that way.
Having found a dataset, do not train yet. Andrej Karpathy's A Recipe for Training Neural Networks (2019) puts it first: become one with the data. Spend hours, not minutes, scanning thousands of examples. Look for duplicates, corrupted files, label noise, and imbalance. Ask what features seem to matter and which ones the model could use as a shortcut.
Here is a ten-line first look, as real Python. It uses pandas, a table library. Run it before any model.
pythonimport pandas as pd df = pd.read_csv("meld_train.csv") # one row per spoken line print(df.shape) # (rows, columns) print(df["Emotion"].value_counts()) # is any label rare? print(df["Utterance"].str.len().describe()) # lengths: any empty or huge? print(df.duplicated(subset=["Utterance"]).sum()) # repeated lines print(df.sample(20, random_state=0)) # READ twenty rows by eye
The column names above follow the public MELD files, but check them against your copy. The last line matters most. Reading twenty random rows takes two minutes and catches problems no statistic shows. If a label looks wrong to you, it will look wrong to the model too, only it will not tell you.
Suppose the first look shows that 47% of lines are labelled neutral and 2% are labelled fear. That is a label imbalance. Here is the symptom, the measurement and the fix.
Notice that you could only see this because you looked at the data first. That is the pattern across the whole tutorial: the cheapest checks come first.
Chapter 3
Use a package to pull each modality out of a clip, and see the tensor each one returns
You have 2,000 video clips and you want the audio, the words, the faces and the body pose from each one. A tempting thought: write the code that finds a face in a picture, or turns a sound wave into text. How hard can it be? The TA's slide for this moment asks how to code up all the algorithms for a modality of interest. The next slide answers in two words: Do not.
This is the tutorial's most practical advice. Detecting a face, tracking a body or recognising speech took research teams years to get right. The result is already a free package. Rewriting it costs weeks, makes bugs, and gives you a worse tool than the one you could install in a minute.
The slide adds a fair caveat. There are exceptions: a project that is about a better extractor, or a modality where no good tool exists. Outside those cases, two things have changed in recent years.
So the rule is: use what is already out there, and code only the glue.
The lecture lists example packages by modality. The table keeps their names and adds, from each package's documentation, what it hands back. A tensor is a grid of numbers with a fixed shape, such as 300 frames by 224 by 224 by 3 colour values.
| Modality | Packages on the slides | What you get back |
|---|---|---|
| Video | FFmpeg, OpenCV, MoviePy | Frames: a stack of images |
| Audio | FFmpeg, openSMILE, pyAudioAnalysis | A waveform, or low-level features such as pitch and loudness |
| Facial features | MediaPipe, OpenFace | Landmarks: x, y, z points on the face, per frame |
| Text | OpenAI Whisper, SpeechRecognition, MoviePy, OpenCV | A transcript of the speech |
| Pose | OpenPose, MediaPipe | Landmarks: joints of the body, per frame |
Each package has a failure you should learn once. A face tracker returns nothing when the face turns away or the light is poor, so some frames will have no landmarks. A speech recognizer may invent words in a noisy room. A pose tracker can swap a left hand for a right one. None of this is a reason to write your own. It is a reason to count the empty frames, listen to five transcripts, and record what fraction of clips each tool handled.
For niche modalities, the slide says: either a package exists, or the dataset has done it already, or you must do it yourself. Look deep and be creative.
The device below takes one ten-second clip and runs it through the extractor you choose. Watch the shape change. The point is to see how much data each modality carries: a modality is not just a name, it is a tensor you must store and feed to a model.
The left box is one 10-second clip. The pink box is the package. The right box is the tensor you get. Choose a modality to change the package and the shape.
Sizes are for an illustrative clip: 10 seconds, 30 frames per second, frames resized to 224 by 224, audio at 16,000 samples per second. The package choices are from the slides. The landmark counts are those of MediaPipe Face Mesh (468 points) and MediaPipe Pose (33 points); the 88 audio features are openSMILE's eGeMAPS set. The word count is a typical speaking rate.
The video is by far the biggest: 45 million numbers for ten seconds. The face landmarks, by comparison, take about 421,000 numbers, and they carry the part of the video that mattered for emotion. That is the reason to extract at all. A model trained on 88 well-chosen audio numbers can learn faster than one trained on 160,000 raw samples, because someone has already done the hard part of finding what matters.
Do the arithmetic for all 2,000 clips. Raw frames at 45,158,400 bytes a clip make 90.3 gigabytes (GB). Face landmarks at 300 frames by 468 points by 3 numbers, stored as 4-byte floats, are 1,684,800 bytes a clip: 3.4 GB for the lot. The extraction takes hours, but you pay once. After that every experiment reads the small table, and your laptop can train on it. So the habit is: extract, save to disk, and train from the saved features. Do not re-run the package inside every training run.
This is the whole job. Two shell-style calls with FFmpeg cut the audio and the frames, and one call to Whisper writes the text. The glue is the loop over clips.
pythonimport subprocess, pathlib, whisper asr = whisper.load_model("base") # a small speech recognizer rows = [] for clip in sorted(pathlib.Path("clips").glob("*.mp4")): wav = clip.with_suffix(".wav") # -vn: drop video, -ac 1: mono, -ar 16000: 16 kHz, the rate Whisper expects subprocess.run(["ffmpeg", "-y", "-i", str(clip), "-vn", "-ac", "1", "-ar", "16000", str(wav)], check=True) text = asr.transcribe(str(wav))["text"] # str rows.append({"clip": clip.name, "wav": wav.name, "text": text})
Everything hard sits inside ffmpeg and whisper. Your code only names the files and keeps the results in a table. That table is the dataset for the next homework.
The slide ends with a warning in capital-letter spirit: be aware of package dependencies, and always set the packages up in a virtual environment. A dependency is another package yours needs, at a version it can use. Two packages may need different versions of the same third one.
pip list and compare versions with what each package asks for.python -m venv .venv), install into it, and save the exact versions with pip freeze > requirements.txt. A teammate, or you in six months, can rebuild it.Chapter 4
Follow a research pipeline that cuts hours of video into clips and asks models for features
Suppose your data is not 2,000 neat ten-second clips. It is a few hundred sessions, each close to an hour long. Nobody can hand that to a model and hope. The lecture closes its data section with a real research example of exactly this, and it shows how every earlier step fits together.
The example comes from work on autism assessment. ADOS is an official diagnostic exam. A child sits in a specific room with an examiner and a parent. The examiner presents various activities and notes the child's responses. The session is video recorded, so a clinician can review it. The responses fill in items on a form, grouped by type of response, and the item scores add up to a final score.
The slide calls this a research effort to test whether better data helps a model score those items. It is not a tool that diagnoses anyone. Keep that line clear: the question is about what data makes a model's job easier.
The slide gives two reasons. First, the sessions are very long videos, and current multimodal models cannot handle them. The slide marks this as an open research problem. Second, training directly on such videos would probably not give a useful model. Small signals, such as a body movement, the pitch of a voice or a facial expression, would get lost in the mass of pixels.
The fix follows Chapters 2 and 3: extract the modalities first, so the model sees the subtle signals in a form it can use.
The slide lists six things the team keeps for each session. Audio and text share one line below.
Try the first step, segmenting. The device cuts a session into clips of the length you choose and shows the table that results. The 45-minute session and the three-second call time are illustrative numbers of ours.
The bar is one 45-minute session. Each tick is a cut. Drag the clip length. The table below it gets one row per clip and one column per feature.
Illustrative: a 45-minute (2,700 s) session, four mid-level questions per clip (emotion, sentiment, anxiety, depression, from the slide), and 3 seconds per model call. Real costs depend on the model and the hardware.
Short clips mean many rows and many model calls, but each call is small and each row is precise. Long clips mean few rows, and each clip may hold several different moments. There is no best length. You pick the one that keeps a single moment in a single clip and keeps the bill affordable.
How do you get a written summary of every clip? The slide's answer: use models that already exist. The idea comes from a paper on affective computing, the study of emotion in machines (Luo et al., 2025, arXiv 2508.02429, cited on slide 62).
The summaries will not be perfect. But the slide's argument is sound. Current multimodal language models still lean heavily on text. If you give them extra text, such as a generated summary of what is happening, they do better, even when the summary has small errors. So a rough summary is worth adding.
Now the hard question. If you only have the video, where do emotion, anxiety and depression labels come from? Humans cannot supply them at this scale. Their judgments are subjective and take painful effort. So the team asks a model, using instruction tuning: training a model on examples of the form instruction in, answer out, so that it follows instructions well.
The slide's recipe is to tell a baseline model, or one trained for the purpose, "Here is the video. Tell me what X is, using only the following options: 1, 2, 3, 4." Here is that as code. The slide only shows the options 1, 2, 3, 4. The six emotions in the code are our own illustrative choice.
pythonOPTIONS = {1: "happiness", 2: "sadness", 3: "fear", 4: "disgust", 5: "anger", 6: "surprise"} prompt = ("Here is a video clip with its transcript. Tell me the dominant " "emotion using only these options: 1 happiness, 2 sadness, 3 fear, " "4 disgust, 5 anger, 6 surprise. Answer with the number only.") reply = model(prompt, clip_frames, transcript) # str, for example "3" try: emotion = OPTIONS[int(reply.strip())] except (ValueError, KeyError): emotion = None # retry, or drop this clip
The options and the "number only" rule are what make the output usable. A free answer like "the child seems a bit worried but also curious" is correct and useless: you cannot put it in a table column.
The last slide of this part shows the result in action: the ADOS video on top, and below it the extraction running live over a face. The team built the pipeline to test whether adding these features improves performance. Notice the test: it is an empirical finding, one of the project types from Chapter 1.
Chapter 5
Run one training step by hand and see what a tensor, a gradient and an optimizer do
You have a table of data. Now you need a program that learns from it. The lecture's second tutorial, shown on slide 65, points you to two notebooks to open before class: one for PyTorch and one for Hugging Face. It also asks you to register for a Hugging Face account and a Weights and Biases account. This chapter covers the PyTorch half. The notebook is the course's own, MAS.S60 Pytorch Introduction. We teach the same ground from zero in our own way, so open the notebook after this chapter and run its cells.
PyTorch is a library that does two jobs. It stores numbers in tensors, grids of numbers that can live on a graphics card. And it computes gradients for you. A gradient tells you, for each number inside the model, which way to nudge it so the answers get better. Every model in this course, from a small classifier to a large language model, trains this way.
Start with a model of two numbers: a weight w and a bias b. Given an input x, it answers w × x + b. That is a straight line. Our data are four points that happen to lie on y = 2x + 1: the inputs 1, 2, 3, 4 and the answers 3, 5, 7, 9. The model starts knowing nothing, with w = 0 and b = 0. Training means finding w = 2 and b = 1 from the data alone.
One training step has four stages. Remember the order, because every PyTorch program repeats it.
The device lets you walk through these stages with real arithmetic, not a drawing. Press Step four times to see one full training step. Then press Run 10 steps to watch it learn. Finally push the learning rate past about 0.12 and watch the same four stages blow up.
The dots are the data. The teal line is the model. The list shows the four stages with their numbers. Press Step to run the next stage.
The arithmetic is exact: the same four formulas PyTorch evaluates for this model. The data (y = 2x + 1 at x = 1, 2, 3, 4) are our own toy example, not the notebook's.
Work the first step by hand once. With w = 0 and b = 0 the predictions are all 0, so the misses are −3, −5, −7, −9. Square and average: (9 + 25 + 49 + 81) / 4 = 41. The gradient for w is the average of 2 × miss × x, which is −35. The gradient for b is the average of 2 × miss, which is −12. With a learning rate of 0.05, the update gives w = 0 + 0.05 × 35 = 1.75 and b = 0 + 0.05 × 12 = 0.6. One step moved both numbers most of the way. The device shows the same values.
This is the entire program. Each line matches a stage of the device.
pythonimport torch x = torch.tensor([1., 2., 3., 4.]).unsqueeze(1) # shape [4, 1]: 4 examples, 1 feature y = torch.tensor([3., 5., 7., 9.]).unsqueeze(1) # shape [4, 1] model = torch.nn.Linear(1, 1) # weight [1, 1], bias [1] opt = torch.optim.SGD(model.parameters(), lr=0.05) for step in range(100): pred = model(x) # forward -> [4, 1] loss = ((pred - y) ** 2).mean() # loss -> scalar opt.zero_grad() # clear last step's gradients loss.backward() # backward -> fills .grad on w and b opt.step() # update -> w -= lr * w.grad
Three details are worth knowing. First, nn.Linear(1, 1) starts with random numbers, so your first loss will not be 41. Second, loss.backward() does the calculus for you by recording every operation as it runs. That recording is called autograd. Third, the shapes carry the meaning: the first number is always the batch, the count of examples processed together.
For real data you add three helpers. A Dataset says how to fetch one example, for instance one clip's frames and its label. A DataLoader groups examples into batches and shuffles them. A device move, .to("cuda"), puts the model and each batch on the graphics card. A batch of 32 colour images has shape [32, 3, 224, 224], and its labels have shape [32]. Your model eats the first and the loss compares its output to the second.
The slide's two accounts fit in here. Hugging Face hosts thousands of pretrained models and datasets you can download with one line, which is how you will start most projects (Chapter 6). Weights and Biases records your loss curves online as the run goes, with wandb.log({"loss": loss.item()}), so you can read the curve from any machine. The next chapters teach you how to read it.
To go deeper on each part, see our lessons on PyTorch tensors, autograd and the training loop.
model.weight.grad before and after a step. If it keeps growing instead of resetting, the old gradients are piling up.opt.zero_grad() each step. PyTorch adds each new gradient to the old one by default, because that is useful in some advanced cases.Chapter 6
Look first, start simple, reach for a pretrained model, and keep every baseline
A new dataset lands on your desk. Your hands itch to build something impressive: a deep network with many layers, an exotic design from last month's paper. The PyTorch tutorial's first debugging slide, How to design ML models for new data, says to resist. It has four lines, and each one saves weeks.
Let us unpack each term, then watch the rule work.
Low-dimensional data has only a few numbers per example: a person's height, weight and age, say. High-dimensional data has thousands: a photo of 224 by 224 pixels has 150,528 numbers. The number of numbers per example is the dimension.
A simple model has few adjustable numbers and a simple shape. Logistic regression draws one straight line between two classes. A random forest votes among many small decision trees. An SVM, a support vector machine, finds the line with the widest margin. A shallow MLP is a small neural network, a multi-layer perceptron with one or two hidden layers.
A pretrained model is a network somebody else already trained on a huge dataset. You download it and reuse what it learned. Images and language have so much structure that a model trained on millions of examples nearly always beats one you train alone on thousands.
A baseline is a simple result you must beat to claim progress. If your clever network scores 91% and a one-line baseline scores 90%, the cleverness bought you one point. If you never ran the baseline, you would not know.
The device trains four models on three tiny datasets, for real, in your browser. The first is the laziest possible baseline: always guess the commonest class. The others are logistic regression, k-nearest neighbours (label a new point by a vote among its five closest training points) and a small MLP with eight hidden units. Each is scored on 160 validation points it never saw. Switch the dataset and see where simple is enough.
Dots are validation points. The coloured regions are what the chosen model predicts. The bars show every model's validation accuracy. Choose a dataset, then a model.
Everything runs live: 160 training and 160 validation points per dataset, drawn from a fixed random seed. Logistic regression and the MLP (8 tanh units) train by gradient descent for a fixed number of steps. The datasets are toy shapes we generated, not the course's data.
Two blobs: logistic regression already reaches the ceiling, and the MLP adds nothing. Two moons: a straight line cannot follow the curve, so the line loses points and the MLP and nearest neighbours win. XOR corners: one straight line fails completely, because the classes sit on opposite diagonals. Only models that bend can solve it. The lesson is not "simple always wins". It is that the baseline tells you how much each upgrade bought, and you can only know that if you ran it.
For tables of numbers, scikit-learn gives you the baselines in a few lines. For images or text, a pretrained model gives you a strong start in two.
pythonfrom sklearn.dummy import DummyClassifier from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier for name, m in [("majority", DummyClassifier(strategy="most_frequent")), ("logreg", LogisticRegression(max_iter=1000)), ("forest", RandomForestClassifier(n_estimators=200, random_state=0))]: m.fit(X_train, y_train) # X_train: [n, d]; y_train: [n] print(name, m.score(X_val, y_val)) # accuracy on held-out data # language data: start from a model somebody already trained from transformers import pipeline clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english") print(clf("The clip made me smile")) # a list with a label and a score
The scikit-learn part gives you three honest numbers in under a minute. The Hugging Face part downloads a language model once and runs it on your text. Neither needs a single line of network design.
Use the numbers from the two-moons run. The majority guess scores 50.0%. The line scores 88.7%, which is 38.7 points above guessing. Nearest neighbours score 97.5%, which is 8.8 points above the line. The small MLP scores 98.1%, which is only 0.6 points above nearest neighbours. Now ask what each step cost. The line is one multiplication per example. Nearest neighbours must keep all training data and compare against it. The MLP needs a training loop and tuning. The first jump is huge and cheap. The last is tiny and costly. Write the first three numbers in your notes, and decide with them.
Andrej Karpathy's A Recipe for Training Neural Networks gives the slide's rule a motto: don't be a hero. In the early stages, do not stack exotic parts. Find the most closely related paper and copy its simplest architecture that works. Then complexify only one thing at a time: if you have several input signals, plug them in one by one, and check that each one gives the boost you expect. This is the multimodal fusion advice of Homework 2 in one sentence.
Chapter 7
Turn each loss symptom into one cause and one fix, then overfit one batch to prove the pipeline
You press run. Numbers scroll past. Somewhere in that stream is the one that tells you whether your model is learning: the loss. The loss is a single number that scores how wrong the model is on the current batch. A batch is a handful of examples processed together. Lower is better, and zero means perfect.
Plot the loss against the training step and you get the loss curve. It is the heartbeat of a training run. The tutorial's debugging slide teaches you to read it in a fixed order, from the cheapest check to the most expensive.
The order matters because a bug at step 1 makes every later check meaningless. A model trained on shuffled labels will still produce a pretty loss curve.
Andrej Karpathy's A Recipe for Training Neural Networks gives a check that costs nothing: verify loss at init. Before any learning, a model with sensible starting weights knows nothing. For a classifier with n classes it should give each class the same probability, 1/n. The usual loss, cross-entropy, scores the true class by minus the log of its probability, so the starting loss should be −log(1/n) = log n.
For three classes that is log 3 = 1.0986. If your first printed loss is 7.4 or 0.02, something is wrong before training even starts: a bad initialisation, a wrong loss, or labels out of range. The device below draws this value as a dashed line. Every run starts exactly on it, because its weights start at zero.
The device trains a real three-class softmax classifier on 96 points, in your browser, using minibatch gradient descent with batches of 16 points. Choose a learning rate and watch the curve. Then turn on the two fixes the slide names for a spiky loss and the check Karpathy calls overfit one batch.
The teal line is the loss of each batch over 100 steps. The dashed line is the starting value, log 3. Pick a learning rate, then try the two toggles.
This is real training, not a drawing: three clusters of points (our toy data, from a fixed seed), a linear softmax model with 9 numbers, batches of 16 drawn at random. Spikes above 3.0 run off the top of the plot. The one-batch test trains on only the first 8 points.
At 0.002 the loss barely moves: after 100 steps it is still near 1.0, close to where it started. At 0.3 it drops fast, then wobbles around a floor. The floor is not zero because the three clusters overlap, so even a good model is sometimes unsure. At 8 each step overshoots, and the loss jumps up to values far above the start. Turn on clipping and the spikes shrink at once. Turn on the one-batch test and the loss falls toward zero, because eight points are easy to memorise.
The slide pairs each loss pattern with a cause and a short list of cures. Here they are in one place. We have added what each cure does.
| Symptom | Likely cause | Fixes from the slide |
|---|---|---|
| Loss is NaN | Numbers overflowed, or bad inputs or weights | Inspect weights and inputs for NaN; initialise weights; for a language model use bfloat16 (Chapter 8) |
| Loss not changing | Underfitting: the model cannot, or is not allowed to, learn | Raise the learning rate; lower the weight decay; add model complexity; use a better optimizer |
| Loss varies wildly or rises | Steps are too big | Lower the learning rate; clip gradients; use a better optimizer |
| Model predicts one label | Label imbalance | Data augmentation; scale the loss per class |
Weight decay is a small pull that shrinks every weight a little each step, so the model prefers simple answers. Too much of it stops learning (Chapter 9 uses it to fight overfitting). An optimizer is the rule that turns gradients into updates. Plain SGD steps straight against the gradient. Adam, which Karpathy calls safe, scales each step by recent history. The slide adds a personal tip: try second-order optimizers from packages such as Heavyball. We have not tested that tip here, so treat it as the speaker's experience.
Gradient clipping sets a ceiling on the size of an update. If the gradient's length, its norm, is above the ceiling, the whole gradient is scaled down to the ceiling. The direction is kept. Suppose one bad batch gives a gradient of norm 4.0 and the ceiling is 0.1. The scale is 0.1 / 4.0 = 0.025. Every component shrinks by a factor of 40, and the step stays small. A normal batch with norm 0.05 is untouched.
Loss scaling for imbalance multiplies the loss of rare classes by a larger weight. If class A is 95% of the data and class B is 5%, weights of 1 and 19 make a mistake on B count 19 times more, so the two classes pull equally.
pythonimport torch loss_fn = torch.nn.CrossEntropyLoss(weight=torch.tensor([1.0, 19.0])) # rare class counts 19x for x, y in loader: # x: [B, ...], y: [B] loss = loss_fn(model(x), y) opt.zero_grad(); loss.backward() norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) print(float(loss), float(norm)) # log both: the norm shows the spikes opt.step()
clip_grad_norm_ returns the norm it measured before clipping, so the same call both fixes and measures.
Karpathy's most useful check is this: take a handful of examples, as few as two, and train on them alone. A model with enough capacity should drive that loss to nearly zero. If it cannot, you have a bug in the code, the labels or the loss, and you must not continue to full training. If it can, the pipeline works end to end. The device's second toggle does exactly this.
Chapter 8
Find out why numbers overflow and why bfloat16 survives where float16 fails
Your run has been going for three hours. The loss prints 2.31, then 2.18, then 2.05. Then the screen shows nan, and every line after it says nan too. The model is dead and you cannot revive it. The tutorial gives this symptom its own row: loss is NaN: inspect the weights and inputs for NaN values, make sure the weights are initialised, and for a language model use bfloat16 instead of float16.
That one line hides a good deal of machinery. Let us open it.
NaN means "not a number". A computer produces it when an operation has no sensible answer: zero divided by zero, infinity minus infinity, or zero times infinity. The nasty part is how it behaves afterwards. Any operation that touches a NaN returns a NaN. Add 1 to NaN and you get NaN. Multiply it by 0 and you still get NaN.
Follow one NaN through a training step. A single activation (a number inside the network) becomes NaN. The loss, which averages many activations, becomes NaN. The gradient of a NaN loss is NaN for every weight. The update subtracts a NaN from each weight, so all the weights become NaN. From then on, every output is NaN. One bad number kills the model in one step, and that is why the symptom looks sudden.
A computer stores each number in a fixed count of bits. Think of scientific notation: 1.234 × 105. The exponent (the 5) says how big the number is. The mantissa (1.234) says how many digits you keep. A number format splits its bits between the two, plus one bit for the sign.
Float16 is small and fast, but its range is tiny. Any result above 65,504 becomes infinity (inf). Bfloat16, the "brain float", was designed to keep float32's range and give up digits instead. For deep learning that is usually the better trade, because a model can tolerate a rough digit but cannot tolerate overflow. This is the reason behind the slide's advice for language models.
Here is the most common way a healthy network reaches infinity. Softmax turns scores into probabilities. It first computes e to the power of each score, then divides each by the total. Take two equal scores, say s and s. The right answer is 0.5 and 0.5, whatever s is. But es grows very fast. At s = 11 it is 59,874, still inside float16. At s = 12 it is 162,755, past the limit, so float16 stores inf. Then the division is inf / (inf + inf), which is NaN.
Slide the score in the device below and watch three formats store the same number. The bars are drawn on a log scale, so each tick is ten times bigger than the last.
Each bar is the range a format can hold. The marker is exp(score). When it passes the end of a bar, that format stores infinity. Drag the score.
Format limits are from the IEEE half-precision and bfloat16 definitions: float16 tops out at 65,504, bfloat16 and float32 near 3.4 × 1038. The rounding is emulated in JavaScript. The two-equal-scores softmax is our illustration of the mechanism, not a step of the course's code.
Three zones appear as you drag. Up to s = 10 all formats agree: the softmax of two equal scores is 0.5. At s = 11, float16 can still store es but not the sum es + es. The sum becomes inf, and es / inf is 0, so the answer is silently wrong, with no NaN to warn you. From s = 12 the numerator is inf too and the result is NaN. Bfloat16 survives up to s ≈ 88, which is where float32 itself overflows.
Real libraries already protect you here. A stable softmax subtracts the largest score first, which turns every exponent into a number at most 0 and every result into at most 1. Hand-written code does not. Compare the two.
pythonimport torch s = torch.tensor([12., 12.], dtype=torch.float16) naive = torch.exp(s) / torch.exp(s).sum() # tensor([nan, nan]) stable = torch.exp(s - s.max()) / torch.exp(s - s.max()).sum() # tensor([0.5, 0.5]) builtin = torch.softmax(s, dim=0) # [0.5, 0.5]: already stable # train in bfloat16 where the hardware allows it with torch.autocast(device_type="cuda", dtype=torch.bfloat16): loss = model(x, y)
The first result follows the arithmetic above. In the second, s - s.max() turns both scores into 0, so the exponentials are 1 and the answer is exactly 0.5. The third uses PyTorch's own softmax, which does this for you. The last block runs a forward pass in bfloat16, which is the slide's recommendation. With float16 you would also need loss scaling, a trick that multiplies the loss by a large factor so that small gradients do not round to zero. Bfloat16 needs no such trick.
Overflow is only one source. The slide lists two more checks, and both are cheap.
torch.isnan(x).any() before training.When you cannot see where the NaN started, ask PyTorch to find it. Wrapping the step in torch.autograd.set_detect_anomaly(True) makes the backward pass stop at the first operation that produces a NaN and print the line of your code that created it. It slows the run, so use it only to hunt.
set_detect_anomaly; check isnan on the input, then on each layer's output. The first layer whose output is NaN or inf is the culprit.Chapter 9
Measure the gap between train and validation and see what closes it
A student studies for an exam by memorising last year's paper, answer by answer. On last year's paper she scores 100%. On this year's paper she scores 55%. She did not learn the subject. She learned one exam.
Models do this too. The tutorial's second debugging slide tells you to compare two numbers: the model's score on the data it trains on, and its score on data it never trains on. The first is the training score. The second is the validation score, and the held-out data is the validation set. Think of it as a mock exam the model never gets to study.
The slide gives two rules.
The device fits a polynomial curve to noisy points. The true pattern is a smooth wave. The model's complexity is the degree of the polynomial: degree 1 is a straight line, degree 9 has ten adjustable numbers that can bend into nearly any shape. Teal dots are the training points, and warm dots are validation points the curve has not seen. The device solves the fit exactly, in your browser. Start at degree 9 with 12 training points.
The teal line is the model. The dashed line is the true wave. The bars below compare training error and validation error. Change the complexity, the weight decay or the amount of data.
The wave is sin(2πx) with Gaussian noise of standard deviation 0.25, so the best possible error is about 0.06. The points come from a fixed random seed. Weight decay here is ridge regression, a penalty on large polynomial coefficients. Error is mean squared error.
Here are the numbers from the starting position. At degree 9 with 12 points, the training error is 0.030 and the validation error is 0.145. The model fits its own points about five times better than new ones. That ratio is the symptom the slide names.
Now try the fixes in order. Turn on a small weight decay: the error on training points rises a little to 0.048, but the validation error falls to 0.069, close to the noise floor. Return to no decay and switch to 48 points: validation error is 0.060 and training error 0.062, so the gap is gone. Karpathy's recipe agrees on the order: add real data first, because it is the one fix that is nearly always safe. Slide the degree to 11 with 12 points and no decay, and the curve passes through every training point, with a training error of 0.000 and a validation error above 1,000. That is memorisation, in its purest form.
Read the whole sweep for 12 points and no decay. At degree 1 the training error is 0.276 and the validation error is 0.299: the line is too simple to follow the wave. At degree 3 they are 0.054 and 0.072. At degree 5 they are 0.048 and 0.067, the lowest validation error of the sweep. At degree 9 they are 0.030 and 0.145. Training error never rose, yet validation error was best at degree 5. Choose the model by the validation number, and stop adding complexity where it starts to climb. That is early stopping applied to model size.
Weight decay penalises large weights, which pushes the model toward gentle curves. Reducing complexity removes the freedom to wiggle: degree 3 cannot memorise 12 points. Data augmentation invents extra training examples by changing real ones in ways that keep the label, such as flipping or cropping an image, or adding noise to audio. More data makes memorisation impossible, because there is more to memorise than the model can hold. Karpathy adds early stopping: end the run at the step where validation loss is lowest, before the model starts to memorise.
Now turn on the leak toggle. The "validation" set becomes the training points themselves, and the validation error equals the training error. Both look wonderful, and both are meaningless. This is the second rule of the slide: train and validation both near 100% usually means leakage, not genius.
Leakage in multimodal projects is subtle. Suppose you cut a one-hour therapy video into 120 clips, then split the clips at random into train and validation. Clips from the same child land on both sides. The model learns the child's face, voice and room. It looks brilliant on the validation clips, which show the same child, and fails on a new child. The fix is to split by session or person, never by clip.
pythonfrom sklearn.model_selection import GroupShuffleSplit # groups[i] = the person (or session) clip i came from split = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0) train_idx, val_idx = next(split.split(X, y, groups=groups)) # prove it: no person appears on both sides assert set(groups[train_idx]).isdisjoint(set(groups[val_idx]))
The assert line is the measurement. If it fails, the split leaks. Run it every time you change how you build the data.
Chapter 10
Count the bytes a fine-tune needs and see how LoRA and 4-bit weights fit a big model on one card
Your team has its own code library, with its own function names and house style. A general code model, such as StarCoder, knows Python well but has never seen your library. You would like it to suggest your functions when a developer types the first half of a line. The answer is to fine-tune the model: keep training it for a while on your own code.
There is a catch. A model with a billion weights is big, and training needs far more memory than running. The reading for this session is a Hugging Face cookbook notebook by Maria Khalusova, Fine-tuning a Code LLM on Custom Code on a single GPU. It shows how to fit the job on one graphics card. This chapter weaves its ideas into the tutorial: it is a worked case of Chapters 2, 5, 8 and 9 on one real project.
A parameter is one adjustable number inside the model. The notebook's model, StarCoderBase-1B, has 1,142,761,472 of them. To run it, you store each parameter. To train it with the Adam optimizer, you store three more things per parameter.
That is 16 bytes per parameter. For 1.14 billion parameters it is 18.3 gigabytes (GB), before counting the activations, the values each layer produces while a batch passes through. A 16 GB card cannot hold that. A 7-billion-parameter model would need 112 GB. The notebook avoids this with two tricks.
LoRA (low-rank adaptation) freezes the original weights and trains a small correction beside each big weight matrix. The correction is the product of two thin matrices. Instead of changing a d × d matrix W, you learn A of shape r × d and B of shape d × r, where r (the rank) is small, such as 8. The layer then computes with W + (α / r) × B × A, where α (alpha) is a fixed scale.
Count the savings for a matrix with d = 2048 (an illustrative size). W holds 2048 × 2048 = 4,194,304 numbers. B and A together hold 2 × 2048 × 8 = 32,768, which is 128 times fewer. The notebook reports the total effect: 5,554,176 trainable parameters out of 1,142,761,472, which is 0.486%. Only that sliver needs gradients and Adam states.
Quantization stores each frozen weight with fewer bits, like saving a photo at lower quality. The notebook loads the base model in 4 bits (the nf4 format from the bitsandbytes library, with double quantization), and computes in bfloat16, the format Chapter 8 recommended. Four bits is half a byte, so the weights shrink from 4 bytes (float32) to 0.5 bytes each. The combination of 4-bit weights and a LoRA add-on is called QLoRA.
Now try it. The device counts the bytes for weights, gradients and optimizer state under three methods, for four model sizes, against four common GPU memory sizes. It uses the exact byte counts above.
The bar is the memory for weights, gradients and Adam states. The lines mark common GPU sizes. Choose a model size and a method.
Byte counts: full fine-tune is 4 (weights) + 4 (gradients) + 8 (Adam) = 16 bytes per parameter; a bfloat16 base is 2 bytes; a 4-bit base is 0.5 bytes. Adapters are 0.486% of the parameters, the notebook's figure for the 1B model, and the same fraction is assumed for the larger models (illustrative). Activations, the CUDA context and quantization constants are left out, so real use is higher. The notebook ran on an A100 High-RAM GPU.
For the notebook's own 1.14B model, a full fine-tune counts 18.3 GB. LoRA on a bfloat16 base counts 2.4 GB, and QLoRA 0.7 GB. Raise the size to 7B and full fine-tuning jumps to 112 GB, beyond any card on the list, while QLoRA stays near 4 GB. The rule of thumb is simple: the frozen part is cheap, and the trained part is expensive. LoRA shrinks the trained part to under 1%.
The notebook still needs an A100, because the activations of 16 sequences of 2,048 tokens are large. Its code uses gradient checkpointing (set in its training arguments) to cut that cost. The method recomputes some activations during the backward pass, instead of storing them all. It trades time for memory.
pythonfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True) model = AutoModelForCausalLM.from_pretrained("bigcode/starcoderbase-1b", quantization_config=bnb, device_map={"": 0}) model = prepare_model_for_kbit_training(model) cfg = LoraConfig(r=8, lora_alpha=32, lora_dropout=0.0, bias="none", task_type="CAUSAL_LM", target_modules=["c_proj", "c_attn", "q_attn", "c_fc"]) model = get_peft_model(model, cfg) model.print_trainable_parameters() # trainable params: 5,554,176 || all params: 1,142,761,472
Read the notebook with the rest of this lesson in mind, and you will find each earlier chapter at work.
smangrul/hf-stack-v1. The notebook checks the characters per token ratio, which came out at 2.43. It says a ratio between 2.0 and 3.5 is good for code.One more idea in the notebook is worth knowing. Fill in the middle (FIM) trains the model to complete code given both the text before and the text after a gap. A prompt looks like <fim_prefix>...<fim_suffix>...<fim_middle>, and the model writes the middle. This is the task a developer's editor really needs. For more on the method, see our lessons on LoRA and PEFT and fine-tuning in practice.
CUDA out of memory.torch.cuda.max_memory_allocated() / 1e9, the peak in GB, and compare it with your card.Chapter 11
Keep the numbers, the checklist, the core code and the links onward in one place
Here is the whole tutorial in one paragraph. A multimodal AI project starts with a goal written as an input and an output. It then needs data that holds the right modalities, and you must read that data by eye before you trust it. You pull each modality out with existing packages, inside a virtual environment, and you let models label what people cannot. You train in PyTorch with one fixed loop. You begin with simple models and baselines, and you debug in a fixed order: data, outputs, loss, then train against validation. Each symptom points to a small set of fixes. When the model is big, LoRA and 4-bit weights let one GPU do the job.
| Number or term | What it means | Where from |
|---|---|---|
| 5 homeworks and a final project | Datasets, fusion, multimodal LLMs, reasoning and RL, agents | Slide 5 |
| 13,000 lines, 1,433 dialogues | The size of MELD, with audio, video, text and emotion labels | MELD paper, arXiv 1810.02508 |
| 45,158,400 numbers | 10 s of 30 fps video at 224 by 224 by 3 (illustrative clip) | Chapter 3 arithmetic |
| log n | The loss of an untrained n-class classifier: 1.099 for 3 classes, 1.609 for 5 | Karpathy, verify loss at init |
| 65,504 | The largest number float16 holds; bfloat16 and float32 reach about 3.4 × 1038 | Number-format definitions |
| 3e-4 | A safe first learning rate for Adam | Karpathy |
| 16 bytes per parameter | Weight 4, gradient 4, Adam states 8, in a full fine-tune | Chapter 10 arithmetic |
| 0.486% | Trainable share of StarCoderBase-1B with LoRA, rank 8 | Code-LLM notebook |
| 2.0 to 3.5 | A good characters-per-token ratio for code | Code-LLM notebook |
| Modality | One kind of signal: text, audio, video, pose, and so on | Chapters 0 to 3 |
| Baseline | A simple result you must beat to claim progress | Slide 66 |
| Data leakage | Information from validation that reached training | Slides 67, 68 |
Run down this list before every long training run. Each item is a cheap check that rules out a whole family of bugs. Mark the ones you have done and see where you stand.
Each stone is one check, in the order the tutorial teaches. Press a check when you have done it. The flag marks the next one to do.
The checks are drawn from the lecture's slides 50 to 68, Karpathy's recipe and the code-LLM notebook. The order is the one this lesson teaches, from the cheapest check to the most expensive.
This is the tutorial's advice as one short script. It splits by person, checks the starting loss, overfits one batch, then trains with clipping and bfloat16, logging both losses.
pythonimport math, torch from sklearn.model_selection import GroupShuffleSplit torch.manual_seed(0) # same run every time tr, va = next(GroupShuffleSplit(1, test_size=0.2, random_state=0).split(X, y, groups=who)) assert set(who[tr]).isdisjoint(set(who[va])) # no person on both sides model = make_model().cuda(); loss_fn = torch.nn.CrossEntropyLoss() opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01) def step(xb, yb): with torch.autocast("cuda", dtype=torch.bfloat16): # bf16, not fp16 loss = loss_fn(model(xb), yb) opt.zero_grad(); loss.backward() norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); return float(loss), float(norm) print(step(xb0, yb0)[0], "vs log n =", math.log(n_classes)) # check loss at init for _ in range(200): l, _ = step(xb0, yb0) # overfit ONE batch first print("one-batch loss", l) # should be near 0
This is Week 2, Lecture 1 of the course. It sets up the habits that Homework 1 (datasets) uses at once, and that every later homework relies on: fusion models, multimodal LLMs, reinforcement learning and agents all train in this same loop, and all of them fail in the ways you have just met.
To go deeper on the pieces:
Credit. This lesson re-teaches Lecture 2.1 of MIT MAS.S60 / 6.S985, Modeling Multimodal AI (Spring 2026), from the MIT Multisensory Intelligence group led by Paul Pu Liang. The first part, Thinking about Thinking about AI Projects, is by Valdemar Danry. The data-collection, preprocessing and debugging parts are by teaching assistant Edgar Morfin. See the lecture slides. The diagrams, devices, code and examples are our own.
Now press Present or Teach and explain, from memory, how you would debug a run whose loss turns to NaN at step 400. Then go back to the broken runs at the top, pick each symptom, and say aloud which check you would run first and why.