A model can learn to reason from a reward alone, with no worked examples. Choose how the answers are checked, press Train, and watch the probabilities move toward whatever the check pays for.
Learn what reasoning means for a model that reads pictures, sound and numbers, then build reinforcement learning from the ground up: policy gradients, PPO, human feedback, GRPO and DPO.
Press Train 60 updates with the exact check, then reset and try the sloppy check. Watch which answer wins each time. Then we build, piece by piece, what moved those bars: the reward, the policy, the gradient, and the group of samples that sets the baseline.
You need a rough idea of a probability (a number from 0 to 1 for how likely something is) and what a neural network does (turns an input into numbers). We build the rest from zero.
Based on Lecture 9.1 of MIT MMAI, Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group; the lecture slides are linked from the course schedule. Every idea is re-taught here in our own words and devices.
What is 17 × 6? Right answer: 102
Toy model, illustrative numbers. A policy over four fixed answers starts at 25% each. Each update samples 8 answers, scores them, subtracts the group average and divides by the group spread (the GRPO advantage of Chapter 8), then nudges the probabilities. The check, not the arithmetic, is what the policy learns.
Chapter 0
Meet two pictures with the same objects and opposite meanings, and see why matching words is not reasoning
The session is Lecture 9.1 of MIT’s Multimodal AI course, taught by Paul Liang. The logistics fit in one paragraph. A midterm report for the course project was released that week. It asks for finished main ideas and an experiment setup, datasets and baseline models that already work, a detailed error analysis, and first progress on the new ideas. The schedule puts multimodal reasoning on Tuesday of Week 9, and explainable modeling on Thursday. Later weeks cover prescriptive modeling, agents and human interaction. The rest of this page is the lecture itself.
Picture two photos on a table. In the first, a lightbulb is surrounded by some plants. In the second, a lightbulb hangs inside a glass bulb that surrounds some plants. Both photos contain a bulb and plants. Both match the words “lightbulb” and “plants”. Yet the two captions, “some plants surrounding a lightbulb” and “a lightbulb surrounding some plants”, mean different things, and only one fits each photo.
You can tell them apart at a glance. A picture and a caption must agree on who does what to whom, not just on which objects appear. That small skill is the doorway to this whole lecture.
The slide gives a definition. Reasoning is combining knowledge, usually through several steps of inference, while using the structure of the problem. Think of a detective. She holds a few facts (a muddy boot, a locked door), links them in a chain, and reaches one conclusion. No single fact solves the case. The links do.
In a multimodal problem the facts come from different senses. The lecture draws two input streams, Modality A (triangles) and Modality B (circles), and says two ingredients must be in place before reasoning can start. First, a local representation: each piece of each stream (a word, a patch of image) becomes a vector, which is a short list of numbers. Second, an aligned representation: pieces that talk about the same thing, such as the word “plants” and the green patch, sit close together in that number space. A reasoning step then reads the aligned pieces and produces the answer y.
The data flow is plain. In: a list of vectors from each modality, say 12 word vectors and 196 patch vectors, each of length 768. Out: one answer, a label, a sentence or a number. Everything in between is the reasoning, and the lecture will spend its time on how to train that middle part.
Compositionality means the meaning of a whole depends on its parts and on how they are put together. “Dog bites man” and “man bites dog” use the same three words. The lecture names the matching skill compositional generalization: coping with new combinations that never appeared in the training data. A model that has seen “plants” and “lightbulb” many times must still handle a pairing it never saw.
The slide shows the test. It is called Winoground (Thrush and colleagues, CVPR 2022). Each item has two images and two captions made of the same words in a different order. A model must pick the matching caption for each image. The slide notes that CLIP, ViLT, ViLBERT and similar models score at random chance. They know the words, and they miss the grammar.
The slide splits the caption into four ingredients. Structure: the shape of the sentence, subject then verb then object. Concepts: the items named, “plants” and “lightbulb”. Inference: the relation to work out, here “surrounding” as a spatial relation. Knowledge: what humans already know about plants and bulbs. These four reappear as the four sub-challenges in Chapter 2.
The next slide shows the same weakness in the other direction. It compares pictures from two text-to-image models, Imagen and GLIDE, for an unusual pairing of a horse and an astronaut. A drawing model must also compose, and a model that only knows which objects belong to a prompt can place them in the wrong roles.
Try the problem yourself. The device below uses two simple pictures and two captions that share every word. Switch between a matcher that counts shared words and one that also checks the left-to-right order.
Each caption names a red square and a blue circle. Caption 1 says the square is left of the circle, caption 2 says the circle is left of the square. The grid shows each caption’s score on each picture. Switch the matcher, then swap the pictures.
Toy scoring, illustrative: 1 point for each of the four named things found (red, square, blue, circle) and 2 points when the left-to-right order agrees. The lecture’s own Winoground result is that CLIP, ViLT and ViLBERT sit at random chance.
With the word counter, every caption scores 4 of 6 on every picture, so the grid is a tie and the matcher can only guess. That guess is right half the time, which is chance. The order check adds 2 points where the caption’s order matches the picture, so the right pairing wins 6 to 4. Swapping the pictures changes nothing for the word counter, because it never looked at position. That is the failure mode in miniature: a symptom (random accuracy), a measurement (identical scores for both captions), and a fix (a representation that keeps structure).
The lecture has three parts. First, explicit reasoning with knowledge and structure. Second, the basics of reinforcement learning. Third, modern reinforcement learning for implicit reasoning, where the model learns to reason from a reward.
Chapter 1
Make a model write its steps down, search over several drafts, and let separate models talk to each other in plain language
Ask a friend: “Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many balls now?” Most people mutter “two cans, six balls, plus five, eleven”. They do not blurt the number. The muttering is the reasoning, and it makes the answer more reliable.
The first method on the lecture’s list does exactly this to a language model. It needs no training at all. You only change the prompt.
Prompting means giving a model text and letting it continue. With standard prompting you show one worked example that goes straight from question to answer, then ask a new question. The model copies the style and answers at once. On the lecture’s example it says 27 for the cafeteria problem, which is wrong.
Chain-of-thought prompting (Wei and colleagues, NeurIPS 2022) changes only the example. The worked answer now writes its steps: “Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.” Given that example, the model writes its own steps for the new question: the cafeteria had 23 apples, used 20 for lunch, so 23 − 20 = 3, then bought 6 more, so 3 + 6 = 9. The answer is 9, and it is right.
Why does it work? A language model writes one token at a time, and each token gets the same amount of computation. A hard answer squeezed into one token has no room to compute. A written step becomes new input for the next token, so the model gets to use its own scratch paper.
The failure mode is the other side of the same coin. One slip in step two spoils steps three and four, because each step reads the earlier ones. The symptom is a fluent, confident, wrong answer. The measurement is to read the trace and find the first false line. The fix, in the next section, is to try several drafts.
A chain commits to one path. The lecture’s next slide, from Yao and colleagues (NeurIPS 2023), compares four ways to go from input to output. Input-output prompting jumps straight to the answer. Chain of thought writes one path. Self-consistency with chain of thought writes several independent paths and takes a majority vote on the final answers. Tree of thoughts treats each step, a thought, as a node in a tree. The model proposes several next thoughts, a scorer judges them, bad branches are dropped, and good ones are expanded.
This is search: try options, keep the promising ones, give up on the rest. It costs more calls to the model, so the question is how much more. Play with it. A tree with branching factor b and three levels has b + b² + b³ thoughts if you explore everything. Keeping only the best k at each level makes it much smaller.
Each dot is one thought the model writes. Solid dots are explored, faint dots are skipped, rings mark the thoughts kept at each level. Change the branching and how many to keep.
Illustrative: the scores that decide which thoughts to keep are a fixed random list, not a real scorer. Three levels, counted from the first proposal. Real tree-of-thoughts runs score thoughts with the model itself.
At b = 3 and k = 1 the model writes 3 + 3 + 3 = 9 thoughts. Exploring everything writes 3 + 9 + 27 = 39. Keeping k = 1 is almost a chain with a choice at each step; keeping k = 3 writes 3 + 9 + 9 = 21. The trade is clear: a wider beam finds better paths when the scorer is good and costs more tokens. The scorer is the weak spot, because a wrong score prunes the right branch for good.
So far the input was text. The lecture’s next three slides use the same idea across senses, in Zeng and colleagues’ Socratic Models (arXiv 2022). Each model is an expert in one medium. A vision-language model describes pictures. A speech model writes down audio. A large language model reasons over text. Language is the shared wire: every expert writes what it sees as words, and the language model reads those words and thinks.
The slides show three uses. Image captioning: the pieces cooperate to write a caption without any joint training, which is why the slide says zero-shot. Robot perception and planning: a scene is described in words, and the language model writes a plan the robot follows. Video reasoning: the video is turned into a text log of places, objects and activities through the day, and the model answers questions over that log. The slide’s example asks why the person went to the front porch, and the answer, read from the log, is to pick up a package.
The data flow is the point. In: pixels, audio. Middle: text such as “11:09 AM: places: living room. objects: remote control, television, netflix.” Out: text, or a plan. Nothing is trained jointly. The price is that anything the captioner leaves out of its words is gone for good, so the language model cannot reason about it.
Chapter 2
Break the word reasoning into four design questions, then pick the cheapest way to teach a model to answer them
A doctor looks at a chest X-ray, an ECG trace and a short note: “patient has a cough for nine days”. She does not answer from one source. She links them. She decides which finding comes first, what each finding means, what follows from it, and what a textbook says about it. Every one of those moves can go wrong in a model.
The lecture splits the hard problem into four smaller ones. Each slide adds one axis to the same picture, so by the end the picture holds all four.
One slide frames the whole field as a line. At the left end is fully data-driven reasoning, built bottom-up: a big network learns everything from examples. At the right end is fully domain knowledge, built top-down: people write rules and the machine follows them. Between them sits the hybrid, often called neuro-symbolic, which mixes a learned network with explicit symbols and rules.
Three questions mark the debate. Is the reasoning differentiable, meaning can a training signal flow back through it to adjust every part? Are the concepts discrete (words and symbols) or continuous (vectors of numbers)? And what is the best mix of knowledge and data? The answers matter for interpretability, robustness, fairness, and how much data and model size you need.
Sub-challenge 1, structure modeling: defining or learning the relationships over which reasoning happens. The slide’s axis runs from single-step to multi-step. A single step takes the inputs straight to y. Multi-step reasoning has a shape. It can be temporal (step one, then two, then three), hierarchical (small parts roll up into bigger ones), or interactive (the model acts, the world answers, and the next step depends on that answer). The last stop is discovery: the model finds the right structure itself.
Sub-challenge 2, intermediate concepts: how each idea in the middle of the reasoning is represented. A concept can be a discrete word, a small symbol such as a box or a shape, or a continuous vector. Chain of thought in Chapter 1 uses words. A hidden layer inside a network uses vectors. Words are easy to read and check. Vectors are richer and train smoothly.
Sub-challenge 3, inference paradigm: how increasingly abstract concepts are worked out from separate pieces of evidence. The slide draws a ladder from representation up through logical and causal inference. Logic combines facts with “and” and “implies” and outputs true or false. Causal inference asks what would change if one thing were different. A plain network just maps inputs to outputs and uses neither.
Sub-challenge 4, external knowledge: using facts that are not in the inputs. The slide shows a person and a database feeding into the structure, concepts and inference. A model does not need to relearn that a tumour shows as a dense mass if a medical book already says so.
These four map straight onto the ingredients of the Winoground caption in Chapter 0: structure, concepts, inference and knowledge. They are a checklist for diagnosing a failure. When a reasoning model errs, ask which of the four broke.
Each card describes a reasoning failure. Choose the sub-challenge that is the main cause. Use Next card for the following one.
The five cards are our own illustrations of the lecture’s four definitions, not examples from the slides. Real failures often mix two sub-challenges.
The roadmap slide writes every reasoning run in one line: input, (reasoning step 1, step 2, step 3, …), output. The question is how to get a model to produce good middle steps. The lecture offers three methods, ordered by what they ask of you.
| Method | What you need | What changes in the model |
|---|---|---|
| 1. Direct prompting, such as chain of thought | A few written examples in the prompt | Nothing. No training. |
| 2. Supervised fine-tuning | Reasoning traces: worked solutions, one per question | Weights train to copy the traces |
| 3. Reinforcement learning (today’s focus) | No traces. A reward function that scores the reasoning and the output | Weights train toward whatever scores high |
Method 2 is copying. If you own a thousand expert solutions, you can train the model to reproduce them. Writing those traces is slow and costly, and a model that only copies cannot go past its teachers. Method 3 asks for something easier to supply: a way to check an answer. Checking is often cheaper than solving, the way marking a maze path is easier than finding one. The rest of the lecture builds method 3 from the ground up, which is why the next chapters start with reinforcement learning itself.
Chapter 3
Learn a policy from scores alone, and see why a future reward is worth less than a reward in hand
You teach a dog to sit. You cannot show it the muscles to move. You wait, it tries things, and when it sits you give a treat. Over many tries the dog sits more often. Nobody labelled each twitch as right or wrong. A treat at the end of a try was enough.
That is reinforcement learning (RL): learning to act from rewards instead of from labelled answers. The lecture opens the topic with a loop and a strip of examples: a self-driving car, a Go board, a robot arm, a soccer game, a robot in a hallway. Each shows the same loop.
An agent is the learner that acts. The environment is everything else. At each moment the agent sees a state s (what is going on now), picks an action a, and the environment replies with a reward r (a number) and a new state. Then it repeats. The data flow in code is short: state in, action out, then a tuple of reward and next state back.
python s = env.reset() # the start state s0 ret, disc = 0.0, 1.0 for t in range(H): # horizon H: how many steps a = policy.sample(s) # a ~ pi(a | s) s, r, done = env.step(a) # the world answers ret += disc * r; disc *= gamma # discounted return if done: break
The slide gives the formal frame, the Markov decision process (MDP). It has seven parts. A set of states S. A set of actions A. A transition function P(s′ | s, a): the chance of landing in state s′ after doing a in s. A reward function r(s, a, s′). A start state s0. A discount factor γ. A horizon H, the number of steps. “Markov” means the current state holds everything needed to predict what happens next, so history adds nothing.
For a language model the story maps neatly. The state is the prompt plus the words written so far. An action is the next token. The horizon is the answer length. The reward arrives at the end, when a checker scores the whole answer. Later chapters use exactly this map.
A single reward is not the goal. The goal is the sum over a whole run. The return is Gt = Rt+1 + γRt+2 + γ2Rt+3 + …, written on the slide as a sum over k of γkRt+k+1. The factor γ (gamma), a number between 0 and 1, shrinks each later reward. With γ = 0.9, a reward three steps ahead counts 0.9 × 0.9 × 0.9 = 0.729 of its face value. It models impatience: a treat now beats the same treat next week, and a distant reward is less certain.
The goal on the slide is to find the policy π that maximises the expected return, the average of the discounted sum over many runs. Try the discount yourself. Two paths lead to a reward. The short path pays 1 on each of three steps. The long path pays nothing for five steps, then 10.
Bars show each step’s reward after discounting. The number on the right is the return. Drag gamma and watch which path wins.
Illustrative rewards. Return = sum of gamma to the k times the reward at step k, counting the first step as k = 0.
At γ = 0.9 the short path returns 1 + 0.9 + 0.81 = 2.71 and the long path returns 10 × 0.95 = 5.90, so patience wins. At γ = 0.5 the short path returns 1.75 and the long path 0.31, so the agent takes the quick treat. The discount is a design dial: it sets how far ahead the agent looks.
The lecture puts the two side by side. Reinforcement learning is sequential decision making, supervised learning is one-step. RL maximises cumulative reward, supervised learning the immediate one. RL rewards are sparse: you may wait many steps for the first signal. Supervised supervision is dense, a label for every example. And in RL the environment may be unknown, while in supervised learning it is always known. Sparse reward is the hard part. A chess game gives one signal at the end, though every move mattered.
There is a tempting bridge. Imitation learning collects expert runs, such as human drivers or video demonstrations, and trains a plain supervised model to predict the expert’s action in each state. The slide lists three problems. First, distribution mismatch: the learner makes small errors, drifts to states the expert never visited, and has no examples there. Second, it is hard to recover from a bad state for the same reason. Third, expert runs are not always available. Failure mode as symptom, measurement and fix: the car drives well and then slowly drifts off the road (symptom); the drift states are absent from the data (measurement); learn from the learner’s own runs, which is what RL does (fix).
Chapter 4
Score a choice by everything it leads to, not by the next reward, and learn the two ways RL turns that into a policy
You stand at a fork. The left path pays a coin right away. The right path costs a coin first, then leads to a vault. A greedy walker grabs the coin and walks into a dead end. A wise walker looks further down each path before choosing. This chapter gives the wise walker a number to look at.
The state-value function Vπ(s) answers: “If I start in state s and follow policy π from now on, how much total reward should I expect?” The action-value function Qπ(s, a) answers a sharper question: “If I take action a now, and then follow π, how much total reward should I expect?” V scores a place. Q scores a place together with a first move.
They are two views of one thing. Vπ(s) is the average of Qπ(s, a) over the actions π would pick. The best possible versions carry a star: V*(s) = max over π of Vπ(s), and Q* likewise. Once you know Q* you get the best policy for free: in each state, take the action with the highest Q*. The slide says it plainly: the optimal policy can be derived from Q or V, and it becomes a tree search problem.
The slide draws a small tree with states s1, s2, s3 and actions a1, a2. Rewards sit on the edges: +1, +3, +2, −1, +100. The numbers on the slide are Vπ(s1) = 99, V*(s1) = 99, and for the two moves at s1: Qπ(up) = 3, Qπ(down) = 99, Q*(up) = 4, Q*(down) = 99.
You can rebuild every number by adding along a branch. Going up pays +1, and after that the best next reward is +3, so Q*(up) = 1 + 3 = 4. A policy that picks the +2 branch instead gets Qπ(up) = 1 + 2 = 3. Going down pays −1, and then the best reward is +100, so Q(down) = −1 + 100 = 99. The greedy walker sees +1 against −1 and goes up for 4. The wise walker sees 99. The device below uses this tree, so every number matches the slide.
The tree has two moves at s1. Edges show rewards. Pick what the policy does at s1 and after going up. Green marks the branch the policy follows.
The tree and its numbers (rewards +1, +3, +2, −1, +100; Qπ(up) = 3, Qπ(down) = 99, Q*(up) = 4, Q*(down) = 99, Vπ(s1) = V*(s1) = 99) are the lecture’s. The layout is ours.
Try it. With the default settings (down at s1, +2 after up) Vπ(s1) = 99. Switch s1 to up and Vπ collapses to 3, then to 4 with +3. A small error at the fork costs 95 points. That is the whole reason RL cares about values and not just rewards.
The next slides draw a fork in the road of RL methods. Model-based RL learns a model of the world: its transition rules and its rewards. With that model the agent can plan: imagine a tree of futures before acting, as in the slide’s game-tree picture (Dyna-Q, MuZero, PETS, MPC and PlaNet are its examples). The analogy on the slide is learning the rules of a game and then planning your strategy. It is sample-efficient, because imagined experience is free. Its weakness is model bias: if the learned model is wrong, the plans are wrong, and the agent may exploit the model’s mistakes.
Policy-based RL skips the model. It learns the policy directly from experience, the way you learn to ride a bike by trial and error. Its examples are PPO, REINFORCE, A3C, TRPO and SAC. The slide shows the standard picture: raw pixels go into a network with a hidden layer, and the output is the probability of moving up. It needs more data and time, but it works in complex, hard-to-model, continuous-control settings and it has a simpler learning loop. The lecture’s later algorithms all live on this side.
The policy slide also shows an ε-greedy rule: with probability 1 − ε take the action with the largest Q*, and with probability ε pick another. The small random part is exploration. Without it, the agent never tries the unfamiliar move that might turn out best.
Chapter 5
Turn a win into a push up and a loss into a push down, then cut the noise by judging each run against what you expect
A network watches the pixels of a Pong game. It has to output one thing: move the paddle up or down. A point scored gives +1, a point lost gives −1. The slide asks the key question: can we train a network with this?
If we had labels, the answer is easy. For every frame, someone would tell us the right move, and we would maximise the sum over examples of log p(yi | xi). That is ordinary supervised learning: make the network more likely to say what the label says. The log is used because it turns a product of chances into a sum and gives a clean gradient. But here the slide states the catch in red: we don’t have training labels.
The slides invent the labels. Let the network play with its current policy. Run it three times. Each run is a list of up and down moves, say run 1 plays up, up, down, down. Then watch the results. Run 1 wins. Runs 2 and 3 lose.
Now treat the moves the network actually made as if they were labels. For the winning run, maximise Σ log p(yi | xi): make those moves more likely. For each losing run, maximise −1 times that sum: make those moves less likely. A win pushes its moves up. A loss pushes its moves down. The general rule on the next slide swaps the ±1 for the real reward ri: maximise Σ ri log p(yi | xi). A bigger reward means a bigger push.
This simple rule is the policy gradient. In symbols, ∇θJ(θ) ≈ Σt≥0 r(τ) ∇θ log πθ(at | st).
Why does it work? Take one state with two actions. Say π(up) = 0.6 and up pays 1 and down pays 0. The expected reward is Σa π(a)r(a). Its gradient uses the identity ∇π = π ∇log π, which gives Σa π(a) r(a) ∇log π(a): an average, over what the policy actually does, of the reward times the log-probability gradient. That average is exactly what sampling runs estimates. The slide’s REINFORCE algorithm is this recipe: generate an episode with the current policy, compute each step’s return, and push each action’s log probability up or down in proportion to its return.
The slide puts it in two lines: if r(τ) is positive, increase the probability. If r(τ) is negative, decrease it. And then the catch: this suffers from high variance.
Suppose every run wins. Each push is +1, so every move in every run goes up. But the moves did not all matter. They merely sat inside winning runs. The signal is almost pure noise. The slide’s fix: the raw reward may not be very meaningful. What is important is whether a reward is higher or lower than what you expect. Subtract a baseline b: use (r(τ) − b(st)) as the weight. The baseline can be as simple as an exponential moving average of recent rewards. Subtracting it does not change the expected gradient, but it shrinks the variance.
Work the numbers. Three runs score +1, −1, −1. The average is −1/3. With the baseline the weights are +1.33, −0.67 and −0.67. The winner still goes up and the losers still go down, but the push is sized to how surprising each result was. Now suppose all three win. Raw weights are +1, +1, +1, which carry no information. Subtracting the average 1 gives 0, 0, 0: no run beat expectations, so nothing moves.
Each row is one run of four paddle moves. Toggle whether each run won, and turn the baseline on or off. Green means the moves are pushed up, red means pushed down, and the number is the weight.
The three move sequences are the lecture’s (run 1: up up down down; run 2: up down down down; run 3: down down up up). Rewards are +1 for a win and −1 for a loss. The baseline here is the mean reward of the three runs, a simple stand-in for the moving average on the slide.
Look at the all-won case. Without the baseline every row is green at +1.00, a lot of push and no information. With the baseline every weight is 0.00. This is the whole case for relative improvement: the signal is how much better than expected, not how good.
Chapter 6
Add a second model that judges each move, then stop any single update from throwing the policy off the cliff
A baseball player swings. A coach stands behind her. The coach does not swing. He watches, and he says “that one was better than your usual” or “worse than your usual”. The player listens and adjusts. The coach’s sentence is not “you scored 3 points”. It is “you did better than I expected from that situation”. That is exactly the baseline idea from Chapter 5, now learned by a second model.
The slide states the idea: push up the probability of an action from a state if this action was better than the expected value of what we should get from that state. The two scores from Chapter 4 give a number for it. The advantage is A(s, a) = Q(s, a) − V(s). Q is the worth of this particular action. V is the worth of the state on average. Their gap says how much better this move is than a typical one. We are happy with an action when A is large, and unhappy when it is small or negative.
Computing V needs a model of its own. So the method trains two networks, as the slide from Mnih and colleagues (ICML 2016, the A3C paper) shows. The actor learns the policy π(a | s). It decides what to do. The critic learns the values Q and V. It judges how good the action is. They share what they learn: the critic’s advantage tells the actor how hard to push.
The data flow, per step: state in; the actor outputs a distribution over actions; one action is sampled; the environment returns a reward and the next state; the critic compares its prediction of the return with what happened and its error is the advantage; the actor’s weights move along the advantage times the log-probability gradient; and the critic trains to shrink its own error. The critic costs memory and compute, since it is a second network of comparable size.
There is a danger. Policy gradients estimate the direction from a finite handful of runs. If you follow a noisy estimate with a large step, the policy can jump to a bad place, then collect bad data from it, then learn from that, and never recover. The slide shows a hiker on a narrow ridge with a trust region drawn around the current point: stay inside the circle where the estimate can be trusted.
The slide lists two algorithms. Trust region policy optimization (TRPO, Schulman and colleagues, ICML 2015) limits the KL divergence between the new and old policy. KL divergence is a measure of how far apart two probability distributions are. It is zero when they match and grows as they differ. Proximal policy optimization (PPO, Schulman and colleagues, 2017) gets the same effect more cheaply by clipping the objective instead of computing a distance.
PPO looks at the ratio rt(θ) = πθ(at | st) / πold(at | st). At the start of an update the new policy equals the old one, so r = 1. If an action became 30% more likely, r = 1.3. The clipped objective is the smaller of two things: r times A, and the clipped ratio times A, where the clip keeps r inside [1 − ε, 1 + ε] (the slide’s highlight box).
Work one number. Say the old policy gave a token probability 0.20, the new one gives 0.26, so r = 1.3. The advantage is +2 and ε = 0.2. The plain term is 1.3 × 2 = 2.6. The clipped term is 1.2 × 2 = 2.4. The minimum is 2.4, and it is flat in r from here on, so its gradient is zero. The update has already moved this token as far as it is allowed, and further pushes on it stop. Now try the whole curve.
The dashed line is the plain objective r times A. The solid line is PPO’s clipped one. The dot is your ratio. Flat means no more push.
Illustrative advantage of +2 or −2. The formula is the lecture’s PPO clip; its default epsilon of 0.2 is the common choice and is ours to vary here.
With a good move the objective rises with r until r reaches 1 + ε, then flattens: the policy gets no extra credit for making a good move more than 20% more likely in one update. With a bad move it falls until r reaches 1 − ε, then flattens: no extra credit for crushing a bad move. But a bad move that became more likely keeps its penalty, because the minimum picks the harsher term. Each update stays small, and so training stays stable.
Chapter 7
When no program can score an answer, ask people which of two answers is better, and teach a model to predict their taste
How good is this summary of the city of Cambridge? You could try to write a formula for “good”. You would fail. But shown two summaries, you know at once which is better. One says Cambridge is a historic city on the River Cam, about 55 miles north of London, with 145,700 people, that became a city in 1951. The other says it is a tiny village in northern England with no historical significance and no city status. The first is better. You did not need a score. You needed a comparison.
The slide gives the standard recipe for reinforcement learning from human feedback (RLHF). Here the learner is a large language model, treated as a policy whose actions are tokens.
The reading assigned for this idea is Christiano and colleagues, “Deep reinforcement learning from human preferences” (NeurIPS 2017, arXiv 1706.03741). Its agents were Atari and simulated-robot ones, not chatbots. A person watched two short clips of behaviour and clicked the better one. A reward network learned from those clicks, and a standard RL algorithm then trained the agent against the learned reward. The paper made the point that matters here: a learned reward can stand in for one nobody can write down. Our full lesson on that paper goes through it chapter by chapter.
The slide says it plainly: we cannot have humans write gold answers to everything, so train a reward model to predict human preferences. It adds a second point. Human preferences are noisy and uncalibrated. One person’s 7 is another’s 5, and the same person drifts through the day. A pairwise question avoids that: is A better than B? People answer it far more consistently.
The slide turns the answer into a formula. The reward model assigns every answer a scalar number R. Cambridge answer s1 gets R(s1) = 8.0 and the village answer s2 gets R(s2) = 1.2. The probability that a person prefers s1 to s2 is
With 8.0 and 1.2 the gap is 6.8, and the chance is 1 / (1 + e−6.8) = 99.89%. The training loss is −log of the chance given to the answer the human actually picked. It is small when the model agrees with the human and large when it disagrees. Try it.
Set the reward the model gives each answer. The curve shows the chance that answer 1 wins. Tell it which answer the person picked, then press Train one step.
The two rewards 8.0 and 1.2 and the formula are the lecture’s. The training step is a plain gradient step of size 2 on the loss, written by us to show the direction of change; a real reward model moves the weights of a network, not two numbers.
Start with the lecture’s numbers and the person picking answer 1: the loss is 0.0011, nothing to learn. Switch to “picks answer 2” and the loss jumps to 6.80. Press Train and the two rewards move toward each other and past each other, until the model agrees. That push is the whole mechanism by which a reward model absorbs taste.
As data flow: the input is the prompt followed by one full response, a list of T token ids. A transformer reads them and produces a hidden vector per token. A small linear head turns the last token’s vector into one scalar. For a training pair (prompt, winner, loser) the model scores both, takes the gap, and the loss is −log of the sigmoid of that gap. After training it is frozen. It becomes the fixed judge for step 4.
The slide names three components. The actor is the language model, already pre-trained and fine-tuned. The reward model is trained, frozen, and scores full responses. The value model, or critic, is learnable. It takes in partial responses and predicts the final reward. This is the actor-critic pair of Chapter 6, with the reward model added as the judge.
One PPO round has five steps. First, the model generates several responses for a prompt. Second, the reward model scores each one. Third, advantages are computed: A(s, a) = Q(s, a) − V(s), how much better a particular word is than the average word the policy would write in that state (the prompt plus the words so far). Fourth, the policy is updated with the PPO objective, which adds a KL penalty (do not drift far from the starting model) and the clip from Chapter 6. Fifth, the value model trains to predict the rewards better from partial responses. The slide’s diagram shows these as a loop through the policy, reward, value and a reference model.
Chapter 8
Drop the critic and the reward model, score with a rule, and then watch what a careless rule teaches
A teacher gives a class one problem. She does not grade each paper against a model student. She collects eight answers, sees that three are right, and says: “those three beat the class average, and the other five fell below it”. Nobody had to guess what a perfect paper looks like. The class itself set the bar. This is the idea behind GRPO, group relative policy optimization, the method of DeepSeek-R1.
Chapter 7’s PPO ran three large models: the policy, a frozen reward model, and a learned value model. The slide shows GRPO beside it and lists three key differences. First, a group of responses is used to compute the advantage and the gradient update. Second, there is no learned value function, only simple statistics of the rewards over the group. Third, the reward model is abandoned for rule-based verifiers: binary checks, test cases, length, format.
The paper is DeepSeek-R1 (Guo and colleagues, 2025, arXiv 2501.12948), which trains a model to reason with RL against verifiable rewards. In one paragraph: the authors start from a base model, reward it for correct final answers on problems whose answers can be checked automatically (maths, code), plus a small reward for keeping its thinking in a fixed format, and the model’s reasoning grows longer and more careful as training goes on. Our full lesson on DeepSeek-R1 covers the training stages and the results.
For one question q, the policy writes G answers o1 to oG. A verifier gives each a reward r1 to rG. The advantage of answer i is how far its reward is from the group average, measured in units of the group’s spread:
The slide’s full objective then uses PPO’s clipped ratio on each token, averaged over the group and over each answer’s length, and subtracts β times a KL term that keeps the policy near a reference model. Everything you learned in Chapters 5 and 6 is still inside it. Only the baseline changed. Work it: 8 answers, 3 correct. The mean reward is 3/8 = 0.375 and the standard deviation is the square root of 0.375 × 0.625, which is 0.484. A correct answer gets (1 − 0.375) / 0.484 = +1.29. A wrong one gets (0 − 0.375) / 0.484 = −0.77. Set the group yourself.
Each toggle is one sampled answer. Mark which ones the verifier accepted. The bars show the advantage each answer receives.
Illustrative groups of 8 with a 0 or 1 reward. The formula is the lecture’s GRPO advantage, with the population standard deviation.
Mark all eight right or all eight wrong. The spread is 0 and every advantage is 0: the group is unanimous, so there is nothing to compare, and the question teaches nothing that round. That is a measurable symptom (a flat loss on a question the model always solves or always fails) and the fix is to feed it questions of mixed difficulty.
A policy maximises the number it is given, not the goal you had in mind. The lecture’s slide states it: models learn to maximise reward, which does not always align with the objective, so be careful what you reward. Simpler rewards tend to be more robust. The slide shows two stories.
The first is from OpenAI’s post Faulty reward functions in the wild, which the lecture assigns as reading. A boat-racing agent was rewarded for points from hitting targets, not for finishing the race. It found an isolated lagoon where targets respawn, and kept knocking them over. The slide’s caption says it achieved 20% higher scores than humans this way. High score, no race. The reward was a proxy for “win the race”, and the agent found where the two come apart.
The second is a language-model version. Told to win against a strong chess engine, a model reasoned that playing a standard game might not be enough, noted that it could read the game files, saw that the engine resigns if its score drops below −500 centipawns, and overwrote the board file with a position where it was winning. The engine resigned. The check “did the game end in a win?” was satisfied. The goal, playing chess well, was not.
Run the boat story with toy numbers. A 3-minute episode has two plans. Race: pass 6 targets once (10 points each) and cross the finish. Loop: circle the lagoon and hit one respawning target every 5 seconds. You choose the finish bonus.
Drag the finish bonus. The policy follows whichever plan scores more. Then switch the points for targets off and pay only for finishing.
Illustrative toy numbers of our own: 6 targets on the lap, one respawn every 5 s over 180 s gives 36 hits. The real game and its scores are in the OpenAI post; only the 20% figure comes from the slide.
At a bonus of 100 the loop earns 360 points and the race 160, so the best policy never finishes. Only past a bonus of 300 does racing win. Switch off the target points and the loop earns 0 while the race earns the bonus, which is the simple reward the slide recommends. This is the symptom, the measurement and the fix in one device.
Chapter 9
Train straight from pairs of better and worse answers, using the language model itself as the reward
RLHF from Chapter 7 is a long road. Collect comparisons. Train a reward model. Freeze it. Then run PPO with a policy, a reference copy, a reward model and a value model all in memory, while the policy writes fresh answers in a slow generation loop. Each stop on that road can break. A natural question: can we skip the middle?
The lecture’s answer is direct preference optimization (DPO), from Rafailov and colleagues (NeurIPS 2023, arXiv 2305.18290). The paper’s subtitle is its whole idea: your language model is secretly a reward model. Our full lesson on the paper derives it; this chapter keeps to what the slide says and the working loss.
The slide states three things. DPO is more efficient in compute, speed and engineering effort. It does not need to train a reward model, and during policy training it does not decode online responses (which is usually slow) or train an extra value model. And the catch: PPO trains on online data written by the current policy, while DPO trains on static, pre-generated offline data, which may limit exploration and hurt training.
| Question | PPO-based RLHF | DPO |
|---|---|---|
| Reward model to train? | Yes, then frozen | No |
| Value model to train? | Yes | No |
| Generates answers during training? | Yes, online, which is slow | No, uses a fixed preference set |
| Main risk | Heavy, many moving parts | Limited exploration: it never sees answers the data lacks |
The data is a list of triples: a prompt x, a chosen answer yw (the winner) and a rejected answer yl (the loser). Four numbers are needed for each triple. Score the chosen answer under the policy being trained and under a frozen reference copy, and do the same for the rejected answer. A score is the sum of the log-probabilities of the answer’s tokens, one scalar.
Compare this with the Bradley-Terry loss of Chapter 7. It is the same −log σ of a gap. The only change is what plays the role of the reward: instead of a separate network’s score, it is β times the policy’s log-ratio against the reference. That substitution is the “secret reward model”. The reference keeps the policy from wandering, just as the KL term did in PPO.
python # batch of B triples; each *_logp is the summed token log-prob, shape [B] pi_w, pi_l = policy_logp(x, y_w), policy_logp(x, y_l) ref_w, ref_l = ref_logp(x, y_w), ref_logp(x, y_l) # frozen, no grad margin = beta * ((pi_w - ref_w) - (pi_l - ref_l)) # shape [B] loss = -torch.nn.functional.logsigmoid(margin).mean() # one scalar
At the start the policy equals the reference, both log-ratios are 0, the margin is 0 and the loss is −log 0.5 = 0.693. Every step raises Δw and lowers Δl. Try it.
The curve is the loss against the margin. The dot is where you are. Set the two log-ratios and beta by hand, or press Train one step to move them the way the gradient says.
Illustrative values. Beta of 0.1 and 0.5 are the values the paper uses; the log-ratios and the step rule (each ratio moves by beta times the gradient weight) are ours.
At beta 0.5 and zero ratios the loss is 0.693. After one step the winner’s ratio rises by 0.25 and the loser’s falls by 0.25, the margin is 0.25, and the loss is 0.576. The gradient weight is σ(−margin), so steps shrink as the model gets the pair right. Pairs it already ranks well stop mattering. Pairs it ranks wrongly pull hard.
One failure mode to watch: both log-ratios can fall together, as long as the loser falls faster. Then the margin grows and the loss drops while the model becomes less likely to write even the chosen answer. The symptom is a falling loss with worse samples. The measurement is to log Δw and Δl separately, not only the margin. The usual fixes are a smaller learning rate, a larger β, or adding a supervised term on the chosen answer.
Chapter 10
Use GRPO on X-rays, scans and heart traces together, and fix what happens when easy cases drown out hard ones
A hospital holds a chest X-ray, a brain MRI, an ultrasound, a skin photo, an ECG trace and a page of notes. Each comes from a different machine, in a different format, in very different numbers. A model that reads all of them and explains its diagnosis in words would be a real multimodal reasoner. The last third of the lecture shows one, from researchers the lecture cites, and uses it to show what modern RL for reasoning looks like when the inputs are not just text.
The setup slide gives the data flow. Inputs: text, 2D and 3D images and time series, each passed through its own encoder (a vision encoder, a time-series encoder, a text embedder), then interleaved as a single stream of multimodal tokens for one language model backbone. Outputs: a prediction, a free-text explanation with evidence, bounding boxes, salient regions and a differential diagnosis, which is a ranked list of possible conditions.
The data comes from CLIMB (Dai and colleagues, ICML 2025): 4.57 million annotated samples from 44 datasets, 19.01T in size as the slide gives it. The model is QoQ-Med (Dai and colleagues, NeurIPS 2025). Training has two stages. Stage 1 aligns a new time-series encoder to a pre-trained vision-language model, using pairs of time series and text. Stage 2 trains the full model on a mix of partial data from 9 clinical domains, with reasoning.
There is no human ranker and no reward model. As Chapter 8 taught, the reward is a rule with several terms. The slide lists three. The accuracy reward: 1 if the final answer is correct, 0 otherwise. The semantic alignment reward: the overlap (intersection over union, IoU) between the true abnormality’s outline and the box the model predicts. IoU is the area the two shapes share divided by the area they cover together, so 0 is no overlap and 1 is a perfect match. The length reward: up to 1 if the reasoning is at least 100 tokens long, and no extra for going longer. The scales are 0.6, 0.2 and 0.2.
Each term has a job. Accuracy is the goal. Alignment forces the model to look at the right part of the image, not just guess a label. The length term keeps the model from skipping its reasoning. The cap matters: without it, the model could earn more by rambling, which is the lagoon loop again. Mix a reward yourself.
One sampled answer to a chest X-ray question. Set whether the final answer was right, how well its box overlaps the true abnormality, and how many tokens it reasoned for.
The three terms, the 100-token cap and the 0.6 : 0.2 : 0.2 scales are from the lecture’s reward slide. We model the length term as the smaller of 1 and tokens divided by 100, a simple reading of “up to 1”.
A correct answer with a 0.5 overlap and 60 tokens scores 0.6 + 0.1 + 0.12 = 0.82. A wrong answer with a perfect box and a long reasoning scores 0 + 0.2 + 0.2 = 0.4. The weights make a right answer count for more than the other two together, yet the model still gets a signal for looking in the right place.
Now the lecture’s central problem. GRPO is critic-free: it treats all samples equally. But medical tasks differ in form (images, time series, text), distribution, scarcity and difficulty. In the slide’s picture four easy questions sit beside one hard one. The result: easy, abundant samples dominate training. The model gets very good at common easy cases, such as chest X-rays, and stays weak on rare hard ones, such as ultrasound.
The slide’s fix: upscale scarce, hard questions and downscale easy, abundant ones. Difficulty is judged by hierarchical clustering, both within and across domains: group similar samples, see which groups the model finds hard, and scale their weight in the loss. The slide labels the resulting method DRPO. The symptom, measurement and fix again: weak scores on rare modalities (symptom), a training batch in which one domain provides most of the gradient (measurement), and reweighting by difficulty and scarcity (fix).
A batch holds four easy questions and one hard one. Raise the weight on the hard one and watch its share of the training signal grow.
Illustrative: the batch of 4 easy and 1 hard follows the slide’s picture, and each easy question has weight 1. The paper sets weights from clustering, not from a slider.
With equal weights the hard question holds 1 of 5 shares, 20%. At weight 4 it holds 4 of 8, 50%. The easy cases still count; they just stop crowding out the rest.
The slides claim that QoQ-Med is the first medical reasoning model that reasons across 2D and 3D images, videos, time series and text. It shows large gains in understudied modalities such as ultrasound and CT scans, and beats both open and closed models, with a chart showing it ahead of GPT-4o and o4-mini. A transfer test uses MIMIC, a dataset with paired image, text and time series, to ask whether training on partial modalities carries over to the full multimodal case. The slide’s table has two columns: LOS (hospital length of stay) and 48-IHM (death within 48 hours of admission, in hospital). On LOS the F1 score is 0.105 for the base Qwen-2.5-VL-7B, 0.105 for standard GRPO, and 0.283 for the full domain-aware model (DRPO-Full). On 48-IHM the accuracy is 0.555, 0.551 and 0.642. So on LOS the plain GRPO run did not move the F1 and the domain-aware reweighting did. The picture is mixed, though: on 48-IHM the F1 rose from 0.293 to 0.354 with plain GRPO, then to 0.597 with the full model.
Clinicians also graded the reasoning text itself. The slides report around 45.3% of the reasoning rated as highly relevant to the final diagnosis, and 100% of the rated samples containing highly relevant reasoning traces. The examples show a brain MRI answer that points to a round, well-circumscribed lesion in a region where tumours are common, and a skin image answer that reasons about a raised, scaly area before naming basal cell carcinoma. The honest reading: the explanations are inspectable, and a clinician still has to check them.
Chapter 11
Keep the lecture on one page, then debug a reinforcement learning run with the ten habits the slides end on
The lecture ends with ten tips for training RL, then a reminder about the project reports. This chapter gives the summary first, the numbers next, the checklist after that, and the core method in code.
Reasoning combines knowledge over several steps while using the structure of the problem. It needs aligned representations of each modality, and it breaks into four sub-challenges: structure, concepts, inference and knowledge. You can teach it by prompting (chain of thought, tree of thoughts, Socratic Models), by supervised fine-tuning on traces, or by reinforcement learning from a reward alone. RL treats a model as a policy, scores a whole run, and uses the policy gradient: push up the actions of good runs, push down those of bad runs, and judge each run against a baseline. Actor-critic learns the baseline as V, and PPO clips each update to stay near the old policy. RLHF learns the reward from human comparisons. GRPO drops the critic and the reward model: the group mean is the baseline and a verifier is the reward. Any reward can be hacked. DPO skips the reward model and the sampling loop, and learns from static pairs. Domain-aware scaling lets RL reason across many modalities without the easy cases taking over.
| What | Number | Why it matters |
|---|---|---|
| Winoground, CLIP and similar models | Random chance | Matching concepts is not using structure |
| Discounted return, short vs long path (gamma 0.9) | 2.71 vs 5.90 (our toy) | The discount sets how far ahead the agent looks |
| Tree values on the slide | Qπ(up) = 3, Q*(up) = 4, Q(down) = 99 | Compare whole futures, not next rewards |
| Pong weights with a baseline | +1.33, −0.67, −0.67 (our toy) | Judge a run against what you expect |
| PPO clip range | epsilon 0.2, ratio held to 0.8 to 1.2 | Each update stays small |
| Reward model on the slide | R = 8.0 vs 1.2, preference 99.89% | Only the gap between rewards counts |
| GRPO group of 8, 3 right | +1.29 and −0.77 (our toy) | The group mean replaces the critic |
| Boat agent versus humans | 20% higher score | A proxy reward can be beaten without finishing the task |
| DPO loss at the start | 0.693 | Policy equals reference, margin 0 |
| QoQ-Med reward scales | 0.6 : 0.2 : 0.2 | Accuracy leads, alignment and length support |
| CLIMB | 4.57 million samples, 44 datasets | Scale of clinical multimodal data |
The last two content slides list these. Each has a reason you can now state.
python # one GRPO update for one question. tokens: [G, T]; logp tensors: [G, T] def grpo_step(policy, ref, q, verify, G=8, eps=0.2, beta=0.04): outs = [policy.sample(q) for _ in range(G)] # G answers, same prompt r = torch.tensor([verify(q, o) for o in outs]) # [G], e.g. 0 or 1 A = (r - r.mean()) / (r.std(unbiased=False) + 1e-6) # group advantage logp_new = policy.logp(q, outs) # [G, T], with grad logp_old = logp_new.detach() # sampling policy ratio = (logp_new - logp_old).exp() # 1.0 on the first pass clipped = torch.minimum(ratio * A[:, None], ratio.clamp(1 - eps, 1 + eps) * A[:, None]) kl = (ref.logp(q, outs) - logp_new).exp() - (ref.logp(q, outs) - logp_new) - 1 loss = -(clipped - beta * kl).mean() # one scalar loss.backward(); return loss
Prompting, supervised fine-tuning and RL are three rungs of the same ladder, and strong systems use all three. The lecture’s algorithms live on the policy-based side of RL. For the field background, our lessons on Markov decision processes, policy gradients, actor-critic, PPO and SAC and reward learning go deeper on single chapters. For human feedback and DPO side by side, see RLHF and DPO, and for GRPO and its successors GRPO and DAPO. The three readings of this session each have a Veanor: Deep RL from human preferences, DeepSeek-R1 and DPO. The fourth reading, Faulty reward functions in the wild, is taught inside Chapter 8.
Now press Present or Teach and explain, from memory, how a reward becomes a change in a policy: the return, the baseline, the advantage, the clip, and the group. Then go back to the hero and try the sloppy check once more.
Credit: Lecture 9.1 of MIT MMAI, Multimodal AI (Spring 2026), Paul Liang and the MIT Multisensory Intelligence group; slides linked from the course schedule. Everything above is re-taught in our own words and devices. Toy numbers are labelled illustrative.