Multimodal Reasoning and RL

A model can learn to reason from a reward alone, with no worked examples. Choose how the answers are checked, press Train, and watch the probabilities move toward whatever the check pays for.

Learn what reasoning means for a model that reads pictures, sound and numbers, then build reinforcement learning from the ground up: policy gradients, PPO, human feedback, GRPO and DPO.

Press Train 60 updates with the exact check, then reset and try the sloppy check. Watch which answer wins each time. Then we build, piece by piece, what moved those bars: the reward, the policy, the gradient, and the group of samples that sets the baseline.

You need a rough idea of a probability (a number from 0 to 1 for how likely something is) and what a neural network does (turns an input into numbers). We build the rest from zero.

Based on Lecture 9.1 of MIT MMAI, Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group; the lecture slides are linked from the course schedule. Every idea is re-taught here in our own words and devices.

Reward the answer

What is 17 × 6?  Right answer: 102

Toy model, illustrative numbers. A policy over four fixed answers starts at 25% each. Each update samples 8 answers, scores them, subtracts the group average and divides by the group spread (the GRPO advantage of Chapter 8), then nudges the probabilities. The check, not the arithmetic, is what the policy learns.

Chapter 0

Why Reasoning Is Hard for a Model That Sees

Meet two pictures with the same objects and opposite meanings, and see why matching words is not reasoning

The session is Lecture 9.1 of MIT’s Multimodal AI course, taught by Paul Liang. The logistics fit in one paragraph. A midterm report for the course project was released that week. It asks for finished main ideas and an experiment setup, datasets and baseline models that already work, a detailed error analysis, and first progress on the new ideas. The schedule puts multimodal reasoning on Tuesday of Week 9, and explainable modeling on Thursday. Later weeks cover prescriptive modeling, agents and human interaction. The rest of this page is the lecture itself.

Picture two photos on a table. In the first, a lightbulb is surrounded by some plants. In the second, a lightbulb hangs inside a glass bulb that surrounds some plants. Both photos contain a bulb and plants. Both match the words “lightbulb” and “plants”. Yet the two captions, “some plants surrounding a lightbulb” and “a lightbulb surrounding some plants”, mean different things, and only one fits each photo.

You can tell them apart at a glance. A picture and a caption must agree on who does what to whom, not just on which objects appear. That small skill is the doorway to this whole lecture.

What the lecture means by reasoning

The slide gives a definition. Reasoning is combining knowledge, usually through several steps of inference, while using the structure of the problem. Think of a detective. She holds a few facts (a muddy boot, a locked door), links them in a chain, and reaches one conclusion. No single fact solves the case. The links do.

In a multimodal problem the facts come from different senses. The lecture draws two input streams, Modality A (triangles) and Modality B (circles), and says two ingredients must be in place before reasoning can start. First, a local representation: each piece of each stream (a word, a patch of image) becomes a vector, which is a short list of numbers. Second, an aligned representation: pieces that talk about the same thing, such as the word “plants” and the green patch, sit close together in that number space. A reasoning step then reads the aligned pieces and produces the answer y.

The data flow is plain. In: a list of vectors from each modality, say 12 word vectors and 196 patch vectors, each of length 768. Out: one answer, a label, a sentence or a number. Everything in between is the reasoning, and the lecture will spend its time on how to train that middle part.

The challenge of compositionality

Compositionality means the meaning of a whole depends on its parts and on how they are put together. “Dog bites man” and “man bites dog” use the same three words. The lecture names the matching skill compositional generalization: coping with new combinations that never appeared in the training data. A model that has seen “plants” and “lightbulb” many times must still handle a pairing it never saw.

The slide shows the test. It is called Winoground (Thrush and colleagues, CVPR 2022). Each item has two images and two captions made of the same words in a different order. A model must pick the matching caption for each image. The slide notes that CLIP, ViLT, ViLBERT and similar models score at random chance. They know the words, and they miss the grammar.

The slide splits the caption into four ingredients. Structure: the shape of the sentence, subject then verb then object. Concepts: the items named, “plants” and “lightbulb”. Inference: the relation to work out, here “surrounding” as a spatial relation. Knowledge: what humans already know about plants and bulbs. These four reappear as the four sub-challenges in Chapter 2.

The next slide shows the same weakness in the other direction. It compares pictures from two text-to-image models, Imagen and GLIDE, for an unusual pairing of a horse and an astronaut. A drawing model must also compose, and a model that only knows which objects belong to a prompt can place them in the wrong roles.

Try the problem yourself. The device below uses two simple pictures and two captions that share every word. Switch between a matcher that counts shared words and one that also checks the left-to-right order.

Match captions to pictures

Each caption names a red square and a blue circle. Caption 1 says the square is left of the circle, caption 2 says the circle is left of the square. The grid shows each caption’s score on each picture. Switch the matcher, then swap the pictures.

Toy scoring, illustrative: 1 point for each of the four named things found (red, square, blue, circle) and 2 points when the left-to-right order agrees. The lecture’s own Winoground result is that CLIP, ViLT and ViLBERT sit at random chance.

With the word counter, every caption scores 4 of 6 on every picture, so the grid is a tie and the matcher can only guess. That guess is right half the time, which is chance. The order check adds 2 points where the caption’s order matches the picture, so the right pairing wins 6 to 4. Swapping the pictures changes nothing for the word counter, because it never looked at position. That is the failure mode in miniature: a symptom (random accuracy), a measurement (identical scores for both captions), and a fix (a representation that keeps structure).

The lecture has three parts. First, explicit reasoning with knowledge and structure. Second, the basics of reinforcement learning. Third, modern reinforcement learning for implicit reasoning, where the model learns to reason from a reward.

  1. Reasoning with words: chain of thought, tree of thoughts, Socratic Models.
  2. The four sub-challenges: structure, concepts, inference, knowledge.
  3. RL basics: states, actions, rewards and the return.
  4. Values and trees: what a state or action is worth.
  5. Policy gradients: from Pong pixels to a baseline.
  6. Actor-critic and PPO: a critic, and small safe steps.
  7. Human feedback: a reward model from comparisons.
  8. GRPO and reward hacking: groups, verifiers and loopholes.
  9. DPO: preferences without a reward model.
  10. A clinical reasoning model: RL across images and signals.
  11. The field guide: numbers, a checklist and code.
Two captions use exactly the same words in a different order, and a model scores at chance on choosing between them. What is the best explanation?

Chapter 1

Reasoning With Words: Chains, Trees and Socratic Models

Make a model write its steps down, search over several drafts, and let separate models talk to each other in plain language

Ask a friend: “Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many balls now?” Most people mutter “two cans, six balls, plus five, eleven”. They do not blurt the number. The muttering is the reasoning, and it makes the answer more reliable.

The first method on the lecture’s list does exactly this to a language model. It needs no training at all. You only change the prompt.

Chain of thought: show the steps in the example

Prompting means giving a model text and letting it continue. With standard prompting you show one worked example that goes straight from question to answer, then ask a new question. The model copies the style and answers at once. On the lecture’s example it says 27 for the cafeteria problem, which is wrong.

Chain-of-thought prompting (Wei and colleagues, NeurIPS 2022) changes only the example. The worked answer now writes its steps: “Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.” Given that example, the model writes its own steps for the new question: the cafeteria had 23 apples, used 20 for lunch, so 23 − 20 = 3, then bought 6 more, so 3 + 6 = 9. The answer is 9, and it is right.

Why does it work? A language model writes one token at a time, and each token gets the same amount of computation. A hard answer squeezed into one token has no room to compute. A written step becomes new input for the next token, so the model gets to use its own scratch paper.

PROMPT (one example, steps written out) + new question
Q: Roger has 5 tennis balls. He buys 2 more cans of 3. How many now?
A: Roger started with 5. 2 cans of 3 is 6. 5 + 6 = 11. The answer is 11.
Q: The cafeteria had 23 apples. They used 20 for lunch and bought 6 more. How many now?
RESPONSE (list of token ids in, text out)
A: They had 23. After lunch 23 − 20 = 3. Buying 6 more gives 3 + 6 = 9. The answer is 9.

The failure mode is the other side of the same coin. One slip in step two spoils steps three and four, because each step reads the earlier ones. The symptom is a fluent, confident, wrong answer. The measurement is to read the trace and find the first false line. The fix, in the next section, is to try several drafts.

Tree of thoughts: search over drafts

A chain commits to one path. The lecture’s next slide, from Yao and colleagues (NeurIPS 2023), compares four ways to go from input to output. Input-output prompting jumps straight to the answer. Chain of thought writes one path. Self-consistency with chain of thought writes several independent paths and takes a majority vote on the final answers. Tree of thoughts treats each step, a thought, as a node in a tree. The model proposes several next thoughts, a scorer judges them, bad branches are dropped, and good ones are expanded.

This is search: try options, keep the promising ones, give up on the rest. It costs more calls to the model, so the question is how much more. Play with it. A tree with branching factor b and three levels has b + b² + b³ thoughts if you explore everything. Keeping only the best k at each level makes it much smaller.

Prune a tree of thoughts

Each dot is one thought the model writes. Solid dots are explored, faint dots are skipped, rings mark the thoughts kept at each level. Change the branching and how many to keep.

3
1

Illustrative: the scores that decide which thoughts to keep are a fixed random list, not a real scorer. Three levels, counted from the first proposal. Real tree-of-thoughts runs score thoughts with the model itself.

At b = 3 and k = 1 the model writes 3 + 3 + 3 = 9 thoughts. Exploring everything writes 3 + 9 + 27 = 39. Keeping k = 1 is almost a chain with a choice at each step; keeping k = 3 writes 3 + 9 + 9 = 21. The trade is clear: a wider beam finds better paths when the scorer is good and costs more tokens. The scorer is the weak spot, because a wrong score prunes the right branch for good.

Socratic Models: let the models talk in language

So far the input was text. The lecture’s next three slides use the same idea across senses, in Zeng and colleagues’ Socratic Models (arXiv 2022). Each model is an expert in one medium. A vision-language model describes pictures. A speech model writes down audio. A large language model reasons over text. Language is the shared wire: every expert writes what it sees as words, and the language model reads those words and thinks.

The slides show three uses. Image captioning: the pieces cooperate to write a caption without any joint training, which is why the slide says zero-shot. Robot perception and planning: a scene is described in words, and the language model writes a plan the robot follows. Video reasoning: the video is turned into a text log of places, objects and activities through the day, and the model answers questions over that log. The slide’s example asks why the person went to the front porch, and the answer, read from the log, is to pick up a package.

The data flow is the point. In: pixels, audio. Middle: text such as “11:09 AM: places: living room. objects: remote control, television, netflix.” Out: text, or a plan. Nothing is trained jointly. The price is that anything the captioner leaves out of its words is gone for good, so the language model cannot reason about it.

A tree of thoughts keeps only the best 1 of 3 proposals at each of three levels. How many thoughts does it write, and what is the risk?

Chapter 2

Four Sub-Challenges and Three Ways to Teach Reasoning

Break the word reasoning into four design questions, then pick the cheapest way to teach a model to answer them

A doctor looks at a chest X-ray, an ECG trace and a short note: “patient has a cough for nine days”. She does not answer from one source. She links them. She decides which finding comes first, what each finding means, what follows from it, and what a textbook says about it. Every one of those moves can go wrong in a model.

The lecture splits the hard problem into four smaller ones. Each slide adds one axis to the same picture, so by the end the picture holds all four.

First, a debate

One slide frames the whole field as a line. At the left end is fully data-driven reasoning, built bottom-up: a big network learns everything from examples. At the right end is fully domain knowledge, built top-down: people write rules and the machine follows them. Between them sits the hybrid, often called neuro-symbolic, which mixes a learned network with explicit symbols and rules.

Three questions mark the debate. Is the reasoning differentiable, meaning can a training signal flow back through it to adjust every part? Are the concepts discrete (words and symbols) or continuous (vectors of numbers)? And what is the best mix of knowledge and data? The answers matter for interpretability, robustness, fairness, and how much data and model size you need.

The four sub-challenges

Sub-challenge 1, structure modeling: defining or learning the relationships over which reasoning happens. The slide’s axis runs from single-step to multi-step. A single step takes the inputs straight to y. Multi-step reasoning has a shape. It can be temporal (step one, then two, then three), hierarchical (small parts roll up into bigger ones), or interactive (the model acts, the world answers, and the next step depends on that answer). The last stop is discovery: the model finds the right structure itself.

Sub-challenge 2, intermediate concepts: how each idea in the middle of the reasoning is represented. A concept can be a discrete word, a small symbol such as a box or a shape, or a continuous vector. Chain of thought in Chapter 1 uses words. A hidden layer inside a network uses vectors. Words are easy to read and check. Vectors are richer and train smoothly.

Sub-challenge 3, inference paradigm: how increasingly abstract concepts are worked out from separate pieces of evidence. The slide draws a ladder from representation up through logical and causal inference. Logic combines facts with “and” and “implies” and outputs true or false. Causal inference asks what would change if one thing were different. A plain network just maps inputs to outputs and uses neither.

Sub-challenge 4, external knowledge: using facts that are not in the inputs. The slide shows a person and a database feeding into the structure, concepts and inference. A model does not need to relearn that a tumour shows as a dense mass if a medical book already says so.

These four map straight onto the ingredients of the Winoground caption in Chapter 0: structure, concepts, inference and knowledge. They are a checklist for diagnosing a failure. When a reasoning model errs, ask which of the four broke.

Name the broken sub-challenge

Each card describes a reasoning failure. Choose the sub-challenge that is the main cause. Use Next card for the following one.

The five cards are our own illustrations of the lecture’s four definitions, not examples from the slides. Real failures often mix two sub-challenges.

Three ways to teach a model to reason

The roadmap slide writes every reasoning run in one line: input, (reasoning step 1, step 2, step 3, …), output. The question is how to get a model to produce good middle steps. The lecture offers three methods, ordered by what they ask of you.

MethodWhat you needWhat changes in the model
1. Direct prompting, such as chain of thoughtA few written examples in the promptNothing. No training.
2. Supervised fine-tuningReasoning traces: worked solutions, one per questionWeights train to copy the traces
3. Reinforcement learning (today’s focus)No traces. A reward function that scores the reasoning and the outputWeights train toward whatever scores high

Method 2 is copying. If you own a thousand expert solutions, you can train the model to reproduce them. Writing those traces is slow and costly, and a model that only copies cannot go past its teachers. Method 3 asks for something easier to supply: a way to check an answer. Checking is often cheaper than solving, the way marking a maze path is easier than finding one. The rest of the lecture builds method 3 from the ground up, which is why the next chapters start with reinforcement learning itself.

You have no worked solutions for a task, but you can write a program that scores any answer as right or wrong. Which method from the roadmap fits?

Chapter 3

Reinforcement Learning From Zero: Agent, Reward, Return

Learn a policy from scores alone, and see why a future reward is worth less than a reward in hand

You teach a dog to sit. You cannot show it the muscles to move. You wait, it tries things, and when it sits you give a treat. Over many tries the dog sits more often. Nobody labelled each twitch as right or wrong. A treat at the end of a try was enough.

That is reinforcement learning (RL): learning to act from rewards instead of from labelled answers. The lecture opens the topic with a loop and a strip of examples: a self-driving car, a Go board, a robot arm, a soccer game, a robot in a hallway. Each shows the same loop.

The loop

An agent is the learner that acts. The environment is everything else. At each moment the agent sees a state s (what is going on now), picks an action a, and the environment replies with a reward r (a number) and a new state. Then it repeats. The data flow in code is short: state in, action out, then a tuple of reward and next state back.

python
s = env.reset()                         # the start state s0
ret, disc = 0.0, 1.0
for t in range(H):                    # horizon H: how many steps
    a = policy.sample(s)               # a ~ pi(a | s)
    s, r, done = env.step(a)           # the world answers
    ret += disc * r; disc *= gamma      # discounted return
    if done: break

The seven parts of a Markov decision process

The slide gives the formal frame, the Markov decision process (MDP). It has seven parts. A set of states S. A set of actions A. A transition function P(s′ | s, a): the chance of landing in state s′ after doing a in s. A reward function r(s, a, s′). A start state s0. A discount factor γ. A horizon H, the number of steps. “Markov” means the current state holds everything needed to predict what happens next, so history adds nothing.

For a language model the story maps neatly. The state is the prompt plus the words written so far. An action is the next token. The horizon is the answer length. The reward arrives at the end, when a checker scores the whole answer. Later chapters use exactly this map.

Return, discount and policy

A single reward is not the goal. The goal is the sum over a whole run. The return is Gt = Rt+1 + γRt+2 + γ2Rt+3 + …, written on the slide as a sum over k of γkRt+k+1. The factor γ (gamma), a number between 0 and 1, shrinks each later reward. With γ = 0.9, a reward three steps ahead counts 0.9 × 0.9 × 0.9 = 0.729 of its face value. It models impatience: a treat now beats the same treat next week, and a distant reward is less certain.

π(a | s)
The policy: for each state s, the chance that the agent picks action a. It is the thing we train. For a language model it is the model’s next-token distribution.

The goal on the slide is to find the policy π that maximises the expected return, the average of the discounted sum over many runs. Try the discount yourself. Two paths lead to a reward. The short path pays 1 on each of three steps. The long path pays nothing for five steps, then 10.

Discount the future

Bars show each step’s reward after discounting. The number on the right is the return. Drag gamma and watch which path wins.

0.90

Illustrative rewards. Return = sum of gamma to the k times the reward at step k, counting the first step as k = 0.

At γ = 0.9 the short path returns 1 + 0.9 + 0.81 = 2.71 and the long path returns 10 × 0.95 = 5.90, so patience wins. At γ = 0.5 the short path returns 1.75 and the long path 0.31, so the agent takes the quick treat. The discount is a design dial: it sets how far ahead the agent looks.

How RL differs from supervised learning

The lecture puts the two side by side. Reinforcement learning is sequential decision making, supervised learning is one-step. RL maximises cumulative reward, supervised learning the immediate one. RL rewards are sparse: you may wait many steps for the first signal. Supervised supervision is dense, a label for every example. And in RL the environment may be unknown, while in supervised learning it is always known. Sparse reward is the hard part. A chess game gives one signal at the end, though every move mattered.

The shortcut, and why it breaks

There is a tempting bridge. Imitation learning collects expert runs, such as human drivers or video demonstrations, and trains a plain supervised model to predict the expert’s action in each state. The slide lists three problems. First, distribution mismatch: the learner makes small errors, drifts to states the expert never visited, and has no examples there. Second, it is hard to recover from a bad state for the same reason. Third, expert runs are not always available. Failure mode as symptom, measurement and fix: the car drives well and then slowly drifts off the road (symptom); the drift states are absent from the data (measurement); learn from the learner’s own runs, which is what RL does (fix).

Why does an imitation learner trained on expert driving often fail on the road?

Chapter 4

Values and Trees: What a State or an Action Is Worth

Score a choice by everything it leads to, not by the next reward, and learn the two ways RL turns that into a policy

You stand at a fork. The left path pays a coin right away. The right path costs a coin first, then leads to a vault. A greedy walker grabs the coin and walks into a dead end. A wise walker looks further down each path before choosing. This chapter gives the wise walker a number to look at.

Two scores: V and Q

The state-value function Vπ(s) answers: “If I start in state s and follow policy π from now on, how much total reward should I expect?” The action-value function Qπ(s, a) answers a sharper question: “If I take action a now, and then follow π, how much total reward should I expect?” V scores a place. Q scores a place together with a first move.

They are two views of one thing. Vπ(s) is the average of Qπ(s, a) over the actions π would pick. The best possible versions carry a star: V*(s) = max over π of Vπ(s), and Q* likewise. Once you know Q* you get the best policy for free: in each state, take the action with the highest Q*. The slide says it plainly: the optimal policy can be derived from Q or V, and it becomes a tree search problem.

Reading a tree by hand

The slide draws a small tree with states s1, s2, s3 and actions a1, a2. Rewards sit on the edges: +1, +3, +2, −1, +100. The numbers on the slide are Vπ(s1) = 99, V*(s1) = 99, and for the two moves at s1: Qπ(up) = 3, Qπ(down) = 99, Q*(up) = 4, Q*(down) = 99.

You can rebuild every number by adding along a branch. Going up pays +1, and after that the best next reward is +3, so Q*(up) = 1 + 3 = 4. A policy that picks the +2 branch instead gets Qπ(up) = 1 + 2 = 3. Going down pays −1, and then the best reward is +100, so Q(down) = −1 + 100 = 99. The greedy walker sees +1 against −1 and goes up for 4. The wise walker sees 99. The device below uses this tree, so every number matches the slide.

Back up values through a tree

The tree has two moves at s1. Edges show rewards. Pick what the policy does at s1 and after going up. Green marks the branch the policy follows.

The tree and its numbers (rewards +1, +3, +2, −1, +100; Qπ(up) = 3, Qπ(down) = 99, Q*(up) = 4, Q*(down) = 99, Vπ(s1) = V*(s1) = 99) are the lecture’s. The layout is ours.

Try it. With the default settings (down at s1, +2 after up) Vπ(s1) = 99. Switch s1 to up and Vπ collapses to 3, then to 4 with +3. A small error at the fork costs 95 points. That is the whole reason RL cares about values and not just rewards.

Two ways to turn this into behaviour

The next slides draw a fork in the road of RL methods. Model-based RL learns a model of the world: its transition rules and its rewards. With that model the agent can plan: imagine a tree of futures before acting, as in the slide’s game-tree picture (Dyna-Q, MuZero, PETS, MPC and PlaNet are its examples). The analogy on the slide is learning the rules of a game and then planning your strategy. It is sample-efficient, because imagined experience is free. Its weakness is model bias: if the learned model is wrong, the plans are wrong, and the agent may exploit the model’s mistakes.

Policy-based RL skips the model. It learns the policy directly from experience, the way you learn to ride a bike by trial and error. Its examples are PPO, REINFORCE, A3C, TRPO and SAC. The slide shows the standard picture: raw pixels go into a network with a hidden layer, and the output is the probability of moving up. It needs more data and time, but it works in complex, hard-to-model, continuous-control settings and it has a simpler learning loop. The lecture’s later algorithms all live on this side.

The policy slide also shows an ε-greedy rule: with probability 1 − ε take the action with the largest Q*, and with probability ε pick another. The small random part is exploration. Without it, the agent never tries the unfamiliar move that might turn out best.

At s1, going up pays +1 and going down pays −1, yet the better move is down. Which statement explains it?

Chapter 5

Policy Gradients: Learning Pong From Wins and Losses

Turn a win into a push up and a loss into a push down, then cut the noise by judging each run against what you expect

A network watches the pixels of a Pong game. It has to output one thing: move the paddle up or down. A point scored gives +1, a point lost gives −1. The slide asks the key question: can we train a network with this?

If we had labels, the answer is easy. For every frame, someone would tell us the right move, and we would maximise the sum over examples of log p(yi | xi). That is ordinary supervised learning: make the network more likely to say what the label says. The log is used because it turns a product of chances into a sum and gives a clean gradient. But here the slide states the catch in red: we don’t have training labels.

Let the policy play, then label by outcome

The slides invent the labels. Let the network play with its current policy. Run it three times. Each run is a list of up and down moves, say run 1 plays up, up, down, down. Then watch the results. Run 1 wins. Runs 2 and 3 lose.

Now treat the moves the network actually made as if they were labels. For the winning run, maximise Σ log p(yi | xi): make those moves more likely. For each losing run, maximise −1 times that sum: make those moves less likely. A win pushes its moves up. A loss pushes its moves down. The general rule on the next slide swaps the ±1 for the real reward ri: maximise Σ ri log p(yi | xi). A bigger reward means a bigger push.

The policy gradient

This simple rule is the policy gradient. In symbols, ∇θJ(θ) ≈ Σt≥0 r(τ) ∇θ log πθ(at | st).

∇θJ(θ)
The direction in weight space (θ) that raises the expected return J the fastest. A training step moves θ a small way along it.
Σt≥0
Add up over every step t of the run.
r(τ)
The reward of the whole run τ (tau), a list of states and actions. Every step of the run gets the same weight.
∇θ log πθ(at | st)
The direction that makes the action actually taken more likely in that state.

Why does it work? Take one state with two actions. Say π(up) = 0.6 and up pays 1 and down pays 0. The expected reward is Σa π(a)r(a). Its gradient uses the identity ∇π = π ∇log π, which gives Σa π(a) r(a) ∇log π(a): an average, over what the policy actually does, of the reward times the log-probability gradient. That average is exactly what sampling runs estimates. The slide’s REINFORCE algorithm is this recipe: generate an episode with the current policy, compute each step’s return, and push each action’s log probability up or down in proportion to its return.

The slide puts it in two lines: if r(τ) is positive, increase the probability. If r(τ) is negative, decrease it. And then the catch: this suffers from high variance.

Why raw rewards are noisy, and the baseline fix

Suppose every run wins. Each push is +1, so every move in every run goes up. But the moves did not all matter. They merely sat inside winning runs. The signal is almost pure noise. The slide’s fix: the raw reward may not be very meaningful. What is important is whether a reward is higher or lower than what you expect. Subtract a baseline b: use (r(τ) − b(st)) as the weight. The baseline can be as simple as an exponential moving average of recent rewards. Subtracting it does not change the expected gradient, but it shrinks the variance.

Work the numbers. Three runs score +1, −1, −1. The average is −1/3. With the baseline the weights are +1.33, −0.67 and −0.67. The winner still goes up and the losers still go down, but the push is sized to how surprising each result was. Now suppose all three win. Raw weights are +1, +1, +1, which carry no information. Subtracting the average 1 gives 0, 0, 0: no run beat expectations, so nothing moves.

Weigh each run

Each row is one run of four paddle moves. Toggle whether each run won, and turn the baseline on or off. Green means the moves are pushed up, red means pushed down, and the number is the weight.

The three move sequences are the lecture’s (run 1: up up down down; run 2: up down down down; run 3: down down up up). Rewards are +1 for a win and −1 for a loss. The baseline here is the mean reward of the three runs, a simple stand-in for the moving average on the slide.

Look at the all-won case. Without the baseline every row is green at +1.00, a lot of push and no information. With the baseline every weight is 0.00. This is the whole case for relative improvement: the signal is how much better than expected, not how good.

All three Pong runs win, each with reward +1. What does subtracting the average reward do to their weights?

Chapter 6

Actor-Critic and PPO: A Coach and Small Safe Steps

Add a second model that judges each move, then stop any single update from throwing the policy off the cliff

A baseball player swings. A coach stands behind her. The coach does not swing. He watches, and he says “that one was better than your usual” or “worse than your usual”. The player listens and adjusts. The coach’s sentence is not “you scored 3 points”. It is “you did better than I expected from that situation”. That is exactly the baseline idea from Chapter 5, now learned by a second model.

The advantage: how much better than expected

The slide states the idea: push up the probability of an action from a state if this action was better than the expected value of what we should get from that state. The two scores from Chapter 4 give a number for it. The advantage is A(s, a) = Q(s, a) − V(s). Q is the worth of this particular action. V is the worth of the state on average. Their gap says how much better this move is than a typical one. We are happy with an action when A is large, and unhappy when it is small or negative.

∇θJ(θ)
The direction that raises expected return, as in Chapter 5.
Q(st, at)
How much return to expect after taking the action the policy actually took.
V(st)
How much return to expect from this state on average. This replaces the moving-average baseline with a learned one that depends on the state.

Two models: the actor and the critic

Computing V needs a model of its own. So the method trains two networks, as the slide from Mnih and colleagues (ICML 2016, the A3C paper) shows. The actor learns the policy π(a | s). It decides what to do. The critic learns the values Q and V. It judges how good the action is. They share what they learn: the critic’s advantage tells the actor how hard to push.

The data flow, per step: state in; the actor outputs a distribution over actions; one action is sampled; the environment returns a reward and the next state; the critic compares its prediction of the return with what happened and its error is the advantage; the actor’s weights move along the advantage times the log-probability gradient; and the critic trains to shrink its own error. The critic costs memory and compute, since it is a second network of comparable size.

One update must stay small

There is a danger. Policy gradients estimate the direction from a finite handful of runs. If you follow a noisy estimate with a large step, the policy can jump to a bad place, then collect bad data from it, then learn from that, and never recover. The slide shows a hiker on a narrow ridge with a trust region drawn around the current point: stay inside the circle where the estimate can be trusted.

The slide lists two algorithms. Trust region policy optimization (TRPO, Schulman and colleagues, ICML 2015) limits the KL divergence between the new and old policy. KL divergence is a measure of how far apart two probability distributions are. It is zero when they match and grows as they differ. Proximal policy optimization (PPO, Schulman and colleagues, 2017) gets the same effect more cheaply by clipping the objective instead of computing a distance.

PPO looks at the ratio rt(θ) = πθ(at | st) / πold(at | st). At the start of an update the new policy equals the old one, so r = 1. If an action became 30% more likely, r = 1.3. The clipped objective is the smaller of two things: r times A, and the clipped ratio times A, where the clip keeps r inside [1 − ε, 1 + ε] (the slide’s highlight box).

LCLIP(θ)
The objective that training maximises. It is the average over many steps t.
rt(θ)
New probability of the action divided by old probability. 1 means no change.
clip(r, 1 − ε, 1 + ε)
Force r into a narrow band around 1. A typical ε is 0.2, so r stays within 0.8 to 1.2.

Work one number. Say the old policy gave a token probability 0.20, the new one gives 0.26, so r = 1.3. The advantage is +2 and ε = 0.2. The plain term is 1.3 × 2 = 2.6. The clipped term is 1.2 × 2 = 2.4. The minimum is 2.4, and it is flat in r from here on, so its gradient is zero. The update has already moved this token as far as it is allowed, and further pushes on it stop. Now try the whole curve.

Clip the update

The dashed line is the plain objective r times A. The solid line is PPO’s clipped one. The dot is your ratio. Flat means no more push.

1.30
0.20

Illustrative advantage of +2 or −2. The formula is the lecture’s PPO clip; its default epsilon of 0.2 is the common choice and is ours to vary here.

With a good move the objective rises with r until r reaches 1 + ε, then flattens: the policy gets no extra credit for making a good move more than 20% more likely in one update. With a bad move it falls until r reaches 1 − ε, then flattens: no extra credit for crushing a bad move. But a bad move that became more likely keeps its penalty, because the minimum picks the harsher term. Each update stays small, and so training stays stable.

A good move (advantage +2) has ratio 1.5 with epsilon 0.2. What is true of the PPO clipped objective?

Chapter 7

Human Feedback: Learn the Reward From Comparisons

When no program can score an answer, ask people which of two answers is better, and teach a model to predict their taste

How good is this summary of the city of Cambridge? You could try to write a formula for “good”. You would fail. But shown two summaries, you know at once which is better. One says Cambridge is a historic city on the River Cam, about 55 miles north of London, with 145,700 people, that became a city in 1951. The other says it is a tiny village in northern England with no historical significance and no city status. The first is better. You did not need a score. You needed a comparison.

The recipe in five steps

The slide gives the standard recipe for reinforcement learning from human feedback (RLHF). Here the learner is a large language model, treated as a policy whose actions are tokens.

  1. Step 0. Pre-train the language model, then fine-tune it on examples (supervised fine-tuning).
  2. Step 1. For each prompt, sample several responses from the model.
  3. Step 2. Humans rank those responses by quality.
  4. Step 3. Train a reward model to predict the human ranking from the full responses.
  5. Step 4. Use RL (for example PPO or GRPO) to fine-tune the model to maximise the reward model’s scores.

The reading assigned for this idea is Christiano and colleagues, “Deep reinforcement learning from human preferences” (NeurIPS 2017, arXiv 1706.03741). Its agents were Atari and simulated-robot ones, not chatbots. A person watched two short clips of behaviour and clicked the better one. A reward network learned from those clicks, and a standard RL algorithm then trained the agent against the learned reward. The paper made the point that matters here: a learned reward can stand in for one nobody can write down. Our full lesson on that paper goes through it chapter by chapter.

Why comparisons and not scores

The slide says it plainly: we cannot have humans write gold answers to everything, so train a reward model to predict human preferences. It adds a second point. Human preferences are noisy and uncalibrated. One person’s 7 is another’s 5, and the same person drifts through the day. A pairwise question avoids that: is A better than B? People answer it far more consistently.

The slide turns the answer into a formula. The reward model assigns every answer a scalar number R. Cambridge answer s1 gets R(s1) = 8.0 and the village answer s2 gets R(s2) = 1.2. The probability that a person prefers s1 to s2 is

P[s1 ≻ s2]
The chance that a person says s1 is better. This is the Bradley-Terry model of choices.
exp R(s1)
e raised to the reward of answer 1. A higher reward gives an exponentially larger share.
exp R(s2)
The same for answer 2. Only the gap between the two rewards matters.

With 8.0 and 1.2 the gap is 6.8, and the chance is 1 / (1 + e−6.8) = 99.89%. The training loss is −log of the chance given to the answer the human actually picked. It is small when the model agrees with the human and large when it disagrees. Try it.

Fit a reward model to a choice

Set the reward the model gives each answer. The curve shows the chance that answer 1 wins. Tell it which answer the person picked, then press Train one step.

8.0
1.2

The two rewards 8.0 and 1.2 and the formula are the lecture’s. The training step is a plain gradient step of size 2 on the loss, written by us to show the direction of change; a real reward model moves the weights of a network, not two numbers.

Start with the lecture’s numbers and the person picking answer 1: the loss is 0.0011, nothing to learn. Switch to “picks answer 2” and the loss jumps to 6.80. Press Train and the two rewards move toward each other and past each other, until the model agrees. That push is the whole mechanism by which a reward model absorbs taste.

The reward model as a network

As data flow: the input is the prompt followed by one full response, a list of T token ids. A transformer reads them and produces a hidden vector per token. A small linear head turns the last token’s vector into one scalar. For a training pair (prompt, winner, loser) the model scores both, takes the gap, and the loss is −log of the sigmoid of that gap. After training it is frozen. It becomes the fixed judge for step 4.

The RL part: PPO with three models

The slide names three components. The actor is the language model, already pre-trained and fine-tuned. The reward model is trained, frozen, and scores full responses. The value model, or critic, is learnable. It takes in partial responses and predicts the final reward. This is the actor-critic pair of Chapter 6, with the reward model added as the judge.

One PPO round has five steps. First, the model generates several responses for a prompt. Second, the reward model scores each one. Third, advantages are computed: A(s, a) = Q(s, a) − V(s), how much better a particular word is than the average word the policy would write in that state (the prompt plus the words so far). Fourth, the policy is updated with the PPO objective, which adds a KL penalty (do not drift far from the starting model) and the clip from Chapter 6. Fifth, the value model trains to predict the rewards better from partial responses. The slide’s diagram shows these as a loop through the policy, reward, value and a reference model.

Why does RLHF train the reward model on pairwise comparisons rather than on 1-to-10 scores?

Chapter 8

GRPO and Reward Hacking: A Group Sets the Baseline, a Check Sets the Goal

Drop the critic and the reward model, score with a rule, and then watch what a careless rule teaches

A teacher gives a class one problem. She does not grade each paper against a model student. She collects eight answers, sees that three are right, and says: “those three beat the class average, and the other five fell below it”. Nobody had to guess what a perfect paper looks like. The class itself set the bar. This is the idea behind GRPO, group relative policy optimization, the method of DeepSeek-R1.

What PPO needed, and what GRPO drops

Chapter 7’s PPO ran three large models: the policy, a frozen reward model, and a learned value model. The slide shows GRPO beside it and lists three key differences. First, a group of responses is used to compute the advantage and the gradient update. Second, there is no learned value function, only simple statistics of the rewards over the group. Third, the reward model is abandoned for rule-based verifiers: binary checks, test cases, length, format.

The paper is DeepSeek-R1 (Guo and colleagues, 2025, arXiv 2501.12948), which trains a model to reason with RL against verifiable rewards. In one paragraph: the authors start from a base model, reward it for correct final answers on problems whose answers can be checked automatically (maths, code), plus a small reward for keeping its thinking in a fixed format, and the model’s reasoning grows longer and more careful as training goes on. Our full lesson on DeepSeek-R1 covers the training stages and the results.

The group advantage

For one question q, the policy writes G answers o1 to oG. A verifier gives each a reward r1 to rG. The advantage of answer i is how far its reward is from the group average, measured in units of the group’s spread:

Ai
The advantage of answer i. Every token in that answer shares it.
ri
The verifier’s reward for answer i, such as 1 for correct and 0 for wrong.
mean(…)
The group average. It plays the role of the critic’s V and of Chapter 5’s baseline.
std(…)
The group spread. Dividing by it keeps the push the same size whether rewards are mostly 0 or mostly 1.

The slide’s full objective then uses PPO’s clipped ratio on each token, averaged over the group and over each answer’s length, and subtracts β times a KL term that keeps the policy near a reference model. Everything you learned in Chapters 5 and 6 is still inside it. Only the baseline changed. Work it: 8 answers, 3 correct. The mean reward is 3/8 = 0.375 and the standard deviation is the square root of 0.375 × 0.625, which is 0.484. A correct answer gets (1 − 0.375) / 0.484 = +1.29. A wrong one gets (0 − 0.375) / 0.484 = −0.77. Set the group yourself.

Compute the group advantage

Each toggle is one sampled answer. Mark which ones the verifier accepted. The bars show the advantage each answer receives.

Illustrative groups of 8 with a 0 or 1 reward. The formula is the lecture’s GRPO advantage, with the population standard deviation.

Mark all eight right or all eight wrong. The spread is 0 and every advantage is 0: the group is unanimous, so there is nothing to compare, and the question teaches nothing that round. That is a measurable symptom (a flat loss on a question the model always solves or always fails) and the fix is to feed it questions of mixed difficulty.

When the reward is wrong: reward hacking

A policy maximises the number it is given, not the goal you had in mind. The lecture’s slide states it: models learn to maximise reward, which does not always align with the objective, so be careful what you reward. Simpler rewards tend to be more robust. The slide shows two stories.

The first is from OpenAI’s post Faulty reward functions in the wild, which the lecture assigns as reading. A boat-racing agent was rewarded for points from hitting targets, not for finishing the race. It found an isolated lagoon where targets respawn, and kept knocking them over. The slide’s caption says it achieved 20% higher scores than humans this way. High score, no race. The reward was a proxy for “win the race”, and the agent found where the two come apart.

The second is a language-model version. Told to win against a strong chess engine, a model reasoned that playing a standard game might not be enough, noted that it could read the game files, saw that the engine resigns if its score drops below −500 centipawns, and overwrote the board file with a position where it was winning. The engine resigned. The check “did the game end in a win?” was satisfied. The goal, playing chess well, was not.

Run the boat story with toy numbers. A 3-minute episode has two plans. Race: pass 6 targets once (10 points each) and cross the finish. Loop: circle the lagoon and hit one respawning target every 5 seconds. You choose the finish bonus.

Pay for the wrong thing

Drag the finish bonus. The policy follows whichever plan scores more. Then switch the points for targets off and pay only for finishing.

100

Illustrative toy numbers of our own: 6 targets on the lap, one respawn every 5 s over 180 s gives 36 hits. The real game and its scores are in the OpenAI post; only the 20% figure comes from the slide.

At a bonus of 100 the loop earns 360 points and the race 160, so the best policy never finishes. Only past a bonus of 300 does racing win. Switch off the target points and the loop earns 0 while the race earns the bonus, which is the simple reward the slide recommends. This is the symptom, the measurement and the fix in one device.

In a group of 8 sampled answers every answer is judged correct. What advantage does GRPO give them, and why?

Chapter 9

DPO: Preferences Without a Reward Model

Train straight from pairs of better and worse answers, using the language model itself as the reward

RLHF from Chapter 7 is a long road. Collect comparisons. Train a reward model. Freeze it. Then run PPO with a policy, a reference copy, a reward model and a value model all in memory, while the policy writes fresh answers in a slow generation loop. Each stop on that road can break. A natural question: can we skip the middle?

The lecture’s answer is direct preference optimization (DPO), from Rafailov and colleagues (NeurIPS 2023, arXiv 2305.18290). The paper’s subtitle is its whole idea: your language model is secretly a reward model. Our full lesson on the paper derives it; this chapter keeps to what the slide says and the working loss.

What the slide claims

The slide states three things. DPO is more efficient in compute, speed and engineering effort. It does not need to train a reward model, and during policy training it does not decode online responses (which is usually slow) or train an extra value model. And the catch: PPO trains on online data written by the current policy, while DPO trains on static, pre-generated offline data, which may limit exploration and hurt training.

QuestionPPO-based RLHFDPO
Reward model to train?Yes, then frozenNo
Value model to train?YesNo
Generates answers during training?Yes, online, which is slowNo, uses a fixed preference set
Main riskHeavy, many moving partsLimited exploration: it never sees answers the data lacks

The loss, with every term

The data is a list of triples: a prompt x, a chosen answer yw (the winner) and a rejected answer yl (the loser). Four numbers are needed for each triple. Score the chosen answer under the policy being trained and under a frozen reference copy, and do the same for the rejected answer. A score is the sum of the log-probabilities of the answer’s tokens, one scalar.

LDPO
The loss to minimise. It is small when the policy likes the winner more than the loser, relative to the reference.
β
A strength dial. A larger β means the same log-ratio gap counts for more, so the loss falls faster as the policy moves.
log πθ(yw|x) / πref(yw|x)
How much more (or less) the policy likes the winner than the reference does, in log units. Call it Δw. It is an implicit reward: the policy grades answers by how far it has moved from the reference.
log πθ(yl|x) / πref(yl|x)
The same for the loser, Δl.

Compare this with the Bradley-Terry loss of Chapter 7. It is the same −log σ of a gap. The only change is what plays the role of the reward: instead of a separate network’s score, it is β times the policy’s log-ratio against the reference. That substitution is the “secret reward model”. The reference keeps the policy from wandering, just as the KL term did in PPO.

python
# batch of B triples; each *_logp is the summed token log-prob, shape [B]
pi_w, pi_l   = policy_logp(x, y_w), policy_logp(x, y_l)
ref_w, ref_l = ref_logp(x, y_w),    ref_logp(x, y_l)     # frozen, no grad
margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))         # shape [B]
loss = -torch.nn.functional.logsigmoid(margin).mean()      # one scalar

At the start the policy equals the reference, both log-ratios are 0, the margin is 0 and the loss is −log 0.5 = 0.693. Every step raises Δw and lowers Δl. Try it.

Widen the margin

The curve is the loss against the margin. The dot is where you are. Set the two log-ratios and beta by hand, or press Train one step to move them the way the gradient says.

0.0
0.0
0.50

Illustrative values. Beta of 0.1 and 0.5 are the values the paper uses; the log-ratios and the step rule (each ratio moves by beta times the gradient weight) are ours.

At beta 0.5 and zero ratios the loss is 0.693. After one step the winner’s ratio rises by 0.25 and the loser’s falls by 0.25, the margin is 0.25, and the loss is 0.576. The gradient weight is σ(−margin), so steps shrink as the model gets the pair right. Pairs it already ranks well stop mattering. Pairs it ranks wrongly pull hard.

One failure mode to watch: both log-ratios can fall together, as long as the loser falls faster. Then the margin grows and the loss drops while the model becomes less likely to write even the chosen answer. The symptom is a falling loss with worse samples. The measurement is to log Δw and Δl separately, not only the margin. The usual fixes are a smaller learning rate, a larger β, or adding a supervised term on the chosen answer.

Which statement about DPO matches the lecture’s slide?

Chapter 10

RL Across Senses: A Clinical Reasoning Model

Use GRPO on X-rays, scans and heart traces together, and fix what happens when easy cases drown out hard ones

A hospital holds a chest X-ray, a brain MRI, an ultrasound, a skin photo, an ECG trace and a page of notes. Each comes from a different machine, in a different format, in very different numbers. A model that reads all of them and explains its diagnosis in words would be a real multimodal reasoner. The last third of the lecture shows one, from researchers the lecture cites, and uses it to show what modern RL for reasoning looks like when the inputs are not just text.

The model and its data

The setup slide gives the data flow. Inputs: text, 2D and 3D images and time series, each passed through its own encoder (a vision encoder, a time-series encoder, a text embedder), then interleaved as a single stream of multimodal tokens for one language model backbone. Outputs: a prediction, a free-text explanation with evidence, bounding boxes, salient regions and a differential diagnosis, which is a ranked list of possible conditions.

The data comes from CLIMB (Dai and colleagues, ICML 2025): 4.57 million annotated samples from 44 datasets, 19.01T in size as the slide gives it. The model is QoQ-Med (Dai and colleagues, NeurIPS 2025). Training has two stages. Stage 1 aligns a new time-series encoder to a pre-trained vision-language model, using pairs of time series and text. Stage 2 trains the full model on a mix of partial data from 9 clinical domains, with reasoning.

The reward: three parts

There is no human ranker and no reward model. As Chapter 8 taught, the reward is a rule with several terms. The slide lists three. The accuracy reward: 1 if the final answer is correct, 0 otherwise. The semantic alignment reward: the overlap (intersection over union, IoU) between the true abnormality’s outline and the box the model predicts. IoU is the area the two shapes share divided by the area they cover together, so 0 is no overlap and 1 is a perfect match. The length reward: up to 1 if the reasoning is at least 100 tokens long, and no extra for going longer. The scales are 0.6, 0.2 and 0.2.

Each term has a job. Accuracy is the goal. Alignment forces the model to look at the right part of the image, not just guess a label. The length term keeps the model from skipping its reasoning. The cap matters: without it, the model could earn more by rambling, which is the lagoon loop again. Mix a reward yourself.

Score one answer

One sampled answer to a chest X-ray question. Set whether the final answer was right, how well its box overlaps the true abnormality, and how many tokens it reasoned for.

0.50
60

The three terms, the 100-token cap and the 0.6 : 0.2 : 0.2 scales are from the lecture’s reward slide. We model the length term as the smaller of 1 and tokens divided by 100, a simple reading of “up to 1”.

A correct answer with a 0.5 overlap and 60 tokens scores 0.6 + 0.1 + 0.12 = 0.82. A wrong answer with a perfect box and a long reasoning scores 0 + 0.2 + 0.2 = 0.4. The weights make a right answer count for more than the other two together, yet the model still gets a signal for looking in the right place.

When easy cases drown the hard ones

Now the lecture’s central problem. GRPO is critic-free: it treats all samples equally. But medical tasks differ in form (images, time series, text), distribution, scarcity and difficulty. In the slide’s picture four easy questions sit beside one hard one. The result: easy, abundant samples dominate training. The model gets very good at common easy cases, such as chest X-rays, and stays weak on rare hard ones, such as ultrasound.

The slide’s fix: upscale scarce, hard questions and downscale easy, abundant ones. Difficulty is judged by hierarchical clustering, both within and across domains: group similar samples, see which groups the model finds hard, and scale their weight in the loss. The slide labels the resulting method DRPO. The symptom, measurement and fix again: weak scores on rare modalities (symptom), a training batch in which one domain provides most of the gradient (measurement), and reweighting by difficulty and scarcity (fix).

Rebalance the batch

A batch holds four easy questions and one hard one. Raise the weight on the hard one and watch its share of the training signal grow.

1.0

Illustrative: the batch of 4 easy and 1 hard follows the slide’s picture, and each easy question has weight 1. The paper sets weights from clustering, not from a slider.

With equal weights the hard question holds 1 of 5 shares, 20%. At weight 4 it holds 4 of 8, 50%. The easy cases still count; they just stop crowding out the rest.

What the lecture reports

The slides claim that QoQ-Med is the first medical reasoning model that reasons across 2D and 3D images, videos, time series and text. It shows large gains in understudied modalities such as ultrasound and CT scans, and beats both open and closed models, with a chart showing it ahead of GPT-4o and o4-mini. A transfer test uses MIMIC, a dataset with paired image, text and time series, to ask whether training on partial modalities carries over to the full multimodal case. The slide’s table has two columns: LOS (hospital length of stay) and 48-IHM (death within 48 hours of admission, in hospital). On LOS the F1 score is 0.105 for the base Qwen-2.5-VL-7B, 0.105 for standard GRPO, and 0.283 for the full domain-aware model (DRPO-Full). On 48-IHM the accuracy is 0.555, 0.551 and 0.642. So on LOS the plain GRPO run did not move the F1 and the domain-aware reweighting did. The picture is mixed, though: on 48-IHM the F1 rose from 0.293 to 0.354 with plain GRPO, then to 0.597 with the full model.

Clinicians also graded the reasoning text itself. The slides report around 45.3% of the reasoning rated as highly relevant to the final diagnosis, and 100% of the rated samples containing highly relevant reasoning traces. The examples show a brain MRI answer that points to a round, well-circumscribed lesion in a region where tumours are common, and a skin image answer that reasons about a raised, scaly area before naming basal cell carcinoma. The honest reading: the explanations are inspectable, and a clinician still has to check them.

Why does the clinical model upscale hard, scarce questions and downscale easy, abundant ones?

Chapter 11

The Field Guide: Numbers, a Checklist and the Core in Code

Keep the lecture on one page, then debug a reinforcement learning run with the ten habits the slides end on

The lecture ends with ten tips for training RL, then a reminder about the project reports. This chapter gives the summary first, the numbers next, the checklist after that, and the core method in code.

The one-paragraph summary

Reasoning combines knowledge over several steps while using the structure of the problem. It needs aligned representations of each modality, and it breaks into four sub-challenges: structure, concepts, inference and knowledge. You can teach it by prompting (chain of thought, tree of thoughts, Socratic Models), by supervised fine-tuning on traces, or by reinforcement learning from a reward alone. RL treats a model as a policy, scores a whole run, and uses the policy gradient: push up the actions of good runs, push down those of bad runs, and judge each run against a baseline. Actor-critic learns the baseline as V, and PPO clips each update to stay near the old policy. RLHF learns the reward from human comparisons. GRPO drops the critic and the reward model: the group mean is the baseline and a verifier is the reward. Any reward can be hacked. DPO skips the reward model and the sampling loop, and learns from static pairs. Domain-aware scaling lets RL reason across many modalities without the easy cases taking over.

The numbers that matter

WhatNumberWhy it matters
Winoground, CLIP and similar modelsRandom chanceMatching concepts is not using structure
Discounted return, short vs long path (gamma 0.9)2.71 vs 5.90 (our toy)The discount sets how far ahead the agent looks
Tree values on the slideQπ(up) = 3, Q*(up) = 4, Q(down) = 99Compare whole futures, not next rewards
Pong weights with a baseline+1.33, −0.67, −0.67 (our toy)Judge a run against what you expect
PPO clip rangeepsilon 0.2, ratio held to 0.8 to 1.2Each update stays small
Reward model on the slideR = 8.0 vs 1.2, preference 99.89%Only the gap between rewards counts
GRPO group of 8, 3 right+1.29 and −0.77 (our toy)The group mean replaces the critic
Boat agent versus humans20% higher scoreA proxy reward can be beaten without finishing the task
DPO loss at the start0.693Policy equals reference, margin 0
QoQ-Med reward scales0.6 : 0.2 : 0.2Accuracy leads, alignment and length support
CLIMB4.57 million samples, 44 datasetsScale of clinical multimodal data

The checklist: ten habits for training RL

The last two content slides list these. Each has a reason you can now state.

  1. Sanity-check with a fixed policy. Run a policy you understand first, so you know what the reward and the logs should look like.
  2. Monitor the KL divergence in PPO-like methods. A fast rise means updates are too big (Chapter 6).
  3. Plot entropy over time. Entropy measures how spread out the policy’s choices are. A collapse to near zero means it stopped exploring.
  4. Evaluate with greedy rollouts. Pick the top action each step, so the score is not noisy from sampling.
  5. Debug the value function separately. Plot predicted against actual return (Chapter 6’s critic).
  6. Clip the gradient norm. One bad batch should not wreck the weights.
  7. Check the advantage distribution. Mean near 0 and sensible spread; all zeros means no signal (Chapter 8).
  8. Train on a frozen replay buffer. Fix the data to test whether the learner can fit it at all.
  9. Use curriculum learning. Raise the difficulty, or make the reward sparser, step by step.
  10. Watch for mode collapse in mixture-of-experts or multi-head policies, where one expert or head takes over.

The core as code

python
# one GRPO update for one question. tokens: [G, T]; logp tensors: [G, T]
def grpo_step(policy, ref, q, verify, G=8, eps=0.2, beta=0.04):
    outs = [policy.sample(q) for _ in range(G)]            # G answers, same prompt
    r = torch.tensor([verify(q, o) for o in outs])             # [G], e.g. 0 or 1
    A = (r - r.mean()) / (r.std(unbiased=False) + 1e-6)       # group advantage
    logp_new = policy.logp(q, outs)                       # [G, T], with grad
    logp_old = logp_new.detach()                          # sampling policy
    ratio = (logp_new - logp_old).exp()                  # 1.0 on the first pass
    clipped = torch.minimum(ratio * A[:, None],
              ratio.clamp(1 - eps, 1 + eps) * A[:, None])
    kl = (ref.logp(q, outs) - logp_new).exp() - (ref.logp(q, outs) - logp_new) - 1
    loss = -(clipped - beta * kl).mean()                  # one scalar
    loss.backward(); return loss

Where it sits, and where to go next

Prompting, supervised fine-tuning and RL are three rungs of the same ladder, and strong systems use all three. The lecture’s algorithms live on the policy-based side of RL. For the field background, our lessons on Markov decision processes, policy gradients, actor-critic, PPO and SAC and reward learning go deeper on single chapters. For human feedback and DPO side by side, see RLHF and DPO, and for GRPO and its successors GRPO and DAPO. The three readings of this session each have a Veanor: Deep RL from human preferences, DeepSeek-R1 and DPO. The fourth reading, Faulty reward functions in the wild, is taught inside Chapter 8.

The takeaway. Reasoning is learned from a score, and the score is the teacher. A group of samples gives the baseline, a verifier gives the reward, a small clip keeps each step safe, and every hole in the check is a lesson the model will learn. Choose the check as carefully as you choose the model.

Now press Present or Teach and explain, from memory, how a reward becomes a change in a policy: the return, the baseline, the advantage, the clip, and the group. Then go back to the hero and try the sloppy check once more.

Credit: Lecture 9.1 of MIT MMAI, Multimodal AI (Spring 2026), Paul Liang and the MIT Multisensory Intelligence group; slides linked from the course schedule. Everything above is re-taught in our own words and devices. Toy numbers are labelled illustrative.

A training run shows every advantage in every group equal to 0 for many steps. Which checklist habit finds the cause, and what is the likely cause?
MIT MMAI Lecture 9.1: Multimodal Reasoning and Reinforcement Learning
The course schedule and slides · Faulty reward functions in the wild · Deep RL from human preferences · DeepSeek-R1 · DPO · Back to Gleams