Foundation Models in Robotics

Chapter 1: What Is a Foundation Model?

Before we wire one into a robot, let us pin down the term. The phrase comes from a 2021 Stanford report, and its definition is deliberately broad:

Two halves matter. Broad data at scale: not a curated dataset for one job, but the whole web — text, images, code. Self-supervision: the label is built into the data itself, so no human has to annotate it. For a language model the task is "predict the next word"; the next word is the label, free, for every sentence ever written. That is how you reach trillions of training examples without an army of labelers.

The lecture splits the foundation models that matter for robotics into two families:

CLIP deserves a special mention because it bridges both worlds, and we'll use it heavily in Chapter 4. CLIP was trained on hundreds of millions of (image, caption) pairs from the web to put an image and its matching text near each other in a shared embedding space. So you can hand CLIP the word "sponge" and an image, and it tells you how well they match — without ever having been trained on a labeled "sponge" dataset. That is the open-vocabulary superpower: it recognizes things nobody explicitly taught it, because the web already described them.

What "frozen" means — and why it matters

A theme you'll see in every system: the big pretrained model is usually frozen (its weights are not changed during robot training), while a small head — the part that actually talks to motors — is the only thing trained on the scarce robot data. Why? Because you have billions of web examples behind the frozen part and maybe a few hundred robot examples for the head. Train the whole thing on a few hundred examples and you'd wash away everything the web taught you. Freeze the knowledge; train only the connection. That single design rule explains most of the architectures ahead.

Chapter 2: Grounding — The Word the Whole Lecture Turns On

We keep saying it; now let's make it precise. Grounding means associating the output of a foundation model with the real robot and its environment. The model speaks in words and pixels; grounding ties those to things the robot can sense and do.

The lecture lists the flavors of grounding, because each system grounds a different way:

• Ground language to affordances — tie "pick up the sponge" to can I actually pick up the sponge from here? (SayCan, Chapter 3).
• Ground language to pretrained skills — the model may only propose actions the robot already knows how to do.
• Ground language to a reward function — let the model write the objective a controller then optimizes.
• Ground language to scene features — CLIP tells the manipulation network where the sponge is (CLIPort, Chapter 4).

The cleanest grounding signal: an affordance

The single most important grounding tool in this lecture is the affordance: a number saying how possible is this action, right now, from this state? "Pick up the sponge" has high affordance when a sponge is within reach and low affordance when there's no sponge in sight.

Where does that number come from? Here is the beautiful trick the SayCan authors use. Train a skill with reinforcement learning where the reward is 1 if the skill succeeds and 0 if it fails. Then the skill's value function — the expected return from this state — is the probability the skill will succeed from here. A value function trained on success/fail reward is, literally, an affordance function. We met value functions in the RL lectures; here they get a second life as the robot's sense of "can I?"

Chapter 3: SayCan — Say × Can

Now we can build our first real system. SayCan's slogan is its whole idea: Do As I Can, Not As I Say. The language model says what is sensible; an affordance function says what is possible; you multiply them and pick the winner.

The data flow, concretely

The robot is given an instruction i ("I spilled my drink, help") and owns a fixed library of skills — in the paper, 551 of them across 7 families like pick, place, open drawer, go to. Each skill carries a short language label like "find a sponge" and a low-level policy that can execute it. For each candidate skill the robot computes two numbers:

• Say — the LLM's score p(skill label | instruction): how likely is this skill a sensible next step toward the goal? PaLM-540B, frozen, scores each candidate label. "Find a sponge" scores high for a spill; "find an apple" scores low.
• Can — the affordance p(success | state, skill label): from a learned value function, how likely is this skill to actually succeed from where the robot is right now? If no sponge is visible, "find a sponge" might still score high here (the robot can go look), but "pick up the sponge" scores near zero.

The combined score is simply the product, and the robot picks the highest:

The factorization is honest about why you multiply: the chosen skill makes progress only if it is both the right thing to try (Say) and likely to work from here (Can). A skill that fails the affordance contributes nothing. Execute the winning skill, then loop: re-score everything from the new state, pick the next step, until the LLM proposes "done."

A number to make it concrete

Suppose the instruction is "I spilled my drink, help," the sponge is across the room, and we score two candidate skills. The LLM thinks both are roughly sensible first moves, so it gives find a sponge a Say of 0.40 and pick up the sponge a Say of 0.42 — almost a tie, and the language model slightly prefers the more direct-sounding one. Now multiply by the affordances. Find a sponge is feasible (just walk and look), affordance 0.95. Pick up the sponge is impossible from here, affordance 0.02.

The product is decisive where the language score alone was a coin flip. Find a sponge wins by roughly fifty to one, even though the LLM marginally preferred the other. That is the entire point of multiplying: the affordance vetoes the impossible, no matter how eloquent the language model is about it. Run the skill, walk over, and on the next loop the sponge's affordance has jumped — now pick up the sponge wins. The plan unfolds one grounded step at a time.

Why is this a loop and not a one-shot plan? Because the affordance is recomputed from the current state every step. If the robot bumps the sponge off the table mid-task, the next round's affordances reflect that new reality, and SayCan re-plans without anyone telling it to. It inherits the closed-loop spirit from Lecture 1: re-sense, re-decide, act, repeat.

With the sponge far away (slider low), "pick up the sponge" has high Say but near-zero Can, so SayCan instead picks "find a sponge" — the sensible first move. Walk over, the slider effectively rises, and on the next loop "pick up the sponge" wins. The plan emerges step by step from Say × Can, never from Say alone. Let's implement exactly that scoring step.

Chapter 4: CLIPort — What Meets Where

SayCan decides which skill to run. But a skill like "pick up the sponge" still has to figure out exactly where to put the gripper, down to a centimeter. That is a different kind of grounding: language to spatial precision. CLIPort is the lecture's answer, and its design is a clean lesson in combining two foundation-model strengths.

The authors borrow the two-stream hypothesis from cognitive psychology: the brain has a "what" pathway (recognizing objects) and a "where" pathway (locating and acting on them). CLIPort builds both:

How Transporter works — detect actions, not objects

The key Transporter idea is to predict affordance heatmaps over the image instead of estimating object poses. Concretely:

• Pick: a network outputs a heatmap over every pixel — the value at each pixel is "how good is it to pick here?" The argmax is the pick location.
• Place: crop a patch around the chosen pick point, then slide (and rotate) that crop across the image to find where it best fits — literally template matching, like visual tracking. The best-matching location and rotation become the place pose.

This action-centric view is wildly data-efficient: CLIPort learned 9 real-world tasks from just 179 image-action pairs. You never have to model "objectness"; you only have to learn where to act.

CLIPort's contribution is to inject CLIP's features into the Transporter streams, so the heatmaps become language-conditioned. "Pack the blue block" and "pack the red block" produce different heatmaps over the same image, because CLIP's semantic features steer where Transporter looks. The spatial network stays sharp; the semantic network makes it understand words.

Why the place step needs rotation, concretely

The pick step is the easy half: pick the pixel with the highest pick affordance and lower the gripper there. Placing is subtler, because orientation matters — a "put the rook one block forward" or "align the rope" task needs the object set down at the right angle, not just the right spot. Transporter handles this by taking the cropped patch from around the pick point and testing it at many rotations as it slides across the image. The place location and the place angle are read off together as the best-matching pose. So a single forward pass yields both where to set the object down and how to turn it — without ever estimating an object's pose explicitly.

And the data flow is satisfyingly uniform: the network's input is the RGB-D image plus the (CLIP-encoded) instruction; its output is two dense affordance maps, one for pick and one for place. No bounding boxes, no segmentation masks, no symbolic state. The whole tabletop manipulation problem is reframed as "which pixels, at which rotation, are good to act on?" That is what made it possible to learn ten simulated tasks — and one shared multi-task policy across all of them — from a tiny number of demonstrations.

The semantic match is the engine here, so let's build it. CLIP scores how well an instruction's embedding matches each object's embedding — cosine similarity in the shared space — and the manipulator focuses on the winner.

Chapter 5: RT-1 — Tokenize Everything

SayCan plans; CLIPort places. RT-1 (Robotics Transformer 1) takes the most direct route of all: train one big Transformer to map images + instruction directly to actions, end-to-end, the way GPT maps text to text. The bet is the Bitter Lesson made flesh — a high-capacity model fed enough diverse data will learn the patterns better than any hand-built pipeline.

The data flow, shapes and all

The decisive design choice — and the one we'll implement — is how RT-1 represents that 11-number action. It does not regress continuous values. Instead it discretizes each action dimension into 256 bins and predicts a bin, exactly like predicting a word token. The control problem becomes a classification problem: "which of the 256 buckets does this joint command fall into?"

What's frozen vs. trained

The EfficientNet starts from ImageNet pretraining (a vision foundation model) and is then trained along with the rest. The whole RT-1 (a modest 35M parameters) is trained on a huge robot dataset: 130k demonstrations, 700+ tasks, collected with 13 robots over 17 months. Note the contrast with SayCan and CLIPort — RT-1 doesn't freeze a giant LLM; it pours scale into the robot data and the visual backbone. It hit 97% success on training tasks and generalized markedly better than baselines to new tasks, distractors, and backgrounds. It also plugs straight into SayCan as the low-level skill executor.

Two design pressures pull against each other here, and the architecture is the compromise. A Transformer is a high-capacity sponge — exactly what you want to absorb 700 tasks — but Transformers are slow, and a robot controller has to emit an action many times a second or the arm jerks. The TokenLearner is the resolution of that tension: instead of feeding the Transformer hundreds of image patches, it learns to keep only the ~8 tokens that matter for the current task, so each forward pass is cheap enough to hit 3 Hz. That is the same "borrow capacity, but make it run in real time" theme you'll see strain to breaking point with the 55-billion-parameter VLA in Chapter 7.

And notice what RT-1 is really doing about the data problem from Lecture 4. It does not escape it — somebody still acted out 130k demonstrations the slow, physical way. What it changes is the payoff for that data: a single high-capacity model trained on all of it together generalizes far better than 700 narrow models trained one task at a time. The connections between structurally similar tasks become training signal. This is the Bitter Lesson in robot form: pour diverse data into a general model and let it find the patterns.

Notice the round-trip error never exceeds half a bin width — that's the resolution price of turning control into classification. With 256 bins (RT-1's choice) over a normalized range, that error is tiny. Let's prove that bound in code.

Chapter 6: Code as Policies — Let the LLM Program the Robot

SayCan let the LLM pick from a fixed menu of skills. But what if the task needs logic the menu doesn't cover — loops, conditionals, arithmetic, "stack the blocks on the empty bowl"? The Code-as-Policies insight is delightful: code-writing LLMs (trained on billions of lines of GitHub) can write the robot's control program directly.

The loop, concretely

You give the LLM a few example commands paired with their policy code (few-shot prompting), plus the signatures of two API libraries:

• Perception APIs — functions like detect_objects("blocks"), get_obj_pos(name), is_empty(bowl), backed by open-vocabulary detectors.
• Control APIs — primitives like pick_place(obj, target), move_to(pos), set_velocity(x, y, z).

Then you hand it a new command, and it writes fresh Python that calls those APIs to accomplish it. For "stack the blocks on the empty bowl" it might generate:

block_names = detect_objects("blocks")
bowl_names = detect_objects("bowls")
for bowl_name in bowl_names:
    if is_empty(bowl_name):
        empty_bowl = bowl_name
        break
objs_to_stack = [empty_bowl] + block_names
stack_objects(objs_to_stack)

Two things to notice. First, the LLM never touched a motor — it grounds language by writing calls to APIs that do the sensing and acting. The code is the grounding. Second, it can recursively define functions it used but weren't given (is_empty, stack_objects), writing helper code on demand — "hierarchical code-gen."

Chapter 7: Toward VLAs — Actions as Another Language

Step back and look at the trend across the four systems. SayCan keeps the LLM and the robot as separate boxes wired together. CLIPort fuses two pretrained streams. RT-1 trains one Transformer that tokenizes actions. The natural endpoint: one model that ingests vision and language and emits actions — a Vision-Language-Action (VLA) model. This is the bridge to Lecture 6's discussion of diffusion policies and modern generalist robots.

RT-2 is the lecture's flagship VLA. Its move is audacious and simple: take a giant vision-language model already trained on the web (one that answers questions about images), and fine-tune it on robot data with actions written as text tokens. An action like "move arm to (x, y, z), rotate, close gripper" becomes a little sentence of numbers. Control is now literally a vision-question-answering problem where the "answer" is an action string.

The honest engineering cost

It is not free. RT-2's largest version uses a 55-billion-parameter backbone — far too big to run on the robot — so it lives in a cloud TPU service queried over the network, returning actions at only 1–3 Hz. A smaller 5B version reaches ~5 Hz. Compare that to RT-1's 3 Hz from 35M parameters: the bigger the borrowed brain, the slower (and more expensive) the loop. This is the perennial robotics tension — real-time control versus model capacity — and it is wide open.

Chapter 8: Does It Actually Help? — Mapping Tools to Challenges

The guest lecture ends by returning to its opening question: have foundation models made general-purpose robots closer? The honest answer is "on several fronts, yes — and on one, not yet." Recall the challenges from Lecture 4 and the lecture's own list; here is which tool attacks which.

A way to picture the payoff

The promise is a step-change in breadth. A classical or single-task learned robot is excellent on the narrow slice it was built for and useless outside it. A foundation-model robot trades a little peak performance on any one task for the ability to attempt a vast range of tasks zero-shot — because it borrowed the web's breadth. The frontier is closing that small peak-performance gap while keeping the breadth, and adding the safety layer that is still missing.

Chapter 9: Connections & Cheat Sheet

You started with a spilled drink and a robot that didn't know what a sponge was for. You now know four distinct ways to lend it the web's common sense — and why each one needs grounding to keep that knowledge from floating free of reality.

Where to go next on this site

This lesson is Lecture 24 of the CMU 16-831 series. Continue and go deeper:

• Lecture 23 — Multi-task, Adaptation & Transfer — the previous lecture; how a single policy spans many tasks and adapts on the fly.
• Lecture 25 — Course Summary — how every thread of 16-831 ties together.
• Vision-Language-Action Models — a full Gleam on the VLA family RT-2 launched.
• Vision-Language Models — the VLM backbones (PaLI-style) that RT-2 fine-tunes.
• Contrastive Learning & CLIP — how the shared image-text space (the engine behind CLIPort's "what") is actually trained.

"The robot can be the language model's hands and eyes, while the language model supplies the knowledge of the world."