Generalist Robot Policies

Chapter 1: VLM Backbones

Where does a generalist robot get its "understanding" of the world? Not from robot data — there isn't enough. Instead, we start with a pre-trained vision-language model (VLM) that has already learned to see and speak from billions of internet image-text pairs.

A VLM provides three superpowers for free:

1. Visual understanding. It knows what a cup looks like, what "the red object on the left" means, and can distinguish a spoon from a spatula — without ever seeing a robot gripper.
2. Language grounding. It maps words like "pick up" or "move to the bowl" to visual concepts. This is the bridge that lets natural language be the task interface.
3. Semantic reasoning. It knows that cups go on saucers, that liquids pour, that drawers slide open. This common-sense knowledge is impossible to learn from a few thousand robot episodes alone.

The key architectural insight: a VLM is already a sequence model. It takes in image tokens and text tokens, processes them through a transformer, and outputs text tokens. To make it a VLA, we need one more output — action tokens.

Two concrete architectures dominate this space:

The tokenized approach is simpler — you literally treat actions as extra vocabulary. But discretization throws away precision, and autoregressive decoding is slow (one token at a time). The continuous action head approach is more complex but produces smoother, faster actions. This is what π0 and SmolVLA use, and what we will focus on.

Chapter 2: Data at Scale

Language models became powerful when data went from millions to trillions of tokens. Can the same scaling story work for robotics? The answer, circa 2024, is a cautious "yes" — but the data landscape looks very different.

The landmark effort is Open X-Embodiment (OXE), an aggregation of robot datasets from over 20 research labs. It contains:

That sounds like a lot — until you compare it to language. One million episodes, each maybe 5–30 seconds of data, gives you roughly 10,000 hours of robot experience. GPT-4 was trained on the equivalent of millions of years of reading. Robotics data is still many orders of magnitude smaller.

A second major effort is DROID (Distributed Robot Interaction Dataset): 76,000 episodes across 564 tasks, collected in 52 different labs using a standardized setup. What makes DROID special is consistency — same camera positions, same action format, same annotation quality.

But raw quantity isn't enough. Robot datasets have unique challenges:

• Heterogeneous action spaces. One dataset uses joint angles, another uses end-effector deltas, another uses absolute Cartesian poses. You need a common format.
• Variable observation spaces. Some robots have one camera, others have three. Some include proprioceptive state, others don't.
• Quality variation. Human demonstrations range from expert-quality to messy. Some datasets are teleoperated, others are scripted.

The practical solution: standardize on end-effector pose deltas in SE(3) (position change + rotation change + gripper state) as the action representation. This is what π0, SmolVLA, and LeRobot all converge on. A 3D position delta and a rotation delta mean the same thing regardless of whether the underlying robot has 6 joints or 16.

Chapter 3: π0 Architecture

Now we get to the model that put VLAs on the map. π0 (pi-zero), from Physical Intelligence (2024), is the first VLA to demonstrate genuine cross-task, cross-embodiment generalization at scale. Let's take it apart piece by piece.

The architecture has two halves, and understanding why they are separate is the key insight:

The data flow through π0:

The action expert uses flow matching, not diffusion. Recall from Chapter 2 that flow matching learns straight-line paths from noise to data, making denoising faster. At inference, π0 needs only 10 denoising steps (vs 50–100 for DDPM-style diffusion) to generate a full action chunk. Each chunk is 50 timesteps of 7D actions, so a single forward pass produces roughly 1–2 seconds of future motion.

Chapter 4: π0 Training

Having a good architecture is half the battle. The other half is training it correctly — and π0's training recipe is surprisingly nuanced.

Phase 1: Pre-training the backbone

PaliGemma 3B comes already pre-trained on internet-scale image-text data. This gives the VLM backbone its visual and language understanding. π0 inherits all of this for free. No robot data needed for this phase.

Phase 2: Robot fine-tuning

Now we fine-tune the full model (VLM + action expert) on robot demonstration data. The training set is massive by robotics standards:

The action expert's loss

The action expert is trained with a flow matching objective. Given a ground-truth action chunk a₀ from a demonstration and a random noise vector a₁, we create an interpolated point at time t:

The model predicts the velocity field — the direction from noise to data:

At inference time, we start from pure noise a₁ and take 10 Euler steps along the learned velocity field to arrive at a clean action chunk. Fewer steps than diffusion, straighter paths, faster inference.

LeRobot code

python
# Using pi0 with LeRobot
from lerobot.common.policies.pi0.modeling_pi0 import PI0Policy
from lerobot.common.policies.pi0.config_pi0 import PI0Config

# Load pre-trained pi0
config = PI0Config()
policy = PI0Policy(config)

# At inference: pass observation dict
observation = {
    "observation.images.top": image_tensor,     # [B, C, H, W]
    "observation.state": proprio_tensor,        # [B, state_dim]
}
action_chunk = policy.select_action(observation)
# action_chunk shape: [B, chunk_size, action_dim]
# e.g., [1, 50, 7] = 50 steps of 7D actions

Chapter 5: SmolVLA

π0 is impressive, but it has a problem: 3 billion parameters. That requires an A100 or H100 GPU just for inference, and fine-tuning demands even more. Most robotics labs — and certainly most hobbyists with a LeRobot SO-100 on their desk — don't have that kind of hardware.

Enter SmolVLA (Small Vision-Language-Action), developed by HuggingFace. The pitch is radical: 250 million parameters — 12 times smaller than π0 — that achieves competitive performance on standard manipulation benchmarks.

How does SmolVLA get away with being so small? Three design choices:

1. Efficient backbone. Instead of PaliGemma 3B, SmolVLA uses SmolVLM — a compact VLM designed for edge deployment. It packs visual and language understanding into ~200M parameters.
2. Diffusion action head (not flow matching). A lightweight DiT (Diffusion Transformer) generates action chunks. The diffusion process is well-understood and stable at small scales.
3. Aggressive image compression. SmolVLA uses fewer image tokens per frame (e.g., 64 vs. 256), reducing the compute bottleneck in the transformer backbone.

The benchmark results are striking. On the SimplerEnv manipulation suite, SmolVLA achieves within 5–10% of π0's success rate on most tasks, at 1/12th the parameter count. On some tasks (like "pick up cube"), it actually outperforms π0 — likely because the simpler model overfits less on easy tasks.

Chapter 6: SmolVLA Architecture Details

Let's open the hood and trace exactly how data flows through SmolVLA, from raw pixels and text to motor commands.

Stream 1: Vision (SigLIP)

Camera images enter through a SigLIP vision encoder — the same family used in PaliGemma, but a smaller variant. Each 224×224 image is divided into 14×14 patches, and each patch becomes a token. A learned projection compresses these into the model's hidden dimension.

The compression from 196 to 64 tokens is critical for efficiency. SmolVLA uses a perceiver-style pooling layer that learns to summarize visual information into fewer tokens without losing task-relevant details.

Stream 2: Language (SmolLM)

The language instruction is tokenized and processed by SmolLM, a compact language model (~125M parameters). This produces contextualized text embeddings that encode what the robot should do.

Stream 3: Actions (Diffusion Transformer)

The action head is a small Diffusion Transformer (DiT). It takes three inputs:

1. VLM embeddings (fused vision + language) as conditioning
2. Noisy action chunk a_t (Gaussian noise at start, progressively denoised)
3. Timestep embedding t (tells the model how much noise remains)

The DiT uses cross-attention to attend from action tokens to VLM embeddings. This is how the action head "reads" what the VLM sees and understands.

Fine-tuning SmolVLA with LeRobot

python
# Fine-tuning SmolVLA on your own dataset with LeRobot
from lerobot.common.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from lerobot.common.policies.smolvla.config_smolvla import SmolVLAConfig
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

# Load your custom dataset (collected via teleoperation)
dataset = LeRobotDataset("your_hf_username/pick_place_cups")

# Configure SmolVLA for your robot
config = SmolVLAConfig(
    input_shapes={
        "observation.images.top": [3, 224, 224],
        "observation.state": [7],  # 6 joints + gripper
    },
    output_shapes={
        "action": [7],  # same as state
    },
    chunk_size=50,
)

# Initialize and train
policy = SmolVLAPolicy(config)
# Training loop handles flow matching loss internally
# ~8 hours on a single RTX 4090 for 50K steps

Chapter 7: Cross-Embodiment Transfer

Here's where things get genuinely surprising. You train a policy on data from one robot and deploy it on a completely different robot — and it works. Not perfectly, but far better than random. How is this possible?

The key is the shared action representation. When every robot's actions are expressed as end-effector pose deltas in SE(3) — "move the gripper 2cm right, rotate 5 degrees, close gripper" — the action space is the same regardless of the underlying kinematics.

A concrete example from the LeRobot ecosystem:

Even though the raw joint spaces are completely different, the task-level representation — "how should the end-effector move?" — is shared. If the model learned on SO-100 that "pick up" means "move down, close gripper, move up," that same sequence of SE(3) deltas works on ALOHA-2's right arm.

The practical recipe for cross-embodiment transfer:

1. Pre-train on the union of all available datasets (OXE + DROID + your data), all converted to SE(3) end-effector deltas.
2. Fine-tune on a small amount of target robot data (even 50–100 episodes can be enough for known tasks).
3. Zero-shot deployment works for simple tasks (reaching, pushing) but fine-tuning is needed for precise manipulation.

Chapter 8: The Frontier

We have built all the pieces: VLM backbones, massive datasets, flow matching action heads, cross-embodiment training. But the field of generalist robot policies is still in its infancy. Here are the open questions that will define the next 3–5 years.

Scaling laws for robot data

Language models follow predictable scaling laws: double the data, performance improves by a known amount. Do robot policies follow similar curves? Early evidence from RT-2 and π0 suggests yes, but the slope is different. Visual-motor tasks saturate faster than language tasks — you don't need a trillion episodes to learn "pick up the cup." The question is where each task's saturation point lies.

Simulation as data engine

Can simulation data substitute for real data? The answer is nuanced:

The emerging consensus: sim-to-real-to-sim. Train in sim, deploy on real robot, use the real experience to improve the simulator, repeat. Each loop narrows the domain gap. The simulator becomes a better training ground, and the real robot provides grounding.

Open challenges

1. Long-horizon tasks. Current VLAs handle 10–30 second tasks. Making a sandwich requires 5+ minutes of coordinated manipulation. How do you plan, recover from errors, and maintain context over that timeframe?
2. Safety and robustness. A language model that hallucinates is embarrassing. A robot that applies 50N of force in the wrong direction is dangerous. How do we guarantee safety bounds?
3. Emergent capabilities. GPT-3 unexpectedly learned to do arithmetic. What unexpected capabilities might emerge from robot foundation models trained on diverse enough data?
4. Continual learning. How does a deployed robot improve from its own experience without forgetting what it already knows? This is catastrophic forgetting at deployment scale.

Chapter 9: Connections

We've covered the full arc of the Robot Learning Tutorial — from collecting your first demonstrations to training generalist foundation models. Let's pull it all together.

The Book at a Glance

Concept Cheat Sheet

The Stack

Where to Go Next

• Build something. Get a LeRobot SO-100, collect 50 episodes of a pick-and-place task, fine-tune SmolVLA, and deploy. The tutorial's value is in doing, not just reading.
• Read the papers. π0 (Black et al., 2024), SmolVLA (Zouitine et al., 2025), Open X-Embodiment (Padalkar et al., 2024), ACT (Zhao et al., 2023).
• Join the community. HuggingFace LeRobot Discord, the LeRobot GitHub repo, and the emerging ecosystem of shared datasets.