Understand Vision-Language-Action
Models

Chapter 1: Behavioral Cloning

The simplest way to teach a robot: show it what to do. A human demonstrates the task (teleoperation), recording observations and actions. Then we train a neural network to predict the expert's action given the current observation. This is behavioral cloning (BC) — supervised learning for robotics.

In practice, BC training data looks like this: each episode is a sequence of (observation, action) pairs. The observation is a camera image [3, 224, 224]. The action is a 7-DOF vector [Δx, Δy, Δz, Δrx, Δry, Δrz, gripper]. A 10-second demonstration at 10Hz gives 100 training pairs. Typical VLA datasets have 100K-1M such episodes.

BC is simple but fragile. If the robot drifts slightly off the demonstrated path, it encounters states it has never seen before — the distribution shift problem. Small errors compound, and the robot spirals into failure.

How bad is compounding error in practice? If each step has a 1% chance of a small drift, after 100 steps the probability of being on-track is 0.99¹⁰⁰ = 36%. After 500 steps: 0.66%. This is why long-horizon tasks (cook a meal = thousands of steps) are so hard for BC alone. The fix: either make the policy so good that per-step error is near zero (scale + data + architecture), or add error recovery mechanisms (re-planning from new observations, action chunking).

python
# Behavioral cloning training loop (simplified)
for obs, action in demonstration_dataset:
    # obs: [3, 224, 224] camera image
    # action: [7] expert's 6-DoF + gripper command
    predicted = policy(obs)           # [7] predicted action
    loss = F.mse_loss(predicted, action)
    loss.backward()
    optimizer.step()

Chapter 2: Action Representations

How do you represent "move the arm"? This choice is critical — it determines what the model predicts, how precise it can be, and whether it transfers across robots. There are two fundamental frames: joint space (the angles of each motor) and task space (the position and orientation of the end-effector in the world).

A 7-DoF arm (like the Franka Panda) has 7 joint angles, each with a different range. Delta actions are the most common choice for VLAs: the model predicts how much to change each dimension, not the absolute target. A typical delta is tiny — (Δx=0.01m, Δy=−0.005m, Δz=0.02m) per timestep. This keeps predictions small and centered around zero, which is easier for neural networks to learn.

python
# Action discretization (RT-2 style)
action_range = [-1.0, 1.0]          # normalized action space
n_bins = 256

def discretize(a):
    # Continuous [-1, 1] → bin index [0, 255]
    return int((a + 1) / 2 * (n_bins - 1))

def undiscretize(bin_idx):
    # Bin index [0, 255] → continuous [-1, 1]
    return bin_idx / (n_bins - 1) * 2 - 1

# Example: dx=0.3 → bin 166 → token "166"

python
def tokenize_action(action, action_low, action_high, n_bins=256):
    tokens = []
    for i in range(len(action)):
        # Normalize to [0, 1]
        norm = (action[i] - action_low[i]) / (action_high[i] - action_low[i])
        # Clamp to [0, 1] (safety for out-of-range actions)
        norm = max(0.0, min(1.0, norm))
        # Map to bin index [0, n_bins-1]
        token = int(round(norm * (n_bins - 1)))
        tokens.append(token)
    return tokens

def detokenize_action(tokens, action_low, action_high, n_bins=256):
    actions = []
    for i in range(len(tokens)):
        # Bin index to [0, 1]
        norm = tokens[i] / (n_bins - 1)
        # Denormalize to robot-specific range
        val = norm * (action_high[i] - action_low[i]) + action_low[i]
        actions.append(val)
    return actions

Chapter 3: RT-2 — Language as Action

RT-2 (Robotics Transformer 2) by Google DeepMind is the landmark paper that proved VLMs can directly control robots. The key insight: fine-tune a VLM so that instead of generating text, it generates action tokens.

RT-2 takes a PaLM-E or PaLI-X vision-language model, discretizes robot actions into 256 bins per dimension, maps them to string tokens ("128", "64", "255"), and co-fine-tunes on robot demonstrations alongside web-scale vision-language data. The model's output for one timestep is literally a text string like "1 128 91 241 5 127 100" — seven numbers that get de-tokenized into 7-DOF continuous actions. No special action head. No architectural change. Just clever tokenization.

The training data mix is crucial. RT-2 co-trains on web-scale vision-language data (image captioning, VQA) alongside ~130K robot episodes. The VLM data keeps the model's language understanding intact while the robot data teaches it to output actions. Without the VLM co-training, the model forgets how to understand language. Without the robot data, it can't produce valid actions. The balance matters: typically ~50% VLM data, ~50% robot data per batch.

Chapter 4: Diffusion Policies

What if instead of predicting a single action, the robot could sample from a distribution over actions? Diffusion Policy applies the same denoising process used in image generation to robot action prediction.

Starting from random noise, the model iteratively refines an action trajectory. This naturally handles multimodal action distributions — when there are multiple valid ways to do something (reach from the left or right), the diffusion model can represent all of them.

Concretely, the input is noise of shape [k, action_dim] — say [50, 7] for 50 future timesteps of 7-DOF actions. The model runs 10-20 denoising steps. Each step predicts a velocity field and applies an Euler update: at+1 = at + v(at, t) · dt. At ~2ms per step × 10 steps, the total denoising takes ~20ms — fast enough for real-time control.

python
# Flow matching action denoising (simplified)
a = torch.randn(k, action_dim)     # [50, 7] random noise
for t in torch.linspace(0, 1, steps=10):
    v = model(a, t, obs, instruction)  # predict velocity
    a = a + v * (1.0 / 10)          # Euler step, ~2ms
# a is now [50, 7] clean action trajectory

Chapter 5: Action Chunking

Predicting one action at a time is reactive and jerky. Action Chunking (from ACT — Action Chunking with Transformers) predicts an entire sequence of future actions at once. Instead of "what should I do now?", the model answers "what should I do for the next k steps?"

The model output shape is [k, action_dim] — for example, [50, 7] means 50 future actions at 50Hz, covering exactly 1 second of motion. The robot executes these actions open-loop (without re-reading the camera), then re-plans from the new observation.

Why chunk instead of predicting one step at a time? Two big reasons:

python
# Action chunking with temporal ensemble
chunk = model(observation)           # [k, 7] = 50 future actions
for i in range(replan_interval):     # execute 5-10 steps
    a_blend = w_new * chunk[i] + w_old * prev_chunk[i + offset]
    robot.execute(a_blend)           # 50Hz motor commands
# Then replan from new observation

The chunk size k is a critical hyperparameter. Too small (k=1) and you're back to single-step prediction with all its jitter. Too large (k=200) and the robot is flying blind for 4 seconds before it re-observes the world — if something changes (a human moves an object), the robot won't notice. In practice, k=10-50 at 50Hz (0.2-1.0 seconds) works well for manipulation. Fast tasks (catching) need smaller chunks; slow tasks (assembly) tolerate larger ones.

Chapter 6: OpenVLA Architecture

OpenVLA is the open-source counterpart to RT-2: a 7B-parameter VLA built on Llama 2 + SigLIP, trained on the Open X-Embodiment dataset. It demonstrated that smaller, open models can rival proprietary giant VLAs.

The data flow in concrete shapes: a 224×224 image enters SigLIP and becomes 256 visual tokens of dimension 4096 (after MLP projection). The text instruction is tokenized into ~20-30 tokens. Llama 2 processes all ~280 tokens autoregressively and outputs 7 special action tokens. Each action token is one of 256 possible values (bins), which gets linearly de-mapped to a continuous float in [-1, 1], then scaled to the robot's action range.

python
# OpenVLA inference (simplified)
image = camera.capture()                    # [3, 224, 224]
visual_tokens = siglip(image)               # [256, 1024]
visual_tokens = mlp_proj(visual_tokens)     # [256, 4096] (match LLM dim)
text_tokens = tokenizer("pick up the cup") # [22, 4096]
input_seq = concat(visual_tokens, text_tokens)  # [278, 4096]
action_tokens = llama2.generate(input_seq, n=7) # 7 ints in [0, 255]
action = undiscretize(action_tokens)        # [7] floats in [-1, 1]
robot.execute(action * action_scale)        # send to robot

OpenVLA takes the full fine-tune path with LoRA-style efficient tuning. π₀ (Physical Intelligence) takes the third path: its transformer has separate feed-forward experts for different modalities, so action tokens get their own parameters while vision and language tokens keep their pre-trained weights intact.

python
# Mixture-of-Transformers (pi0-style, simplified)
class MoTLayer(nn.Module):
    def forward(self, x, modality_mask):
        # Shared attention across all tokens
        x = self.shared_attn(x)
        # Separate FFN per modality
        out = torch.zeros_like(x)
        out[modality_mask == 'vision']  = self.vision_ffn(x[modality_mask == 'vision'])
        out[modality_mask == 'text']    = self.text_ffn(x[modality_mask == 'text'])
        out[modality_mask == 'action']  = self.action_ffn(x[modality_mask == 'action'])
        return out

Chapter 7: Cross-Embodiment

A human can watch someone else cook and learn the recipe, even though their arms are different lengths. Can robots do the same? Cross-embodiment learning trains on data from many different robot types, hoping that high-level task knowledge transfers even when the hardware differs.

The Open X-Embodiment (OXE) dataset combined demonstrations from 22+ robot embodiments: single-arm manipulators, dual-arms, quadrupeds, dexterous hands. The finding: a single policy trained on all this data outperforms robot-specific policies on most tasks.

The challenge: different robots have different action spaces. A Franka Panda has 7 joints. A WidowX has 6. A quadruped has 12 (3 per leg). How does a single model handle this? The common approach is to normalize to end-effector deltas: regardless of the underlying robot, the model predicts Δxyz + Δrotation + gripper. The robot's own IK solver converts this to joint commands. This way, the VLA never needs to know joint configurations — it thinks in task space.

python
# Cross-embodiment action normalization
def normalize_action(action, embodiment_config):
    # Different robots have different action ranges
    # Franka: dx in [-0.05, 0.05]m, Widowx: dx in [-0.03, 0.03]m
    low, high = embodiment_config['action_range']
    return (action - low) / (high - low) * 2 - 1  # map to [-1, 1]

# During training: normalize all actions to [-1, 1]
# During inference: denormalize back to robot-specific range

Chapter 8: Sim-to-Real Transfer

Real robot data is expensive and slow to collect. A single human demonstrator might produce 10 episodes per hour — at that rate, 100K episodes takes 10,000 hours of human labor. Simulation is cheap and infinitely scalable: a GPU cluster can generate 100K episodes in hours. But policies trained in sim often fail in the real world — the reality gap. Visual differences (lighting, textures), physics mismatches (friction, contact), and sensor noise all contribute.

This is a fundamental shift from pre-VLA sim-to-real. Traditional approaches had to randomize textures and lighting until the sim "covered" the real distribution. With a foundation-model vision encoder, the representations are already invariant to these variations. The remaining gap is in physics: contact dynamics, friction, deformable objects. Domain randomization still helps here, but the visual gap — historically the hardest part — is largely solved by pre-training.

The practical workflow for a VLA team today: (1) collect ~50-100 real demos for a new task (~5 hours of teleop), (2) optionally augment with 10K sim demos using domain randomization, (3) fine-tune the pre-trained VLA on this combined dataset for a few hours on 8 GPUs, (4) deploy and iterate. The pre-trained VLA already knows general manipulation from OXE data; the fine-tuning teaches it the specific task and environment. This recipe can get a new pick-and-place task working in under 24 hours.

Chapter 9: The Embodied Future

We're at the beginning of the VLA era. Current models can follow simple instructions in controlled settings, but the gap to human-level dexterity and generalization remains vast. Let's ground the current state in real numbers.

Key open challenges: