How foundation models learned to move robots — translating pixels and instructions into physical actions in the real world.
Vision-Language Models (VLMs) can look at an image and answer questions. But what if instead of answering with words, the model answered with motor commands? That's the leap from VLM to VLA (Vision-Language-Action): a model that sees the world, understands a language instruction, and outputs physical actions.
The insight is deceptively simple: if a VLM can generate the token sequence "pick up the red cup," why can't it instead generate the action sequence [move_to(0.3, 0.5, 0.2), close_gripper()]? In token-based VLAs, actions are another output modality encoded as symbols. Continuous-action designs instead predict or generate numerical controls.
For a concrete toy ViT-B/16, an RGB image has shape [3, 224, 224]. Each patch is 16×16 pixels, so the grid is 14×14 = 196 positions, each encoded at width 768. A learned projector maps these features into the chosen language-model width d. The instruction tokenizer returns L integer IDs; an embedding lookup produces [L, d]. A policy conditions on both inputs and produces actions: for our single-arm Cartesian interface, six motion values plus a gripper value.
Schematic: a reused VLM backbone can be trained for different output tasks: text for VLMs, actions for VLAs.
For token-based VLAs, a pretrained VLM can be fine-tuned to decode action symbols with next-token cross-entropy. Existing vocabulary tokens may be repurposed rather than extended. This is one useful design: π0 adds a continuous action expert, while RT-1 used EfficientNet, instruction-conditioned FiLM, TokenLearner and a transformer rather than a pretrained generative VLM.
Where else have you seen "same architecture, different output head"? Think about how masked language models become classifiers with a single linear layer swap.
The simplest way to teach a robot: show it what to do. A human demonstrates the task (teleoperation), recording observations and actions. Then we train a neural network to predict the expert's action given the current observation. This is behavioral cloning (BC) — supervised learning for robotics.
For a toy single-arm dataset, each episode contains observation/action pairs: an RGB image and [Δx, Δy, Δz, Δrx, Δry, Δrz, gripper]. A 10-second demonstration recorded at 10Hz gives 100 pairs. Behavioral cloning (BC) trains a policy to imitate demonstrations; deterministic regression can use MSE, token policies use categorical cross-entropy, and generative policies use other suitable objectives. Teleoperation is one way to collect that data.
When an imitation policy drifts, its next observation may be outside the demonstrated distribution. This distribution shift can compound errors. It is a failure mechanism, not proof that every small error inevitably causes failure; feedback and appropriate training coverage can support recovery.
The teal path is a constructed reference. The red path integrates injected random errors. This illustrates drift; it does not evaluate a trained BC policy.
Here is an independent no-error toy calculation: if each command has a 99% chance of being error-free, the probability of no errors in 100 commands is 0.99100 ≈ 36.6%, and in 500 it is about 0.657%. This is not a measured task-success rate: errors can be correlated, some are harmless and feedback can recover. The practical BC problem is distribution shift: an action changes the next observation, and a policy may drift into states its demonstrations did not cover.
python # Behavioral cloning training loop (simplified) for obs, action in demonstration_dataset: # obs: [3, 224, 224] camera image # action: [7] expert's 6-DoF + gripper command optimizer.zero_grad() predicted = policy(obs) # [7] predicted action loss = F.mse_loss(predicted, action) loss.backward() optimizer.step()
How do you represent "move the arm"? This choice is critical — it determines what the model predicts, how precise it can be, and whether it transfers across robots. There are two fundamental frames: joint space (the angles of each motor) and task space (the position and orientation of the end-effector in the world).
| Representation | Format | Dims | Pros | Cons |
|---|---|---|---|---|
| Joint angles (absolute) | [θ1,...,θ7] | 7 | Matches a joint-target controller | Depends on joint conventions and controller |
| End-effector pose (absolute) | [x, y, z, rx, ry, rz, grip] | 7 | Intuitive, more transferable | Needs compatible pose controller; IK may be required |
| Delta end-effector | [Δx, Δy, Δz, Δrx, Δry, Δrz, grip] | 7 | Small values, relative motion | Errors accumulate over time |
| Discrete bins (RT-2 style) | Token index per dim | 8 fields in original RT-2 | Works with discrete-token decoders | Loses precision between bins |
A seven-joint arm has seven joint angles; a Cartesian six-motion-plus-gripper command is a different interface. Delta actions specify changes rather than absolute targets. For an illustrative command, Δx=0.01m,Δy=−0.005m,Δz=0.02m. Absolute joint targets are also useful: ACT's original bimanual setup predicts them. The choice depends on data and controller conventions.
This widget uses equal-width bins reconstructed at their centers. Drag the slider to change the bin count; finer bins reduce the maximum in-range rounding error. The endpoint-level code lab below uses a distinct convention. The green dot shows the discretized position.
python # Toy endpoint quantizer (not an exact RT-2/OpenVLA tokenizer) action_range = [-1.0, 1.0] # normalized action space n_bins = 256 def discretize(a): # Continuous [-1, 1] → bin index [0, 255] return int(round(max(0.0, min(1.0, (a + 1) / 2)) * (n_bins - 1))) def undiscretize(bin_idx): # Bin index [0, 255] → continuous [-1, 1] return bin_idx / (n_bins - 1) * 2 - 1 # Example: dx=0.3 → level 166 → teaching symbol "166"
Our toy quantizer rounds to B=256 endpoint levels on [−0.5m, 0.5m]. Derive its spacing and maximum reconstruction error. Then compare a deterministic MSE prediction with a categorical distribution when equally likely target deltas are −0.2m and +0.2m.
Spacing = 1/255m = 3.922mm; maximum rounding error = 1/510m = 1.961mm. This excludes out-of-range inputs that are clipped. With 256 equal-width center-reconstructed bins, the bounds would instead use 1/256 and 1/512m.
For the two equally likely deltas, deterministic MSE minimizes 0.5(&hat;a+0.2)2 + 0.5(&hat;a−0.2)2 at &hat;a=0. A categorical policy can place mass on the two quantized modes and sample one. That is a representation advantage over a single conditional-mean output, not a guarantee of successful execution.
Autoregressive action tokens can condition later dimensions on earlier ones. A continuous vector head can also couple dimensions, and generative continuous models can represent multiple modes. Token generation reuses LM machinery but requires an action encoding, training and decoding rules; the best interface depends on task and controller precision.
python def tokenize_action(action, action_low, action_high, n_bins=256): tokens = [] for i in range(len(action)): # Normalize to [0, 1] norm = (action[i] - action_low[i]) / (action_high[i] - action_low[i]) # Clamp to [0, 1] (numerical range clipping) norm = max(0.0, min(1.0, norm)) # Map to bin index [0, n_bins-1] token = int(round(norm * (n_bins - 1))) tokens.append(token) return tokens def detokenize_action(tokens, action_low, action_high, n_bins=256): actions = [] for i in range(len(tokens)): # Bin index to [0, 1] norm = tokens[i] / (n_bins - 1) # Denormalize to robot-specific range val = norm * (action_high[i] - action_low[i]) + action_low[i] actions.append(val) return actions
A next-token LM predicts a categorical distribution over vocabulary IDs. Mapping action choices onto discrete symbols lets a token-based policy reuse that decoder and cross-entropy objective. An autoregressive sequence can model dependencies between action dimensions. A categorical distribution can contain several quantized modes; a deterministic MSE output gives one conditional mean. This does not make discretization universally preferable: continuous generative action heads also model multimodality, with different precision and compute tradeoffs.
RT-2 (Robotics Transformer 2) by Google DeepMind is the landmark paper that proved VLMs can directly control robots. The key insight: fine-tune a VLM so that instead of generating text, it generates action tokens.
RT-2 fine-tuned PaLI-X (5B/55B) and PaLM-E (12B). Its original action schema is [terminate, Δx, Δy, Δz, Δrx, Δry, Δrz, gripper]: eight fields, with continuous dimensions uniformly quantized into 256 bins. PaLI-X uses existing integer tokens; PaLM-E repurposes 256 infrequent vocabulary tokens. Robot-task decoding is constrained to valid action symbols. The stated schema includes termination even though the paper's displayed integer example omits a field.
Illustrative original RT-2 schema: discrete termination plus seven control fields. Continuous values map to bin indices, then model-specific vocabulary IDs. This is not a learned action prediction.
RT-2 co-fine-tunes on robot trajectories and the original vision-language data. This improved robot generalization compared with robot-only fine-tuning in its evaluations. Robot data contributes about 50% of the PaLI-X training mixture and 66% of the PaLM-E mixture. These are model-specific sampling choices, not a universal balance that guarantees preserved language ability.
Consider a teaching example with text-output token losses [0.2,0.4,0.6,0.8] and action-output losses [0.3,0.7]. Compute each sample's loss using a token sum, then a token mean. Does either calculation establish the ratio of gradient magnitudes?
Text sum=2.0 and action sum=1.0: a 2:1 ratio of scalar loss contributions. Text mean=2/4=0.5 and action mean=1/2=0.5: equal scalar contributions. Neither gives a gradient-norm ratio.
Input/instruction tokens may be excluded from the supervised output loss; this chooses what is predicted, not how gradient lengths are balanced. Sampling ratios and sum/mean reductions both influence optimization. State both when implementing a mixed-task objective; do not infer a universal ratio from response lengths.
RT-2's approximately 50% or 66% robot sampling fractions are reported empirical training choices. They are not a derivation that text and action gradients have equal magnitude. Original OpenVLA evaluates cross-entropy on action output tokens during robot training.
What if instead of predicting a single action, the robot could sample from a distribution over actions? Diffusion Policy applies the same denoising process used in image generation to robot action prediction.
Starting from random noise, the model iteratively refines an action trajectory. This naturally handles multimodal action distributions — when there are multiple valid ways to do something (reach from the left or right), the diffusion model can represent all of them.
A trajectory generator can start from noise of shape [H, action_dim]. Diffusion Policy learns conditional noise/score predictions and uses a diffusion scheduler to generate an action sequence; its prediction, execution and denoising horizons depend on the task. Published settings include H=16, executing 8 commands before replanning, with 100 evaluation sampling steps. This is distinct from the flow-matching Euler example below.
python # Toy noise-to-data flow inference; model/observation encoding not implemented here a = torch.randn(k, action_dim) # [50, 7] random noise for i in range(10): t = i / 10.0 # left endpoint: 0.0,...,0.9 v = model(a, t, obs, instruction) # predict velocity a = a + v * (1.0 / 10) # Euler step; timing depends on implementation # a is a generated [50, 7] toy action chunk; execution can still fail
This constructed illustration interpolates a noisy curve toward a known target. It runs no learned diffusion network or scheduler; animation progress is not a denoising-step count.
Both categorical and continuous generative policies can model multiple modes. Tokenization reuses a discrete LM decoder; diffusion and flow matching add continuous generation machinery. Their numerical outputs still have finite precision. RL fine-tuning optimizes a reward objective and can complement demonstration pretraining; it is a training choice, not a third action representation.
The same "discrete vs continuous" tradeoff appears in image generation (VQ-VAE tokens vs diffusion pixels). What changes when choosing a discrete decoder versus a continuous generator for actions?
A single-step policy chooses one next command. Action chunking predicts H future actions jointly, helping model temporal dependencies and potentially reducing planning overhead. ACT (Action Chunking with Transformers) uses this idea for imitation learning; chunking is also used by diffusion and flow policies. Neither single-step control nor chunking guarantees a particular smoothness.
The output has shape [H, action_dim]. In a toy 50Hz controller, H=50 represents one second of planned commands. The system may execute E≤H commands before re-observing: E=10 means 0.2 seconds between policy updates. Prediction horizon H, execution window E, policy-update rate and low-level servo rate are different quantities.
Why chunk instead of predicting one step at a time? Two big reasons:
Constructed comparison: the red path receives larger injected noise, while the green path interpolates target endpoints with smaller noise. Change H to inspect piecewise planning; this is not a measured policy comparison.
python # Toy aggregation of predictions for the SAME execution timestep predictions = np.stack([old_plan[t_old], new_plan[t_new]]) # oldest first weights = np.exp(-0.01 * np.arange(len(predictions))) weights = weights / weights.sum() a_blend = weights @ predictions robot.execute(a_blend) # units/interface defined by controller
The execution window E determines how long new visual evidence waits before the next policy update. At 50Hz, E=10 spans 0.2 seconds. In a separate configuration with H≥200, E=200 spans 4 seconds. H can be larger than E; receding-horizon control discards or revises unused predictions. Longer windows may amortize compute but reduce reactivity, and open-loop dynamics can still amplify error. Choose horizons by task dynamics and measured latency.
A toy task needs 100 commands at 50Hz. Each planning call predicts H=50 actions, and E=10 are executed before a new observation. How many planning calls are required, how long is each execution window, and how many predictions from each call are not executed?
100/10=10 planning calls. Each execution window lasts 10/50=0.2 seconds. Each call consumes 10 of its 50 predictions and leaves 40 unused; the task lasts 100/50=2 seconds under this schedule.
These counts do not prove an error reduction. Action errors can propagate through robot dynamics within a window; feedback can correct disturbances, while unfamiliar observations can challenge a learned policy. Compare error empirically or state a dynamical model before deriving bounds.
Choose the execution window E by the required feedback latency, measured inference time and controller dynamics; no universal chunk size follows from the task names. You can keep prediction horizon H fixed and vary E≤H, or use a suitable faster controller for a reactive phase. At 50Hz, E=1 spans 20ms and E=10 spans 200ms, before additional sensing/inference delays. Original ACT temporal ensembling queries every timestep and combines predictions for that timestep; it does not implicitly replan every 5–10 steps. Test the phase-specific schedule rather than assume larger chunks supply precision.
Original OpenVLA is a token-based VLA built on Prismatic: fused DINOv2 and SigLIP vision features, a projector and Llama 2-7B. It was trained on a curated 970k-trajectory manipulation mixture from Open X-Embodiment. Its single-step policy showed strong generalist manipulation results in the reported evaluations; it does not combine the diffusion and chunking mechanisms from the preceding chapters.
A 224×224 input is processed by two patch14 vision encoders. Their 16×16 grids give 256 spatial positions. DINOv2 and SigLIP features concatenate channel-wise, preserving 256 positions, then project to [256,4096]. The full instruction prompt produces L token IDs and [L,4096] embeddings. Llama 2 predicts one action's seven reused vocabulary IDs autoregressively; these are not raw bin indices. De-tokenization reconstructs normalized bin-center values, then dataset-specific quantile statistics restore controller units.
python # Original OpenVLA usage sketch: model and processor already loaded image = camera.capture_pil() prompt = "In: What action should the robot take to pick up the cup?\nOut:" inputs = processor(prompt, image, return_tensors="pt").to(device) action = model.predict_action(**inputs, unnorm_key="bridge_orig", do_sample=False) # [7] values in the selected dataset's controller units robot.execute(action) # Vision: two 256-position feature grids -> channel concat -> [256,4096] # Text: L integer IDs -> embedding lookup -> [L,4096] # Causal decoding -> 7 vocabulary IDs -> centers -> q01/q99 unnormalization
Trace the path from image + instruction to robot action. Each color represents a different processing stage.
| Strategy | How | Pro | Con |
|---|---|---|---|
| Full fine-tune | Update all VLM weights on robot data | Maximum expressivity | Expensive; possible forgetting |
| Frozen VLM + action head | Freeze VLM, add small MLP for actions | Cheap, preserves VLM | Limited — VLM features may not capture motor-relevant info |
| Mixture-of-Transformers | Separate expert params for vision/language vs action tokens | Separate capacity for control; preservation is not automatic | Complex architecture (π0's approach) |
Original OpenVLA adapts its vision-language backbone during robot training. The paper separately studies LoRA, low-rank parameter updates, for efficient downstream adaptation. Original π0 instead adds a roughly 300M-parameter action expert to a PaliGemma VLM. Its transformer has two weight sets: image/language tokens use the VLM expert, while state/noisy-action tokens use the action expert. They interact through attention. Separate weights do not imply the pretrained expert is frozen.
python # Original pi0 routing: two transformer experts, not three modality FFNs token_route = { "image": "vlm", "text": "vlm", "state": "action", "noisy_action": "action", } # Each expert has its own transformer weights. # Projected attention keys/queries/values allow their tokens to interact. # This routing ledger is not a complete transformer implementation.
A human can watch someone else cook and learn the recipe, even though their arms are different lengths. Can robots do the same? Cross-embodiment learning trains on data from many different robot types, hoping that high-level task knowledge transfers even when the hardware differs.
The original Open X-Embodiment (OXE) collection assembled over one million trajectories from 22 embodiments, including single arms, bimanual robots and quadrupeds. The RT-X manipulation models used a compatible robot-data mixture and showed positive transfer across evaluated robots. A collection's full diversity and a particular model's training interface are different scopes.
Different robots expose different action spaces. For compatible single-arm manipulation datasets, a shared interface can use six end-effector motion values plus a gripper value, followed by the robot's task-space controller or IK. Matching vector length is not enough: coordinate frames, units, control modes and normalization must agree. Quadruped, hand and bimanual control require richer or different interfaces; proprioception can also matter.
Schematic dataset collection: toggle embodiment categories. A trained model needs a compatible subset/interface; this does not show one seven-value policy controlling every category.
python # Toy per-dataset action-range normalization (after frame/controller alignment) def normalize_action(action, embodiment_config): # Different robots have different action ranges # Illustrative configured dx ranges: arm A [-0.05,0.05]m; arm B [-0.03,0.03]m low, high = embodiment_config['action_range'] return (action - low) / (high - low) * 2 - 1 # map to [-1, 1] # During training: normalize all actions to [-1, 1] # During inference: denormalize back to robot-specific range
Worked hypothetical design: These interface and timing assumptions are chosen for this exercise, not a published ALOHA/π0 configuration.
1. Action space: Jointly predict [H,16] joint/gripper targets. H=50 at 30Hz plans 1.667 seconds. Alternatively, two Cartesian six-motion-plus-gripper commands would have 14 values; changing interfaces requires suitable training targets and controllers. A generative continuous head can model coordinated alternatives, while autoregressive tokens can condition dimensions on an action prefix.
2. Latency: Choose E=6: six commands at 30Hz cover 200ms. Under the stated 50ms complete-inference assumption, computing a new chunk within this window leaves 150ms for the rest of the schedule. Include sensing, scheduling and communication before declaring a viable buffer. The 30Hz command rate remains distinct from the 5Hz visual policy-update rate.
3. Coordination: A joint action generator can let either arm's prediction depend on both arms' context and target sequence. Bimanual demonstrations teach “hold while turning.” This coupling permits correlations; it does not guarantee successful coordination or contact mechanics.
4. Visual compression: A trained resampler could reduce 784 patch positions to a chosen query count, trading compute against information loss. Evaluate whether it preserves lid/hand alignment. Separate expert weights do not themselves reduce the number of attention positions or its quadratic term.
Robot demonstrations require equipment, operators and useful task coverage. Under an illustrative ten-episodes-per-hour collection rate, 100k episodes cost 10,000 operator-hours; actual throughput depends on episode length and resets. Simulation can scale experiments, but usable demonstrations still require a task definition and a successful controller/policy. Visual differences, dynamics mismatch and sensor noise can all cause a reality gap.
Explore a schematic appearance randomizer. The slider sets variation strength; each refresh changes colors and object placement. No policy or real-world transfer score is computed.
| Technique | How | What it randomizes |
|---|---|---|
| Visual DR | Change appearance every episode | Textures, colors, lighting direction, camera position (±5cm), shadows |
| Physics DR | Change dynamics every episode | Friction (0.3-1.0), mass (±20%), motor delay (0-50ms), gravity (±2%) |
| System ID | Calibrate sim to match real | Measured real-world physics parameters |
| Sim + Real co-train | Mix sim and real data in training | Adds real observations/actions; remaining gaps require evaluation |
| Foundation Model encoder | Use VLM vision backbone | Provides transferable features; visual adaptation may still be needed |
A simulated cup may resemble a real cup while having different friction, geometry or camera calibration. Visual shifts can remain even with a pretrained encoder; OpenVLA found adapting vision features important for control. Domain randomization and system identification address selected mismatches, but neither guarantees coverage of the real deployment distribution.
A concrete experiment is to adapt a pretrained policy on task-specific real demonstrations, compare against a simulation-augmented variant, and evaluate both on held-out real scenes. Record the model, controller, data mix, hardware and success criterion. For context, OpenVLA reported downstream LoRA adaptation in 10–15 hours on one A100 for its experiments; this is not a universal under-one-day deployment recipe.
VLA results span simple manipulation and more complex demonstrations, such as π0 laundry folding and table clearing. Reliable transfer, contact control and recovery remain active challenges. Distinguish a published model's measured result from a proposed engineering budget.
Key open challenges:
| Challenge | Current State | What's Needed |
|---|---|---|
| Dexterity | Task-dependent manipulation and some dexterous demonstrations | In-hand manipulation, tool use |
| Long-horizon | Single-stage and some multistage demonstrations | Multi-step planning, error recovery |
| Safety | Lab environments | Real-world safety guarantees |
| Data scale | Dataset/version dependent | Internet-scale robot data |
| Speed | Model/hardware-dependent policy rate | Reactive control for dynamic tasks (100Hz+) |
Key milestones in the journey of robot policy architectures. Dates mark the cited initial publications; spacing is schematic.
You now understand the path from seeing to acting — the data flow from camera pixels to motor commands, the engineering choices of action representations and chunking, and the architectural decisions that connect pretrained models to compatible robot controllers. VLAs are teaching machines to reach out and touch the world.