pi-0.5: Open-World
Generalization

Chapter 1: The Key Insight

pi-0.5's breakthrough is simple to state: co-train on everything. Don't just train on your target robot's data. Train on data from other robots, web images, language instructions, high-level task plans, and more — all in one model.

Five data sources, one model

What changes from pi-0

pi-0.5 is not a fundamentally different architecture from pi-0. The key differences are:

The architecture is the same PaLiGemma backbone + action expert. What's new is the recipe — the data mixture and the two-stage inference that together enable open-world generalization.

Chapter 2: VLA Background

A Vision-Language-Action model takes an image and a language instruction as input, and outputs robot motor commands. It's a VLM (like GPT-4V) that has been fine-tuned to produce actions instead of (or alongside) text.

pi-0 recap: the data flow

pi-0 (the predecessor) was a 3B-parameter VLA built on PaLiGemma. Its key innovation was using flow matching as the action head — instead of discretizing actions into tokens (like RT-2), pi-0 generates continuous action trajectories by learning a velocity field that transports noise to actions.

pi-0.5 builds on pi-0 but adds the co-training recipe and hybrid inference that enable open-world generalization.

Action representations compared

The concrete action space for pi-0.5

pi-0.5's target platform is a mobile manipulator with:

• 7-DOF arm: 7 joint angles (shoulder, elbow, wrist rotations)
• Gripper: 1 dimension (open/close, continuous)
• Mobile base: 2 dimensions (x velocity, rotation velocity)
• Total action per timestep: 10 continuous values
• Action chunk size: 50 timesteps at 10 Hz = 5 seconds of motion
• Full action tensor: [50, 10] — 500 floating-point numbers per inference call

Note: pi-0.5 runs at 10 Hz (not 50 Hz like pi-0) because mobile manipulation is slower and the higher-level planning stage needs more time. Each 5-second chunk covers one atomic manipulation (reach, grasp, lift, place).

Why 10 Hz instead of 50 Hz?

pi-0 runs at 50 Hz because dexterous manipulation (folding fabric, assembling boxes) requires precise, high-frequency control. pi-0.5 runs at 10 Hz because its tasks are coarser:

• Mobile base: Moves at ~0.5 m/s. At 10 Hz, each command covers 5cm of motion — sufficient granularity for navigation.
• Arm reach-and-grasp: Typical reach takes ~3 seconds. At 10 Hz, that's 30 timesteps — enough to plan a smooth approach trajectory.
• Inference budget: At 10 Hz, the model has 100ms per action chunk (vs 20ms at 50 Hz). This extra time allows the more expensive two-stage inference pipeline.

The tradeoff is clear: pi-0.5 sacrifices the fine-grained dexterity of pi-0 in exchange for the two-stage reasoning that enables open-world generalization. You can clean a kitchen at 10 Hz; you can't fold laundry at 10 Hz.

Chapter 3: The Architecture

pi-0.5 is built on PaLiGemma, a vision-language model that processes images with a SigLIP vision encoder and generates text with a Gemma language model. The VLA extends this by adding action tokens to the output vocabulary.

Key design choices

• Vision encoder: SigLIP ViT-So400m (frozen during post-training), 400M params
• Language model: Gemma-based transformer backbone, ~2.4B params
• Action head: Flow matching (continuous) or FAST tokenizer (discrete, for pre-training)
• Context: Multiple camera views (2 wrist cameras + forward-facing = 3 x 256 tokens = 768 visual tokens)

The full token budget

Every inference call processes this token sequence through the Gemma transformer:

This fits comfortably in PaLiGemma's 1024-token context window, though it's tight. Longer instructions or more cameras would require context expansion.

The mobile manipulator platform

pi-0.5's target robot is fundamentally different from pi-0's tabletop arms. It's a mobile manipulator with:

• Mobile base: Omni-directional wheels. Can translate (x, y) and rotate (θ). Max speed: ~0.5 m/s.
• 7-DOF arm: Mounted on the base. Reaches ~1m from base center. Standard parallel-jaw gripper.
• 3 cameras: 1 forward-facing (navigation), 2 wrist-mounted (manipulation). All 640x480 RGB, no depth.
• Onboard compute: NVIDIA Jetson for image preprocessing. Streams to nearby A100 server via WiFi.

The mobile base adds 3 action dimensions (vx, vy, vθ) on top of the arm's 7+1 (joints + gripper). Total: 11 action dimensions per timestep. This is larger than pi-0's typical 7-8 dimensions, which is why the action chunk is [50, 11] instead of [50, 7].

The mobile base also creates a coordination challenge: the robot must simultaneously navigate (base) and manipulate (arm). The model must learn when to move the base (approach a new cabinet) vs. when to keep the base still and move the arm (pick up an object in reach). This base-arm coordination emerges naturally from training data — no explicit planning hierarchy.

What the VLM backbone provides vs. learns

From web pre-training (already knows):

• Object recognition: "that's a mug" — even novel mug designs
• Scene understanding: "this is a kitchen with a sink on the left"
• Language grounding: "put it in the drawer" vs "put it on the counter"

From robot co-training (must learn):

• Proprioceptive-visual alignment: where is my arm relative to what I see?
• Action semantics: what motor commands correspond to "pick up"?
• Subtask decomposition: "clean the kitchen" → sequence of atomic actions

Chapter 4: Hybrid Inference

Here's the clever part. At inference time, pi-0.5 runs in two stages using the same model:

Stage 1: The "Understanding Expert" (slow, ~every 5-10 seconds)

Given the current image and high-level task ("clean the kitchen"), the model autoregressively generates a subtask in natural language ("pick up the plate and put it in the sink").

Timing: This runs every 5-10 seconds — once per atomic manipulation. The model processes all 768 visual tokens + language instruction + history, then generates ~10-20 text tokens describing the next subtask. Latency: ~200-500ms for the autoregressive generation.

Why autoregressive here: Subtask prediction IS a language task. The model leverages its VLM pre-training to reason about what to do next. "I see dirty plates on the counter and an empty sink → the next subtask is to move plates to the sink." This is pure VLM reasoning, no action generation needed.

Stage 2: The "Action Expert" (fast, ~10 Hz)

Given the image + subtask from Stage 1, predict continuous low-level actions via flow matching. This is the same action generation as pi-0: noise → 10 denoising steps → clean action chunk [50, 10].

Timing: Runs at 10 Hz. Each call generates a 5-second action chunk, but only the first 0.5-1 second is executed before re-planning with fresh observations. This gives closed-loop behavior despite the chunk-based architecture.

What if Stage 1 predicts the wrong subtask? The action expert will execute it faithfully — leading to incorrect behavior. But because Stage 1 re-runs every 5-10 seconds, the model self-corrects: it sees the new scene state, recognizes the situation, and predicts a better subtask. This is analogous to a human realizing "wait, I should've grabbed the sponge first" and changing plans.

The inference pipeline on hardware

Total first-action latency: ~50ms (when subtask is already predicted). The robot carries an onboard NVIDIA Jetson for SigLIP encoding and streams tokens to a nearby A100 server for transformer inference.

A complete kitchen cleaning trace

Here's an actual execution trace from the paper's kitchen experiment (simplified):

Total: 7 subtasks over ~3.5 minutes (partial trace of a 10-15 minute episode). Note the recovery at 2:00 — the model re-observed the scene, detected the failed grasp, and autonomously re-attempted with a different approach angle. This recovery capability comes from pre-training on diverse data that includes many partial failures and corrections.

Chapter 5: The Co-Training Recipe

The magic of pi-0.5 is in the data mixture. Five sources of supervision, each contributing something unique:

How heterogeneous data is unified

These five sources have completely different formats. How do you train one model on all of them? The key: different loss functions applied to different token positions, all in the same forward pass.

• MM/ME/CE data: Flow matching loss on action tokens + language modeling loss on text tokens.
• HL data: Language modeling loss only (predict the next subtask as text). No action tokens.
• Web data: Standard VLM loss (image captioning, VQA). No action tokens, no robot context.

The model sees all data types interleaved in each batch. A single batch might contain: 25% MM episodes, 15% ME, 20% CE, 15% HL, 25% web. The loss computation only applies to the relevant token positions for each example.

Why each source matters — a worked example

Consider the robot encountering a new kitchen with a never-seen coffee mug on the counter:

1. Web data taught: "That cylindrical object with a handle is a mug. Mugs go in sinks or cabinets."
2. MM data taught: "To pick up objects from a counter at this height, approach from above with this grasp strategy."
3. ME data taught: "Countertops have this visual texture. Objects rest stably on flat surfaces."
4. CE data taught: "Grasping cylindrical objects requires wrapping the fingers — this works regardless of robot type."
5. HL data taught: "After picking up a dirty mug, the next step is to place it in the sink."

No single source provides all this knowledge. Together, they let the robot successfully handle a novel mug in a novel kitchen.

Chapter 6: Two-Stage Training

Training happens in two phases, each with a different action representation:

Why two stages with different action representations?

Pre-training with discrete tokens lets the model learn from ALL data sources in a unified format — robot data, web data, language, subtask predictions — all as token sequences. This is where the broad knowledge comes from.

Post-training with flow matching replaces the discrete action head with a continuous one. Flow matching produces smoother, more precise motor commands than discretized tokens. This is where fine-grained control comes from.

The FAST tokenizer: bridging discrete and continuous

FAST (Fine-grained Action Sequence Tokenization) is a key engineering innovation that makes the two-stage recipe possible. It uses DCT (Discrete Cosine Transform) to compress action chunks:

1. Take an action chunk: [50, 10] = 500 continuous values.
2. Apply DCT per dimension: Transform to frequency domain. Low frequencies capture the trajectory shape; high frequencies capture noise.
3. Truncate high frequencies: Keep only the top-K DCT coefficients (K=8-16). This reduces 500 values to ~80-160 values.
4. Quantize: Map each coefficient to one of 1024 discrete tokens.
5. Result: A 50-timestep, 10-dimensional action chunk is represented as ~80-160 tokens from a 1024-token vocabulary.

This is dramatically more efficient than RT-2's approach (which would need 50 x 10 x 1 = 500 tokens for the same chunk). FAST makes it possible to pre-train on actions using the same token-prediction machinery as language.

Reconstruction quality of FAST

How much information is lost in the DCT truncation? For typical manipulation trajectories:

• K=8 coefficients: Reconstructs trajectories with <2% RMSE. Smooth motions captured perfectly. Fine jitter lost — but jitter is harmful anyway.
• K=4 coefficients: ~5% RMSE. Adequate for mobile base navigation but loses dexterous precision.
• K=16 coefficients: Near-perfect (<0.5% RMSE) but 160 tokens per chunk — too many for the context window.

K=8 is the sweet spot for pre-training. The model learns general trajectory shapes while leaving room for language and visual tokens. Post-training with flow matching then recovers full continuous precision that FAST's quantization loses.

Training infrastructure

What the VLM forgets without web co-training

An ablation study showed that removing web data during pre-training causes a 15-20% drop in novel object recognition. The model can still manipulate objects it saw during robot training, but fails to generalize to new objects — exactly because it has forgotten the visual recognition capabilities from web pre-training. The co-training recipe prevents this forgetting.

With vs without web co-training: qualitative examples

The web data doesn't teach the robot to move — it teaches it to see. For open-world generalization, seeing correctly is half the battle.

Chapter 7: Experiments

The headline result: pi-0.5 can clean kitchens and bedrooms in entirely new homes not seen during training. These are 10-15 minute tasks involving multiple stages.

What the robot does in a new kitchen

Given "clean the kitchen," pi-0.5 autonomously:

1. Navigates to the kitchen area (mobile base)
2. Surveys the scene (rotates, captures images from multiple angles)
3. Predicts first subtask: "close the open cabinet door"
4. Executes: approaches cabinet, grasps door edge, pushes closed
5. Re-observes: "good, cabinet is closed. Next: pick up the mug on the counter"
6. Executes: approaches mug, grasps, lifts, navigates to sink, places
7. Continues through 8-12 subtasks over 10-15 minutes

Each step is predicted by the model itself (high-level subtask), then executed with flow matching actions. The full sequence takes 10-15 minutes.

What degrades in new environments

Even pi-0.5 has limits. The degradation patterns reveal what the model has truly learned vs. what it has memorized:

The mobile base challenge

Mobile manipulation adds a dimension of difficulty absent in tabletop robotics: the robot must decide when and where to move its base. This decision is implicit in the action space — the model outputs base velocities alongside arm joint velocities. But the consequences are very different:

• Moving the base changes the entire visual scene. All spatial relationships between the arm and objects change. The VLM must re-parse the scene after every base movement.
• Base odometry drifts. Wheel slip on different floor surfaces (tile vs carpet) causes position errors that accumulate. After 10 minutes of movement, the robot may be 5-10cm from where it thinks it is. The vision system must handle this gracefully.
• Navigation near obstacles. The robot must avoid furniture, walls, and people while navigating to the next manipulation target. There is no explicit obstacle map — the model learns collision avoidance from training data where teleoperators navigated around obstacles.

These challenges explain why pi-0.5 runs at 10 Hz (not 50 Hz) and uses 5-second action chunks (not 1-second): mobile tasks require more planning horizon per chunk, and the base can't respond to new observations as quickly as a stationary arm.

The 10-minute failure modes

Over 10-15 minute tasks, errors compound. The most common failure patterns:

• Subtask prediction loops: The model predicts "pick up the plate" → fails → sees the same scene → predicts "pick up the plate" again. No fallback strategy.
• Drift accumulation: Small positioning errors from the mobile base compound. After 10 subtasks, the robot may be 5-10cm off from where it thinks it is.
• Recovery from drops: If an object is dropped, the model must re-observe, recognize the new state, and re-plan. This works ~60% of the time.

Chapter 8: Scaling

How does generalization scale with the number of training environments? The paper shows a clear trend: more diverse training scenes = better generalization to new scenes.

The scaling curves with actual numbers

The paper tests performance as the number of unique training environments increases:

The scaling is logarithmic — each doubling of environments adds less marginal improvement. But crucially, it hasn't plateaued at 50 environments. More environments would likely continue improving performance, just at a slower rate.

But the scaling isn't just about quantity — it's about diversity. Adding more data from the same kitchen helps less than adding data from a different kitchen. And adding data from a completely different robot or from the web helps in ways that more same-robot data cannot.

Data collection cost

A practical concern: how expensive is this data to collect?

The most expensive component is the human annotation (HL labels), not the robot data collection. This suggests that auto-labeling (using VLMs to generate subtask labels from video) could dramatically reduce the cost of scaling. pi-0.7 explores this direction with automated subtask annotation.

Compute scaling

The relationship between compute and performance:

• Pre-training steps: Performance improves log-linearly from 50K to 280K steps. Returns diminish after 200K.
• Model size: Going from 1B to 3B parameters gives meaningful gains. The paper doesn't test larger models (GPU memory constraints at inference).
• Batch size: Larger batches (more diverse examples per step) help more than equivalent extra steps with smaller batches. This confirms that exposure to diversity is the key factor.

Chapter 9: Ablations

The paper systematically removes each data source to measure its contribution. The finding: every source matters.

The ablation that surprised the authors

The most surprising ablation result: removing verbal instructions (just providing images, no language) only drops performance by 10 pp. This suggests that pi-0.5 has learned to infer task intent from visual context alone — seeing a messy kitchen is enough to trigger cleaning behavior without being told "clean the kitchen." The VLM's web pre-training includes millions of before/after images of clean vs messy spaces, implicitly teaching the concept of "this needs cleaning."

However, language becomes critical when the task is ambiguous ("put the mug in the cabinet" vs "put the mug in the sink" — visually, both cabinets and sinks are present). Without language, the model defaults to the statistically most common action for each object, which is often but not always correct.

The interaction effects

More interesting than individual ablations are the interaction effects. Removing two sources simultaneously hurts more than the sum of removing each individually:

• Remove CE + Web together: -40 pp (vs -12 + -20 = -32 pp individually). They're synergistic — web data helps the model understand what CE robots are doing.
• Remove HL + ME together: -38 pp (vs -15 + -18 = -33 pp individually). Planning ability requires scene diversity to generalize.

This superlinear degradation proves the data sources don't just add independent knowledge — they amplify each other. The whole is genuinely greater than the sum of parts.

What about depth cameras?

An interesting negative result: adding depth cameras (RGBD instead of RGB) did NOT significantly help. The model learns depth understanding implicitly from stereo cues across its three cameras. This is important because depth cameras are fragile, expensive, and fail on transparent/reflective surfaces. pi-0.5 works with cheap RGB cameras only.

The FAST tokenizer ablation

The paper also ablates the two-stage training approach (FAST discrete → flow matching) vs single-stage alternatives:

The two-stage approach wins because it gets the best of both worlds: FAST enables co-training on heterogeneous data (you need tokens to train alongside language), and flow matching provides the continuous precision that discrete tokens can't match.

Chapter 10: Connections

pi-0.5 represents a major step in the VLA lineage:

What pi-0.5 means for deployment economics

pi-0.5 changes the economics of robot deployment. Before pi-0.5, deploying a robot in a new home required:

1. Send robot to new home
2. Collect 50-100 hours of demonstration data in that specific home
3. Fine-tune the model on that data (~2 days of compute)
4. Deploy the fine-tuned model

Cost per home: ~$5,000-10,000 in human labor + compute. Time to deploy: ~2 weeks.

With pi-0.5's open-world generalization:

1. Send robot to new home
2. Deploy the pre-trained model immediately (zero-shot)

Cost per home: ~$0 marginal (the training cost is amortized across all deployments). Time to deploy: immediate. This is the difference between a research prototype and a viable product.

Remaining deployment barriers

Even with zero-shot generalization, real-world deployment faces non-ML challenges:

• WiFi latency: The robot streams observations to an A100 server. WiFi jitter (10-50ms spikes) causes action delays. In-home WiFi is less reliable than lab networks.
• Battery life: The mobile base runs for ~2-3 hours. A full kitchen cleaning takes 15 minutes. Multiple cleanings per charge are possible but battery monitoring is needed.
• Safety certification: No robot with a learned policy has been safety-certified for unsupervised home use. The model can produce unexpected motions (high-velocity arm swings during error recovery). Mechanical speed limits and collision detection are external safety layers.
• User interaction: How does the homeowner specify tasks? A chat interface? Voice commands? The paper uses text input, but natural language understanding for ambiguous household instructions remains an open problem.

Key engineering lessons from pi-0.5

The open-world gap: what remains unsolved

Despite pi-0.5's achievements, the gap between lab success and true household deployment remains large:

• Safety: The model has no concept of "this action could break something expensive" or "this action could hurt a person." No safety constraints are enforced beyond joint limits.
• Long-tail objects: Rare objects (unusual kitchen gadgets, foreign utensils) still cause failures. The web data provides recognition but not manipulation strategies.
• Multi-room navigation: Moving between rooms requires spatial memory that the current architecture doesn't support. The model can clean one room but struggles with "bring the plate from the kitchen to the dining room."
• Human interaction: If a person is in the kitchen, the robot has no model of personal space, social norms, or how to coordinate (e.g., "excuse me, I need to reach that cabinet").

What's next?

pi-0.5 shows that co-training on diverse data sources is the path to generalization. The open questions are: how far can this scale? Can we add simulation data, human video, and internet-scale interaction data to push generalization even further? pi-0.7 answers part of this — adding structured prompt conditioning to absorb even more diverse data without mode averaging.