TiPToP: A Planner That
Just Works on Pixels

Chapter 1: The Key Insight

The insight is almost embarrassingly simple, which is exactly why it's powerful: foundation models have quietly made every hard sub-problem of classical TAMP cheap — so stop training a giant policy, and instead glue the best off-the-shelf model into each slot of a planner.

The acronym is the architecture

TiPToP = TiPToP is a Planner That just works on Pixels. The emphasis on "planner" is the whole thesis: instead of a learned policy mapping pixels→actions, there is an explicit planner in the middle that reasons.

Why "frozen and modular" is the bet

Every component is pretrained and frozen — TiPToP never updates a single weight. This has three consequences that drive the entire paper:

Contrast a VLA: improving it means collecting more data and retraining the whole blob. A modular system improves the way software improves — one library upgrade at a time.

The five frozen experts

Chapter 2: TAMP vs VLA Background

Before we trace TiPToP's pipeline, we need to genuinely understand the two worldviews it bridges — because the experiments later show they fail in complementary ways, and that complementarity is the paper's deepest finding.

The VLA worldview: react, don't plan

A VLA like π_0.5-DROID is a closed-loop reactive policy. At every timestep t it observes images o_t and joint state q_t, and emits an action: a_t = π(o_t, q_t | L). It runs at 15 Hz, outputting chunks of 15 actions: joint velocities q̇ plus a binary gripper g. Crucially, it keeps looking — if a grasp slips, the next frame sees the slip and corrects. Reactivity is its superpower.

Its weakness: it must implicitly discover multi-step structure ("move the can, then grab the cracker") from raw language and pixels, with no explicit notion of geometry or sequencing.

The TAMP worldview: plan once, then commit

TiPToP observes the scene once, at t=0, from a calibrated wrist camera. From that single stereo snapshot o₀ = (I₀^left, I₀^right) it produces a complete timed trajectory a₀ = {(q_t, q̇_t, g_t)} for the entire task — then executes it open-loop, with no further looking. All the reasoning is up-front.

What "jointly reasoning over discrete and continuous" means

This phrase is the heart of TAMP. The discrete part is the symbolic plan skeleton — which actions in what order: [MoveFree, Pick(cracker), MoveHolding, Place(cracker, tray)]. The continuous part is the geometry — exactly which grasp pose, exactly where to place, along what arm trajectory. The two are coupled: a discrete plan might be infeasible because no continuous grasp exists (the can blocks everything), forcing a different discrete plan that moves the can first.

The interactive contrast

Chapter 3: The Full Pipeline

TiPToP is three modules in series. Each consumes a typed input and emits a typed output — and tracing those types is understanding the system. Let's follow one stereo image pair all the way to joint torques.

Data flow with shapes

The end-to-end type signature

Here is the contract, written as data with shapes — read this once and the whole paper unlocks:

The two parallel branches inside Perception

The perception module forks into two branches that run simultaneously and only merge at the end:

• 3D Vision Branch (geometry): stereo → depth → point cloud → grasps. Answers "where are surfaces and how do I grip them?"
• Semantic Branch (meaning): image + language → VLM → object labels, boxes, symbolic goal → SAM-2 masks. Answers "what are these things and what does the human want?"

They are deliberately decoupled: geometry doesn't care what an object is, and semantics doesn't care where it is in 3D. The merge step (Chapter 4) stitches "this mask" to "this chunk of point cloud" to "these grasps."

Chapter 4: The 3D Vision Branch

This branch turns flat pixels into graspable 3D geometry. Three steps: estimate depth, unproject to a world-frame point cloud, and predict grasps. Then the merge: convex-hull meshing and grasp-to-object assignment.

Step 1 — Depth from stereo

FoundationStereo takes the stereo pair and predicts a dense depth map D, aligned to the left image. Why a foundation model instead of the ZED camera's built-in stereo matching? Because the camera's matcher falls apart on the surfaces robots care about most: transparent, specular, and textureless objects (that reflective soda can!). FoundationStereo gives sharp object boundaries where it counts.

Step 2 — Unproject to world frame

A depth map is per-pixel distance. To plan motion you need 3D points in the robot's world frame. Two transforms chained together do it:

Every symbol, in plain terms:

• p_cam — a 3D point in the camera's coordinate frame, recovered from a pixel + its depth using the camera intrinsics K (the lens geometry).
• T_cam^ee — the fixed transform from camera to end-effector. The camera is bolted to the wrist, so this never changes; it's measured once during calibration.
• T_ee^world = FK(q₀) — forward kinematics: given the joint angles q₀ at capture time, where is the wrist in the world? Pure geometry from the robot's known link lengths.
• p_world — the same physical point, now expressed in the world frame the planner reasons in.

Step 3 — Grasps from M2T2

M2T2 looks at the full scene point cloud and predicts ranked 6-DoF grasp poses (position + orientation of the gripper). "Full scene" matters: because M2T2 sees surrounding geometry, its grasps are informed by clutter — though they are not guaranteed collision-free (that's the planner's job later). Some objects get no grasp; for those, the planner falls back to a heuristic top-down sampler.

The merge: convex-hull meshes + grasp assignment

Now the two branches meet. Using each object's SAM-2 mask (from Chapter 5), the corresponding points are pulled from the cloud, projected down to the object's lowest observed point, and a convex hull is computed — a watertight mesh. Each M2T2 grasp is assigned to its nearest object via a KD-tree lookup; grasps too far from any object point are discarded as noise.

Chapter 5: The Semantic Branch & Symbolic Goals

Geometry alone can't tell "peanut butter crackers" from "Cheez-Its." That requires a model that has read the internet. The semantic branch is one VLM call that does two jobs at once, plus a segmentation step.

One VLM query, two outputs

Gemini Robotics-ER 1.5 is queried once with the image and the instruction L. It jointly returns: (1) labels + 2D bounding boxes for every object, and (2) a symbolic goal G — a logical conjunction of predicates over those objects. The VLM's web-scale common sense is what grounds "peanut butter crackers" to the two Lance packages, and reasons that "each tray" means one package per tray.

What a symbolic goal actually is

TiPToP currently supports one predicate: On(a, b) — "object a should end up on object/surface b." The instruction "serve peanut butter crackers on each tray" compiles to:

Why this crushes VLAs on semantic tasks

A VLA has no explicit grounding step. To handle "pick up the largest toy," it must learn that relation implicitly from demonstrations. TiPToP's VLM just reasons about it and emits On(largest_toy, plate). In the experiments, π_0.5-DROID scored 0/5 on four separate semantic scenes; TiPToP won seven of eight. The explicit symbolic layer is the difference.

Segmentation: from box to mask

The VLM gives a coarse bounding box. To extract clean per-object geometry you need a pixel-perfect mask. So each box is fed to SAM-2, which returns a tight segmentation mask from the left image. These masks are what Chapter 4's merge uses to carve the point cloud into per-object pieces.

Chapter 6: cuTAMP — Planning by Particle Optimization ⭐

This is the engine room. We have a symbolic goal G and a scene of meshes and grasps. cuTAMP must find a sequence of actions and the continuous parameters (which grasp, where to place, what trajectory) that satisfies G without collisions. It does this with a beautiful trick: thousands of guesses, optimized in parallel on a GPU.

Step 1 — Enumerate plan skeletons

A PDDL-style symbolic planner lists candidate skeletons: action sequences with unbound continuous parameters (written with ?):

Critically, the planner generates multiple skeletons. Short ones grab the cracker directly. Longer ones first move the obstructing soda can. The planner doesn't know yet which is feasible — that's decided by optimization.

Step 2 — Particles: many guesses at once

For each skeleton, cuTAMP samples a big batch of particles — each particle is one complete guess for all the unbound parameters: a specific grasp pose (from M2T2 or the heuristic sampler), a placement pose on the tray, and robot configurations via inverse kinematics. Most initial particles are infeasible — they collide, or the placement is unstable, or no IK solution exists.

Step 3 — Differentiable optimization

Here's the magic. Each constraint becomes a differentiable cost: collision penetration depth, placement instability, kinematic infeasibility. cuTAMP runs gradient descent on all particles simultaneously on the GPU, nudging each guess toward satisfying every constraint. The total cost for a particle is roughly:

where each C is ≥0 and equals 0 only when that constraint is satisfied. A particle is feasible when C→0. Skeletons are ranked by how easily their particles reach C=0; the first to get enough feasible particles wins. If the "grab cracker directly" skeleton can't drive C to zero (the can always collides), cuTAMP abandons it for the "move can first" skeleton.

Step 4 — Motion planning fills the trajectories

For each feasible particle, cuTAMP calls cuRobo (a GPU motion planner) to solve the remaining trajectory parameters ?τ — collision-free, time-parameterized paths between configurations. Output: the final {(qt, q̇t, gt)}.

The SHOWCASE: watch particles converge

Below is cuTAMP's core loop made interactive. Each dot is a particle — a candidate placement pose for an object. Red = high cost (colliding or unstable), green = feasible (C→0). Press Optimize to run gradient descent; particles flow toward the valid region (the tray, away from the obstacle). Add the obstacle to see particles forced to a different valid zone. This is exactly how "move the can first" emerges from "grab directly" failing.

The cost function in code

# cuTAMP, in spirit: optimize a BATCH of particles in parallel
# Each particle = one candidate (grasp, placement, config).
import torch

def total_cost(place_xy, obstacle_xy, tray_center, tray_r):
    # place_xy: [N, 2] batch of candidate placement poses
    # 1) stay inside the tray (stability/goal)
    d_tray = (place_xy - tray_center).norm(dim=1)
    c_goal = torch.relu(d_tray - tray_r) ** 2
    # 2) avoid the obstacle (collision penetration)
    d_obs  = (place_xy - obstacle_xy).norm(dim=1)
    c_col  = torch.relu(0.12 - d_obs) ** 2   # penalize if too close
    return 1.0*c_goal + 3.0*c_col          # weighted sum, all ≥0

place = torch.rand(64, 2, requires_grad=True)  # 64 particles
opt = torch.optim.Adam([place], lr=0.02)
for step in range(100):
    cost = total_cost(place, obs, tray_c, tray_r).sum()
    opt.zero_grad(); cost.backward(); opt.step()  # all particles, one step
feasible = total_cost(place, obs, tray_c, tray_r) < 1e-4  # C → 0

Chapter 7: The Execution Module

The planner produced a perfect trajectory. But a plan is only as good as the controller that tracks it. TiPToP's planner assumes the robot's joints follow the plan exactly — and even sub-centimeter tracking error makes a grasp miss or a placement topple.

Why off-the-shelf controllers weren't enough

The authors found existing open-source controllers (including DROID's default Polymetis) couldn't track timed trajectories precisely enough. So they wrote their own joint impedance controller for the Franka arms. At every control timestep it computes joint torques:

Decoding every term — this is a PD controller plus physics feed-forward:

• K_p ⊙ (q_d − q) — the position spring. q_d is desired joint angle, q is actual; the gap times stiffness K_p pulls the joint toward target. (⊙ = per-joint elementwise.)
• K_d ⊙ (q̇_d − q̇) — the velocity damper. Penalizes velocity error so the joint doesn't overshoot and oscillate.
• τ_g — gravity compensation. Without it, the arm sags under its own weight; this cancels gravity so the PD terms only fight tracking error, not physics.
• τ_coriolis — cancels Coriolis forces that arise when multiple joints move together.
• M·q̈_d — inertia feed-forward. M is the mass matrix; q̈_d the desired acceleration. This pre-commands the torque needed to accelerate the arm before error builds up.

Open-loop, by design

Once execution starts, TiPToP does not look again. No re-perception, no replanning. This works when the world is static and tracking is tight. It fails when an object moves unexpectedly or a grasp slips — there is no mechanism to notice and retry. This single design choice is the root of the dominant failure mode (Chapter 9).

Chapter 8: The Experiments

The claim is bold — zero-data TiPToP matches a 350-hour VLA — so the evidence has to be careful. The authors ran 165 trials across 28 scenes, in simulation, on their own DROID hardware, and (crucially) on an external team's DROID setup they'd never touched.

Four task categories

• Simple: one-step pick-and-place, no distractors.
• Distractor: the right object hides among irrelevant ones.
• Semantic: complex referring expressions ("largest toy," "matching plate," "sort by color").
• Multi-step: several actions in sequence, often with physical reasoning (move an obstacle, constrained packing).

The headline numbers

Toggle the categories below and watch the gap open as tasks get harder. On simple tasks they're roughly tied. On everything requiring reasoning, TiPToP pulls ahead — and on semantic tasks π_0.5-DROID often scores zero.

Speed, too

TiPToP isn't just more accurate — it's usually faster. It plans one time-optimal trajectory and executes it; the VLA runs a reactive loop that idles and re-attempts. On can → mug: 18.6 s vs 41.0 s. On single-step real tasks, TiPToP finishes in ~15 s, roughly half the VLA's time. Of that, planning (perception + cuTAMP) is ~7–20 s.

Cross-embodiment: the modularity payoff

Because nothing is trained, moving to a new robot is config, not data. Deploying on a UR5e took "a few hours": provide the URDF, generate collision spheres, write a cuRobo config, implement camera/controller interfaces. It also ran on a Trossen WidowX AI. And extending to a new skill (whiteboard wiping) took under a day — two new predicates, one new TAMP operator, a small wiping controller — touching zero existing perception/execution code.

Chapter 9: Failure Analysis — Modularity's Real Payoff

Here's where modularity earns its keep. The authors ran 173 extra trials and traced every failure to a specific module. A VLA can't do this — failures are a single opaque blob. TiPToP turns "the robot failed" into "module X failed," which turns into "swap a better X."

Where 55 failures came from

Hover the bars: grasping dominates, and it traces straight back to the open-loop design choice from Chapter 7.

The four failure modes, decoded

The complementary failure finding

The single biggest limitation

Open-loop execution. Over half of all failures are grasps that a simple retry would fix — but the system commits to one trajectory and never looks again. The most direct improvement: re-run perception and planning after each pick-and-place step. The honest framing here is itself a lesson: a clean modular architecture lets you name your single biggest weakness with a number.

Chapter 10: Connections & Cheat Sheet

TiPToP sits at the seam of two research traditions. Its real contribution is less a new algorithm than a demonstration: foundation models have made modular, training-free manipulation competitive with end-to-end learning. That reframes the field.

Where it connects

• vs π₀/π_0.5 (VLAs): the explicit foil. Same I/O (pixels+language → actions), opposite philosophy (learn everything vs train nothing). Complementary failures.
• SceneComplete: the perceptual ancestor — composing foundation models to build 3D scenes from 2D. TiPToP adds grasp generation + TAMP on top.
• cuTAMP / cuRobo: the GPU-accelerated planner and motion planner that make sub-20-second planning possible.
• Foundation models slotted in: FoundationStereo (depth), M2T2 (grasps), Gemini Robotics-ER (grounding), SAM-2 (segmentation). Each is independently swappable.
• Future: belief-space planning — reasoning about pose/grasp uncertainty, and info-gathering actions (move the camera before planning).

The cheat sheet

The one idea to keep

Try it yourself

Re-run the cuTAMP showcase (Chapter 6) with the obstacle toggled on and off. Notice: with the obstacle, some particles can never reach the tray without colliding — exactly the moment cuTAMP gives up on the direct skeleton and reaches for the "move the obstacle first" plan. You've just watched task-and-motion planning happen.