Lift, Then Sculpt

Chapter 1: The Key Insight

Chapter 0 left you with a widget that never quite reaches 100% coverage, no matter how it's dragged. This chapter is about the design decision that follows from taking that limit seriously instead of trying to engineer around it.

The Tempting Wrong Idea

Here's the tempting idea: train one enormous network that watches a video and outputs a complete, moving 4D object in a single pass. It would be elegant. It would also need a training set of paired (video, ground-truth 4D asset) examples at a scale that simply does not exist — nobody has millions of labeled 4D objects lying around, and synthetic substitutes generalize poorly to a real rhino on a real desk.

Feedforward 4D predictors built this way stay locked to whichever narrow category or synthetic-asset distribution they were trained on, and fall apart the moment "in the wild" means something they never saw. That's exactly the trap Chapter 0's "why hard" list described — more data of the same kind doesn't fix an ambiguity that's baked into a single camera's worth of observation, and a bigger feedforward network trained on that same kind of data inherits the same ceiling.

The paper's related-work section names two separate failure families, and it's worth keeping them distinct instead of lumping "prior methods" into one pile:

• Feedforward 4D predictors (read a video, guess the 4D shape in one pass) — bottlenecked by scarce 4D training data, generalize poorly in the wild.
• Optimization methods that lean on a video-diffusion prior — degrade badly once the camera's viewpoint drifts far from where the clip started.
• Optimization methods that use an image-to-3D prior only for one-time initialization — suffer a domain gap between "a static prior guessed once" and "a dynamic sequence that keeps changing underneath it."

Lift4D's actual move is close to the opposite of training something new. It reaches for three models that already exist, that were never trained on anything resembling "video in, 4D asset out," and it never touches a single one of their weights. What it builds instead is a two-phase optimization loop whose entire job is deciding, phase by phase, which of those frozen models is allowed to speak.

Three Frozen Models, One Loop

Before the two phases below, a third frozen model already did its job upstream, in Chapter 0's setup: SAM3 turned your text prompt into a per-frame mask. Nothing in this chapter's two lanes touches SAM3 again — its output just keeps flowing downstream as an input, the same way it will for the rest of the pipeline.

Every box above carries a small tag: frozen prior for a model that never updates, optimized this phase for the handful of parameters that do. That labeling convention isn't a one-off for this chapter — Chapter 3's pipeline diagram and Chapter 8's appearance-phase diagram reuse the exact same two tags, so at a glance you can always tell which boxes in this lesson are "borrowed, unchanged" and which ones are "what the optimizer is actually touching right now."

Notice the shape of that split. Phase A never asks the appearance prior anything. Phase B never lets the geometry move. Each frozen model gets exactly one job, in exactly one phase, and nothing else touches its output in between. That routing discipline — not a bigger network — is the paper's actual bet.

It's also, specifically, a fix for a failure mode that already showed up in prior work. Some earlier optimization methods use an image-to-3D prior only once, for a one-time initialization, and then hand the whole rest of the job — fixing geometry and fixing appearance and reconciling both against the input video — to a single undifferentiated optimization loop. When geometry and appearance signals get to fight inside the same loss at the same time, an appearance-driven gradient can quietly drag geometry off the shape the video actually supports, and vice versa. Splitting into two phases with a hard freeze between them isn't a minor implementation detail; it's the thing that keeps that fight from happening at all.

Concretely, this isn't two loss terms blended together and hoped into balance — it's two literally sequential blocks of gradient steps. For the first 10,000 of 20,000 total iterations, only Phase A's loss is active; every iteration after that switches over to Phase B's loss instead. There's no single iteration where both phases are simultaneously trying to update the same parameters, which is exactly what makes the "frozen prior / optimized this phase" tags above a description of the actual optimizer schedule, not just a tidy metaphor.

You could, in principle, run the phases in the opposite order and ask why that fails: fit appearance first, geometry second. It fails because appearance has nothing stable to render onto yet — without a settled canonical shape, "paint the belly" doesn't even have a well-defined surface to paint. Order isn't an arbitrary implementation choice here; geometry has to come first because appearance is defined relative to it.

Seen from a distance, the two phases read almost like two separate training runs glued together by one shared canonical gaussian cloud passed from the first into the second — because that is very close to what they literally are.

The routing rule underneath it is simple once you say it out loud: trust each prior only where it is actually good at something, and only when the pipeline needs that particular thing.

That single sentence is the whole chapter, really — everything from here to Chapter 10 is that sentence getting progressively more precise.

Reading The Crossover

Read two concrete points off it by hand. At occlusion certainty 0.05 — a flank pixel the camera looked at almost head-on — geometry-prior reliability is 1−0.05 = 0.95 and appearance-prior necessity is only 0.05: the geometry prior wins overwhelmingly, exactly as it should for a pixel the video already answers. At occlusion certainty 0.95 — a belly pixel the camera all but never saw — those numbers flip: reliability drops to 0.05, necessity climbs to 0.95, and the appearance prior takes over.

At occlusion certainty 0, the geometry prior is doing exactly what it's good at: it watched this patch of surface directly, so its guess about its shape is well grounded. At occlusion certainty 1, the geometry prior is still willing to guess — SAM3D always outputs a full closed shape — but that guess is now unconstrained by anything the camera actually measured, and its color/texture guess for that patch is close to a coin flip. That's exactly the gap the appearance prior exists to close, and exactly why it only gets to speak in Phase B, after the shape question is already settled.

You don't need to land exactly on the thin marker line to interact with the chart, either — click or tap anywhere across it and the marker jumps to that x-position, which matters as much on a phone as it does with a mouse.

The crossing point near the middle of the chart isn't a precise, paper-derived threshold — it's illustrative, the way the chart's caption says. What is real is the shape of the tradeoff: somewhere between "fully seen" and "never seen" sits a large gray zone where neither prior is obviously right, and Lift4D never has to resolve that zone by argument, because the two-phase split resolves it by construction — Phase A already fixed a shape for that zone before Phase B ever asks a texture question about it.

One more small thing worth naming: the "frozen prior" / "optimized this phase" tags on the flow boxes above are plain text, not icons. That's deliberate — a lock glyph and a pencil glyph would say the same thing with more visual noise, and the two-lane color coding (teal for geometry, purple for appearance) is already doing the job of making the phases visually distinct at a glance.

It's worth being honest about what this chapter has and hasn't shown you. The two-lane flow diagram and the crossover chart are both a map of the idea — frozen priors, phase-routed trust. Neither one is the actual mechanism that makes SAM3D temporally consistent in the first place, which is a separate, much more specific trick living entirely inside Phase A's first box. That's Chapter 4, and it deserves its own showcase rather than a footnote here.

Chapter 2 spends a little time on each of these three tools before Chapter 3 shows exactly how the released code wires them together — then Chapter 4 opens the causal trick this chapter has been gesturing at.

Chapter 2: Three Tools

If Chapter 1 told you which priors get trusted and when, this chapter is about what each one actually is under the hood — concretely enough to follow the released code in Chapter 3, and no further than that.

Before the pipeline in Chapter 3 makes sense, three borrowed tools need to already be sitting in your belt. None of them are new — the paper's own contribution starts after these three, not inside any of them. What follows is the minimum viable definition of each, just enough to keep reading; if any one feels shaky, the linked full lessons go from zero.

Think of this chapter as a toolbox check, not a lesson in itself. You'll meet each tool again doing real work: gaussians reappear as the thing every other chapter renders, flow matching reappears by name in Chapter 4, and score distillation reappears, in a more specific released-code form, in Chapter 8.

Tool 1: 3D Gaussian Splatting

A 3D gaussian is a soft, fuzzy little ellipsoid blob: a position, a shape (how stretched, and in which direction), a color, and an opacity. A gaussian-splatted scene is nothing but a pile of these blobs — tens of thousands of them, sometimes millions — projected onto the image plane and alpha-composited (blended nearest-to-camera first) directly into a picture. No neural network runs at render time. The blobs are the scene; rendering is just "project them, sort them, blend them."

Nothing about that definition depends on what the blob is a piece of. The same four fields describe a gaussian sitting on a rhino's flank, a wall in an indoor scene, or a strand of hair — the representation doesn't know or care what it's modeling.

That four-part description isn't a simplification for teaching purposes — it matches the actual fields the released SAM3D decoder writes out per gaussian: a 3D position, a scaling vector and a rotation quaternion (together, the "shape"), a color coefficient, and an opacity. When Chapter 3 says "the decoder outputs gaussians," this is concretely what it hands back, one small struct per blob.

Under the hood, the low end of that slider (5–16) is revealing the rhino's own ~16 base blobs one at a time — body, head, horn, ears, legs, tail, in that order, which is why a count of 5 shows a torso with no legs yet. Past 16, every additional step scatters one more small gaussian near an existing one, jittered a little in position and shrunk in size — a toy stand-in for what a real splatting pipeline's densification does when it clones or splits gaussians to cover detail the coarse set missed.

Notice what the slider does not change: color palette, overall pose, which parts exist. Densification in a real pipeline works the same way — it refines an already-roughly-correct scene with more primitives, it doesn't repaint or restructure it. A gaussian count going up is a resolution knob, not a redesign.

Tool 2: Flow-Matching Image-to-3D

SAM3D, the model Lift4D leans on hardest, turns one photo into a 3D object by flow matching: start from pure noise, and integrate an ODE (a rule for how the noise should continuously deform, step by tiny step) that carries it into a clean structured latent — a compressed code that lives on an occupied 3D voxel grid rather than one flat vector, so it has room to encode where geometry actually sits. A decoder turns that latent into gaussians, directly, no separate rendering loop required.

Put a real number on "structured latent" so it stops being an abstraction. SAM3D's sparse-structure code lives on a 16×16×16 voxel grid with 8 channels per voxel — that's 16×16×16×8 = 32,768 numbers, organized as 4,096 tokens of 8 dimensions each, before a decoder turns the occupied ones into gaussians. Compare that to one flat vector: a plain latent code has no notion of "where," only "what." A voxel-grid latent has both, which is exactly the room it needs to encode a whole 3D shape instead of a single class label.

Hold onto one detail from that inset: a rectified-flow ODE's path from noise to data is close to a straight line, by construction — that is what "rectified" means, and it looks nothing like a generic diffusion SDE's much wigglier path. A near-straight path means you can start integrating from partway along it, instead of from pure noise, and still land somewhere sensible. Chapter 4 is built entirely on that one fact.

One more thing to notice about the pipeline diagram above: nothing in it renders an image at any point. Encode, denoise, decode — the whole path from photo to gaussians never once passes through a differentiable renderer. That will change in Chapter 6, where rendering becomes the entire supervisory signal, but it's worth clocking that SAM3D itself doesn't need one to do its job.

Tool 3: Score Distillation

Score distillation sampling (SDS) is a way to use a frozen 2D image-diffusion model to supervise a 3D representation, without ever generating a picture you look at directly. Render your 3D asset from some camera, hand that render to the frozen diffusion prior, let it judge "here's roughly how you'd nudge this to look more like something plausible," and backpropagate that nudge straight into the 3D parameters — never into any pixels you'd actually display.

Why does it matter that the gradient lands on the 3D parameters and not on the rendered pixels themselves? Because the 3D representation is the only thing that has to stay consistent across every camera angle you might ever render from. A gradient aimed at "these specific pixels, from this specific view" fixes one picture. A gradient aimed at "this gaussian's color" fixes every picture that gaussian will ever appear in, from any angle — which is exactly the property Phase B needs, since it has to leave behind a texture that holds up under camera angles nobody asked for during training.

One honest caveat, so Chapter 8 doesn't surprise you: "SDS" covers a small family of closely related distillation recipes, not one fixed formula. The classic version backpropagates directly through a noise-prediction gradient. The released Lift4D code instead runs a short, no-gradient denoising loop down to a clean target image and then trains against that target with an ordinary pixel loss — same spirit, different plumbing. This chapter only needs you to know the spirit: render, ask a frozen prior how plausible it looks, push the 3D parameters toward "more plausible." Chapter 8 gets specific about which exact plumbing the released code runs.

All three widgets and diagrams on this page are built to survive a phone screen without losing their point: the gaussian-count slider is full width and needs no side labels, and both flow diagrams already stack vertically by default rather than sitting side by side, so nothing here depends on a wide viewport to make sense.

Three tools, three jobs: gaussians are what the scene is made of, flow-matching image-to-3D is how a single photo becomes a first guess at that scene, and score distillation is how a frozen 2D prior can still improve a 3D scene it never rendered pixels for. Chapter 3 shows exactly where the released code wires these three together.

Chapter 3 is where "three tools" stops being a list and starts being a script — the exact commands, files, and tensor shapes that carry a photo through all three of them and out the other side as a moving 4D asset.

Chapter 3: The Pipeline

A paper describes an idea. A repository describes what actually runs.

Lift4D's released code is three top-level Python scripts, launched by hand, one after another, from the command line. There is no single "Lift4D model" you load and call. There is no end-to-end training loop tying anything together at all.

There is a sequence of frozen tools and two short optimization runs, and the only thing connecting them is the files each stage writes to disk for the next one to read.

Keep that image in mind for the rest of this chapter: not a model, not a checkpoint you download once and query — a pipeline you re-run, start to finish, for every new video.

That is worth sitting with, because it explains a design choice that would otherwise look strange. Why does reconstructing one video need FOUR separate stages instead of one clean forward pass? Because each stage does exactly one job it is already good at, then hands off a concrete artifact — not a gradient, not a shared feature space, just files — to the next stage in line.

File-based staging is not just architectural cleanliness — it is a debugging gift. If Stage 3's canonical shape looks wrong, you don't have to touch the optimizer to find out why. Open Stage 2's raw per-frame result.ply files directly in any point-cloud viewer first.

If those already look broken, the bug lives in reconstruction, and no amount of tuning Stage 3's loss weights will fix it. If they look fine and only the CONSOLIDATED shape is wrong, the bug lives downstream instead. A single end-to-end network would give you no such checkpoint to inspect.

Four stages, three scripts

Stage 3 runs the SAME script twice, with different flags, because it is really two separate optimization phases — Chapter 6's geometry phase and Chapter 8's appearance phase — sharing one training loop. Here is the full chain, with what tensor flows across every arrow:

Look at the color coding. The two frozen stages, Segment and Causal SAM3D, never receive a gradient — nothing about either one changes between videos, or between any two runs of this pipeline at all.

The two optimized stages are colored to match Chapter 1's phase-routing lanes: teal for the geometry phase, purple for the appearance phase. Same two colors, same two jobs, now attached to real scripts you can actually run from a terminal.

Hold that mapping loosely for a second, because it is easy to over-read: "frozen" describes the STAGE's neural network, not the stage's output. Stage 2's weights never move, but its per-video OUTPUT — 32 fresh gaussian clouds — is different every single time you run it, on any video. Frozen model, novel output. That distinction is what makes a single pretrained image-to-3D model usable across an unbounded variety of in-the-wild footage without ever being retrained on any of it.

Stage 1 — two SAM3 models, never resident together

segment_video.py actually loads two separate SAM3 models, one after the other. The first is an image model: it looks only at the video's first frame, takes a text prompt, and returns candidate instance masks.

The second is a video model: it takes those first-frame masks as seeds and propagates them forward through every remaining frame by tracking, not by re-detecting from scratch on each one. The two models are never loaded at the same time — the image model is deleted and its GPU memory is explicitly freed before the video model loads.

That is not incidental code hygiene. It is the only reason this whole pipeline fits on a single GPU alongside everything Stage 2 and Stage 3 still have left to do. SAM3's video-propagation model also requires integer-named frames (0.jpg, 1.jpg, ...), so the script quietly builds a temporary folder of symlinks mapping the real frame_000000.jpg-style filenames into that convention, tracks through it, then writes masks back out under the real names.

If an object is fully occluded on some frame and nothing propagates there, the script does not skip that frame — it writes an all-black mask instead, so every downstream stage can always assume exactly one mask file per frame, no missing-file branches required.

The actual API calls, condensed from the released source — two very different calling conventions for the two models:

# image model — first frame only, text-prompted
state = processor.set_image(first_frame)
state = processor.set_text_prompt(args.prompt, state)
masks = state.get("masks", [])                 # one mask per detected instance

# video model — shares the detector backbone, then tracks forward
inference_state = predictor.init_state(video_path=tmp_frames)
for inst, m in first_masks.items():
    predictor.add_new_mask(inference_state=inference_state, frame_idx=0,
                            obj_id=inst, mask=torch.tensor(m))
for frame_idx, obj_ids, low_res, video_res, scores in predictor.propagate_in_video(
        inference_state, start_frame_idx=0, reverse=False):
    video_segments[frame_idx] = video_res > 0   # binarize, one obj_id per instance

Each selected instance keeps its own obj_id, so multiple animals in one clip propagate independently inside a single propagate_in_video call — no per-instance re-run needed.

Stage 2 — the causal reconstruction (mechanics: Chapter 4)

run_inference.py is the stage that turns "N independently-plausible single-image reconstructions" into "one temporally-consistent-ish reconstruction," and it does that without retraining a single weight — the entire mechanism is choosing WHERE each frame's denoising ODE starts. That trick is Chapter 4's whole subject; here, just the plumbing around it.

Depth Anything 3 runs by default (--run_da3) to hand the image-to-3D model metric depth as extra conditioning, on top of the image and the mask. Without it the model only knows RELATIVE depth, and a rhino at arm's length looks geometrically identical to a truck across a parking lot. The actual DA3 checkpoint is depth-anything/DA3NESTED-GIANT-LARGE-1.1, run at a fixed 504px processing resolution and then rescaled back up to the frame's real resolution — nearest-neighbor for depth and confidence, with the principal point re-centered on the full-size image.

An optional --refine_pose flag (off by default) re-renders each frame's silhouette with a differentiable rasterizer and nudges scale, translation, and rotation until it matches the SAM3 mask — a cheap fix for a real failure: the model predicts pose from a depth map that is only correct up to an unknown scale, so the raw pose can put an object at the right depth while getting its physical size wrong.

Stage 3A / 3B — one script, one custom loader, two very different jobs

Both training steps run through the same train_lift4d_scgs.py, and both skip the classical 3D Gaussian Splatting machinery you might expect. There is no COLMAP scene, no calibrated multi-camera rig, none of it. A custom data loader reads SAM3D's own frame_*/result.ply outputs and the raw ground-truth video frames directly.

Step A farthest-point-samples 2048 control nodes straight off the canonical, frame-0 Gaussian cloud, trains the deformation MLP ψ, and sets --disable_rendering_loss_iter 1 — which turns the input-photo rendering loss off almost immediately. For nearly the whole of Step A, the only signal shaping the canonical shape is Chamfer distance plus a multi-view loss compared against SAM3D's OWN per-frame reconstructions (Chapter 6), never the raw video.

Step B loads that checkpoint, freezes ψ entirely, and trains a second, denser set of control nodes whose only job is per-frame color and opacity correction (Chapter 8). Geometry, once Step A finishes, never moves again — Step B's roughly 10,000 additional optimization steps (checkpoint 10000 to iteration 19999) only ever touch appearance.

What each stage reads and writes

Only Stage 2 takes a photometric input that varies from frame to frame in a way its OWN output doesn't yet agree on across frames — which is exactly why Chapter 4 spends a whole chapter on it.

Follow one video through all four stages

Numbers make this concrete. Take a handheld, 32-frame clip of a goat — the paper's own convention for a short in-the-wild clip.

Watch the file count at every arrow: it stays at 32 for two whole stages, then collapses to 1 at exactly the point Step A does its consolidating work.

Stage 1 reads 32 JPEG frames and a text prompt, "goat," and writes 32 mask PNGs: frame_000000_goat_1.png through frame_000031_goat_1.png. One file per frame, always, even on the frames where the goat is briefly out of view.

Stage 2 runs the image-to-3D model 32 separate times, once per frame, chained forward from frame 0 by the causal warm start Chapter 4 dissects. Each run writes its own frame_XXXX/ folder — 32 of them — plus one shared da3/da3_output.npz covering the whole clip. Thirty-two independent gaussian clouds now exist, each roughly agreeing with its neighbors, none of them sharing a single point in common.

Stage 3, Step A loads all 32 result.ply files at once, farthest-point-samples 2048 control nodes from frame 0's cloud, and spends 9999 iterations — roughly half the total compute budget — consolidating those 32 independent clouds into ONE canonical shape plus a deformation schedule. It checkpoints at iteration 10000.

Stage 3, Step B loads that single checkpoint, freezes the shape it found, and spends 10000 more iterations painting in appearance the camera never directly saw. What began as 32 separate files is now one directory: gaussians.ply, deform.pth, and their appearance-phase siblings.

One more real consequence worth knowing before you run this on your own footage: if segment_video.py detects more than one instance of your prompt — two goats, tagged goat_1 and goat_2 — Stage 2 and Stage 3 each run once PER INSTANCE, completely independently. The pipeline reconstructs one 4D asset per object, never one shared asset for a whole multi-object scene. There is no cross-object interaction modeled anywhere in it; two goats sharing a frame become two entirely separate optimization runs that happen never to look at each other.

Eight flags worth knowing by name

Skim the command block below once, then come back to this list — every flag here shows up again in a later chapter, and none of them are self-explanatory from the name alone.

--video_consistency: Stage 2's causal knob, IS t0, Chapter 4's entire subject.

--disable_rendering_loss_iter: Stage 3A, turns off the input-photo rendering loss after this many iterations — set to 1, so effectively "almost immediately." Geometry gets shaped by Chamfer and multi-view terms instead (Chapter 6).

--smooth_transforms_window: Stage 3A, a fixed 5-frame moving-average low-pass applied once, at init, over Stage 2's raw per-frame poses Tⁱ — it removes pose-estimation jitter before optimization ever starts.

--lambda_rc / --lambda_chamfer: Stage 3A, the weights on the two loss terms that do the actual consolidating — Chapter 6's multi-view render loss and Chamfer distance, respectively.

--load_checkpoint: Stage 3B, the iteration number of the Step A checkpoint to resume from — this is literally how ψ gets frozen and carried forward.

--optimize_per_frame_compose_transforms_app: Stage 3B, a learnable per-frame correction applied only in the render/SDS/eval path — never touches the Chamfer loss that shaped Step A's geometry.

--occlusion_compositing: Stage 3B, optional, turns on Chapter 7's occlusion mask and histogram-matched compositing when set.

--enable_lpips: Stage 3B, adds a perceptual (LPIPS) term on top of the L1 + D-SSIM rendering loss — the same metric Chapter 9's scoreboard uses to grade the whole pipeline, now also shaping training, not just measuring the result.

The commands, in order

This is the actual sequence a user runs from a fresh clone — the same flags the paper's own experiments used, in the same order the repo's README lists them.

Notice Stage 2's command below needs an extra --mask_name goat_1 that a DAVIS or Consistent4D run wouldn't. --dataset custom resolves masks from a sibling masks/ folder named per-frame-per-instance — Stage 1's own output convention. The benchmark datasets resolve masks a different way each: DAVIS from a separate Annotations/ directory, Consistent4D from each frame PNG's own alpha channel. Same shared loader, three different mask conventions, one flag deciding which.

# Stage 1 — SAM3 text-prompt segmentation (skip for DAVIS / Consistent4D; masks ship with those)
python segment_video.py --video goat.mp4 --prompt "goat"

# Stage 2 — Causal SAM3D (frozen weights; --video_consistency IS t0, Chapter 4)
cd sam3d
python run_inference.py --dataset custom --object_name goat \
    --mask_name goat_1 --video_consistency 0.2 --render_video

# Stage 3, Step A — geometry phase (Chapter 6)
cd ../lift4d_scgs
python train_lift4d_scgs.py \
    --deform_type node --iterations 9999 \
    --lambda_rc 1 --lambda_chamfer 1 --disable_rendering_loss_iter 1 \
    --reset_opacity_rc --densify_and_prune_rc \
    --optimize_gs_xyz --opt_deform_rot --optimize_canonical_color \
    --smooth_transforms_window 5

# Stage 3, Step B — appearance phase (Chapter 8), resumes Step A's checkpoint
python train_lift4d_scgs.py \
    --deform_type node_delta --load_checkpoint 10000 \
    --iterations 19999 --lambda_sds_rgb 0.1 --enable_lpips \
    --optimize_per_frame_compose_transforms_app --compose_lr_reset \
    --densify_and_prune
    # optionally add --occlusion_compositing for Chapter 7's histogram-matched compositing

Videos longer than 50 frames double both step lengths — Step A runs to 19999 iterations, Step B to 29999, loading from checkpoint 20000 instead of 10000. Everything else about the recipe stays the same. On one H200, the whole 20,000-iteration Stage 3 run takes about 30 minutes for a 32-frame video.

What actually lands on disk

Stage 2 writes one folder per frame, sam3d_output/<tag>/frame_XXXX/, containing result.ply (that frame's Gaussians), obj_transform.json ({scale, translation, rotation, focal} — Tⁱ), and optionally result.glb if a textured mesh was also decoded. One level up, at the <tag> root: da3/da3_output.npz (Depth Anything 3's cached output, reused automatically on a re-run if the frame list matches) and comparison.mp4 if --render_video was set.

Stage 3 writes to lift4d_scgs/output/<tag>_<deform_type>/deform_gs/iteration_N/: gaussians.ply (the canonical shape) and deform.pth (ψ's weights), plus, Step B only, deform_node_base.pth (the frozen Step-A model it built on top of) and compose_transforms_app.pt. Alongside those: comparison and orbit-view videos, the orbit sampled at four fixed azimuths — 75°, −15°, −105°, 165° — matching the Consistent4D benchmark's own evaluation convention exactly.

That matters practically, not just architecturally: anything downstream of Lift4D — a viewer, a game engine, a robotics simulator — has to re-implement the deformation math from Chapter 5 to play this asset back correctly. There is no drop-in file format that already knows how to.

Stage 2 is where all of this either works or quietly starts to fail. Its output — N frames of individually-plausible, warm-started-into-agreement Gaussians — is only as trustworthy as the trick that produced it. Chapter 4 opens that trick up completely.

One last practical note: re-running this exact pipeline on the exact same video will NOT produce a bit-identical result. Stage 2 draws fresh Gaussian noise for every single frame as part of its causal warm start — Chapter 4's Z_i⁰ — and that noise is never fixed to a seed anywhere in the released scripts. Two runs land close, not identical.

Three things worth carrying out of this chapter and into every one after it. First, ONLY two scripts ever touch a neural network, and both stay frozen — everything Chapter 6 through Chapter 8 optimizes is gaussian parameters and a small MLP, never the priors themselves. Second, temporal consistency is decided at Stage 2, before Stage 3 ever runs a single optimization step; Stage 3 consolidates, it does not invent agreement between frames that Stage 2 never produced. Third, every artifact this pipeline produces is a plain file on disk — a PLY, a JSON, a checkpoint — which means every one of the next six chapters can be understood by asking the same two questions: what file does this stage read, and what file does it write.

Chapter 4: Causal Latent Propagation

The image-to-3D model powering Stage 2 has never seen a video. It was trained to look at ONE photograph and guess the object's full 3D shape and texture — nothing in its training ever taught it that "frame 14" and "frame 15" are supposed to agree.

Feed it 32 frames of the same rhino, one at a time, fully independently, and it reconstructs 32 individually-plausible rhinos. Play them back in order and the shape flickers between them. An ear phases in and out. A tail wobbles that never actually moved.

None of those 32 reconstructions is individually WRONG. Each one, judged on its own single photograph, is a perfectly reasonable guess. The problem only shows up in the sequence — thirty-two correct answers to thirty-two slightly different single-image questions, stitched together, look like one wrong answer to the actual question being asked: what did this rhino look like, moving, across one continuous clip?

Retraining this model on video data is exactly what Lift4D refuses to do. There isn't enough real 4D training data in the world to do that well (Chapter 0), and avoiding that trap is the whole reason this paper exists.

Everything in this chapter happens at TEST time — while reconstructing one specific video, never while pretraining v_θ itself. Nothing here produces a better version of the model for the next video; it produces a better sequence of calls to the exact same model for THIS one.

So instead of retraining anything, Lift4D changes ONE thing about how the frozen model gets called: WHERE its denoising process starts.

That framing matters more than it sounds like it should. A bigger fix — a video-native architecture, a temporal attention layer bolted onto the image-to-3D model, anything that touches weights — would need exactly the kind of large, paired 4D training data that Chapter 0 already established doesn't exist at scale. The causal trick sidesteps the data problem entirely by staying inside the model's ALREADY-LEARNED single-image distribution, and only ever changing which point in that distribution's denoising trajectory gets used as a starting condition.

Frame 0: the reference (Eq. 1)

Every flow-matching model like this one works the same way. Start from pure noise, then run an ODE — a smooth, deterministic differential equation with no randomness once it starts — that gradually nudges that noise into a clean sample. Here, "clean sample" means a 3D shape-and-texture latent.

The very first frame has no history to lean on, so it gets the ordinary treatment: pure noise in, the full ODE, one clean latent out.

Unpack every symbol before moving on. Z₀^t is frame 0's latent state at ODE time t, which sweeps continuously from 0 up to 1. v_θ is the flow-matching model's velocity field — a frozen, pretrained function that looks at the current latent and how far along t is, and predicts which direction to nudge it next.

C₀ is frame 0's conditioning: its image embedding, Depth Anything 3's metric depth, and its SAM3 mask. Z₀⁰, the starting point, is drawn fresh from a standard Gaussian distribution — pure noise, zero information about the object.

If you have not seen rectified flow before, park a plain mental picture here before the causal trick arrives: it is a straight-line-flavored cousin of diffusion, trained so v_θ points almost directly from a noisy point toward a clean one, rather than diffusion's more meandering, curved path. Chapter 2's background section builds this comparison up properly; the one thing worth carrying forward here is that "near-straight paths" is exactly the geometric property Eq. 2 is about to exploit.

Integrate from t=0 all the way to t=1 and you get Ẑ⁰, frame 0's finished, clean latent. Nothing exotic here — this is a completely ordinary single-image reconstruction, run exactly once, at the start of the video.

Which frame is "previous," exactly?

It is tempting to assume propagation always runs strictly forward — frame 0, then 1, then 2, in index order — and warm-starts every frame from "frame i−1." The released code is slightly more general than that, and the difference is worth knowing before you trust the word "previous" too literally.

Processing runs bidirectionally FROM the reference frame: forward first, all the way to the last frame in the clip, then backward from the reference back down to frame 0. Each frame warm-starts from whichever neighbor was processed immediately before it in THAT traversal order — frame_idx − 1 while moving forward, frame_idx + 1 while moving backward. "Previous" means "already-processed temporal neighbor," not "lower frame index."

The simulation below simplifies to a single forward chain, frame 0 through frame 6 — the common case, and the one every one of the paper's own numbers assumes. The ghost trail's fan-back visual is exactly this chain, just drawn for the simple forward-only direction.

One consequence worth flagging: the reference frame doesn't have to be frame 0 at all — it is just wherever initial_frame_index points, and the paper's own default happens to set that to the clip's first frame. Pick a reference in the MIDDLE of a clip instead, and propagation runs outward in both directions from there, each direction its own independent forward chain.

Frame i > 0: warm-started (Eq. 2)

Every later frame gets a head start. Instead of beginning at pure noise, its ODE starts partway along — at a point that already leans toward what the PREVIOUS frame decided the object looked like.

Read the blend line first. Z_i⁰ is a BRAND NEW fresh noise sample for frame i, drawn independently, exactly the way frame 0's was. Ẑⁱ⁻¹ is not new at all — it is the previous frame's already-finished, clean latent, the exact Ẑ that frame i−1 produced when ITS OWN ODE integration reached t=1.

t0 is a single number between 0 and 1 that does two jobs at once. It is the mixing weight in the blend above, AND — look at the second line — it is literally where frame i's ODE integration begins. The integral only runs from t0 to 1, never from 0. Frame i never gets a full, independent reconstruction; it gets whatever fraction (1−t0) of a fresh reconstruction still fits in the time left after the warm start.

If SDEdit is new to you: the original technique edits an existing image by adding a controlled amount of noise to it, then running only the LATTER part of a generative model's denoising schedule — enough noise gets added, and removed again, to smooth out unwanted detail while the overall composition survives, because the model was never given the freedom to wander all the way back to pure noise. Lift4D reuses precisely that shape of idea, except the "existing image being edited" is the previous frame's finished 3D latent, not a 2D picture.

Same model, same weights, same frozen v_θ, called on every single frame, video after video. The ONLY thing that ever changes is the starting point of an ODE integration it was always going to run anyway.

A natural instinct here is: why not just AVERAGE each frame's latent with the previous one, 50/50, every time? Two reasons that fails. First, a fixed blend never lets the model see MORE of the new frame's actual evidence when the object genuinely changed a lot — t0 gives you a dial, a fixed average gives you nothing to turn. Second, and more subtly, averaging alone never runs any MORE of the denoising ODE. Eq. 2's second line matters as much as its first: after blending, the model still integrates v_θ forward from t0 to 1, actively refining the blended point using THIS frame's own conditioning C_i. A plain average would just hand back a blurry, un-refined interpolation — not a plausible 3D reconstruction of anything.

The knob: t0

Push t0 up and the blend leans harder toward the previous frame's finished latent, AND the ODE has less remaining time to deviate from it — both effects point the same direction: more consistency, less freedom to follow this frame's own new information.

Push t0 down and the opposite happens: more fresh noise, more ODE time to run, more freedom to follow what this specific frame actually shows — and more flicker against its neighbors. Try it below. This is the exact trade-off, made felt instead of described.

The simulation ties every visible quantity to the SAME t0 you drag: the warm-start point in the top panel, the ghost-trail chain it pulls from, the resulting silhouette's jitter in the bottom panel, and both gauges. Nothing in the interactive is decorative — every pixel that moves is moving because Eq. 2 says it should.

Worked example: t0 = 0.2

Do the arithmetic once, by hand, at the paper's own default. The interactive above updates the same four numbers live as you drag — work the table out first, then confirm it against the "at t0 = ..." readout under the gauges.

Fifty is the sparse-structure generator's real default step count. At t0 = 0.2, warm-starting doesn't just nudge the initial point — it skips a fifth of the diffusion schedule outright, spending its whole remaining budget refining a latent that already starts out 20% "the previous frame."

Ten skipped steps sounds small until you remember that classifier-free guidance strength for this stage is 7 — a strong pull toward the conditioning at every single step the model DOES run. Skipping steps at the noisy end of the schedule, where the model has the least useful signal to guide with anyway, is a cheap place to spend that 20%.

Put in words, with no math notation at all: t0 = 0.2 means the ODE integrates only the final 80% of its usual path, starting from a point that is already 20% previous-frame-latent and 80% fresh noise.

Say it once more, backwards this time, since both directions are worth being able to produce on demand: 20% of the way through the full noise-to-data schedule is where this frame's integration BEGINS, not where it ends — the remaining 80% of the schedule is still real, still frozen-model computation, just aimed at a starting point that already carries a fifth of the previous frame's answer.

One more output rides along for free

Decoding a frame's clean latent Ẑⁱ doesn't only produce that frame's Gaussians Gⁱ. The SAME decode step also emits Tⁱ — the object's pose relative to the camera for that frame, an SE(3) transform plus scale ({scale, translation, rotation}), recovered by a dedicated pose-decoder from the model's own scale-shift-invariant pose prediction.

Chapter 3's pipeline table lists Tⁱ as Stage 2's second output for exactly this reason: it falls out of the SAME forward pass that produces the Gaussians, not a separate camera-calibration step bolted on afterward. Chapter 6 leans on this pose directly; Chapter 7's occlusion mask leans on the render it makes possible.

Tⁱ is estimated independently for every frame — the causal warm start above never touches it, only the shape-and-appearance latent does. That is precisely why Chapter 3 mentions a fixed 5-frame smoothing pass over the raw Tⁱ trajectory before Stage 3 ever runs: pose, unlike shape, gets no causal consistency fix at Stage 2 at all, so a separate cleanup step has to catch its frame-to-frame jitter downstream instead.

Does the warm start actually help?

The paper ran the ablation directly: strip out the causal warm start, reconstruct every frame independently (t0 forced to 0, always), then re-run the exact same downstream optimization on the resulting per-frame Gaussians.

FVD — Fréchet Video Distance, a measure of how believable a video's MOTION looks, not just its individual frames — moves the most: a 35-point jump. The paper's own words for what goes wrong without the warm start: "deformation worsens, geometry jitters across frames."

Notice that batch-wise reconstruction is not catastrophic on its own — 0.120 LPIPS is still a strong per-frame result. The causal trick's job isn't to rescue bad individual frames. It's to make already-good individual frames agree with each other.

That distinction is why Chapter 9's ablation table separates this fix from "no L_reg" — removing motion regularization from Stage 3's optimizer hurts FVD even worse, 794.82, but for a completely different reason: it lets the deformable optimizer overfit per-frame noise it's given, rather than failing to receive consistent input in the first place. Two independent sources of temporal jitter, fixed at two different stages, by two different mechanisms.

Try to break it: t0 = 1.0

The slider above stops at 0.5, on purpose — that already covers the repo's realistic range. But nothing in Eq. 2 stops you from pushing t0 all the way to its mathematical limit. Work out the algebra before you look at the answer.

Follow the chain all the way back. Propagation is sequential: frame i warm-starts from frame i−1, which warm-started from frame i−2, all the way back to the reference frame. Push t0 to 1.0 and that entire chain of "zero fresh information" propagates forward. Every later frame becomes an unchanged copy of frame 0.

You would get perfect 4D consistency and a perfectly rigid, perfectly wrong object — it would never move, because nothing about it after frame 0 was ever actually allowed to get computed.

That is the whole shape of the trade-off, compressed into one extreme: t0=0 throws away all temporal information and risks flicker; t0=1 throws away all NEW information and guarantees a frozen, motionless reconstruction. Every usable value lives strictly between the two, and the paper's own experiments cluster tightly around 0.1–0.2 for exactly that reason — close enough to 0 to let genuine motion through, far enough from 0 to damp flicker.

The paper's own appendix says almost exactly this, in its own words: high t0 can suppress legitimate deformation, low t0 fails to prevent geometric flickering, and rigid sequences — objects that genuinely don't deform much frame to frame — benefit from pushing t0 higher than a heavily-deforming one would tolerate. A tripod-mounted still life and a running dog do not want the same t0.

Stage 2 is now finished producing what it can: N frames of individually-plausible, temporally-consistent-ish Gaussians, each still sitting in its own arbitrary pose. They don't share a single skeleton yet — nothing so far says which gaussian on frame 3's ear IS the same gaussian on frame 4's ear.

The causal warm start bought agreement between neighboring frames' rough shape. It did not, and cannot, buy CORRESPONDENCE — a shared identity for individual points across the whole sequence. That consolidation is Chapter 5's job.

Chapter 5: Control Nodes

Stage 2 handed you something almost usable: per-frame gaussians Gⁱ, one full 3D reconstruction for every frame in the clip. Almost. Frame 3's ear and frame 4's ear are two entirely different gaussians, sitting in two entirely independent per-frame reconstructions, that happen to occupy roughly the same place in space. Nothing in Stage 2's output says "these two blobs are the same physical ear, one frame apart." Chapter 4's causal warm start made the shapes agree with each other frame to frame — but shape agreement is not correspondence. You still don't have one object that moves. You have thirty-two snapshots of an object that happens to look similar in every one of them.

This chapter builds the missing piece: one canonical set of gaussians, G^*, plus a rule for how every single one of them moves at every frame. The rule turns out to have surprisingly few moving parts — and the drag toy below will show you exactly why that's a feature, not a shortcut.

One canonical shape, a handful of levers

Pick frame 0's reconstruction, G⁰, and freeze it as the canonical shape G^*. Every later frame's gaussians get thrown away once training starts — they've already done their job, supplying Chapter 6's loss targets. From here on only G^* and a deformation rule exist.

The tempting rule: give every single canonical gaussian its own learned SE(3) transform (a rotation plus a translation) per frame, and let gradient descent sort out how each one should move. Resist it. A typical G^* here is tens of thousands of gaussians. Learning tens of thousands of independent 7-number transforms, for 32+ frames, from nothing but a handful of Chamfer and render-loss gradients per iteration, is wildly underdetermined — nothing in that parameterization tells two neighboring gaussians on the same patch of rigid skin that they're neighbors, so nothing stops the optimizer from yanking them in unrelated directions.

The fix, borrowed wholesale from prior work on sparse deformable gaussians (SC-GS): control nodes. Sample a small number of points spread evenly over G^*'s surface — the paper's appendix uses 1,024, chosen by farthest-point sampling (repeatedly pick whichever remaining surface point is farthest from every point already chosen, so the nodes spread out instead of clumping) — and give only those points learned per-frame motion. Every canonical gaussian then inherits its motion by blending the motion of its few nearest control nodes, weighted by distance. Drag a node below and you'll feel exactly why this beats the one-transform-per-gaussian idea.

Watch the on-canvas readout above the status line: it tracks one reference gaussian (ringed in pink) and shows the blending weight each of its three nearest nodes currently contributes, in sparse mode. Drag the OTHER nodes around and those numbers don't move — only dragging one of the reference gaussian's own three nearest nodes changes its weight split. That's Eq 13 being literal: which nodes influence a gaussian, and how much, is decided once, from the canonical (rest) geometry. Nothing about a live drag ever changes a weight; a drag only supplies the per-node motion those fixed weights get multiplied against.

One more detail worth knowing before moving on: the same densification-and-pruning machinery that lets a 3D Gaussian Splatting scene grow or shrink its point count during training (clone a gaussian where the loss gradient is large and under-reconstructing; delete one that's faded fully transparent) also runs on this canonical representation. G^* isn't frozen at whatever count Stage 2 happened to hand you for frame 0 — it can still gain or lose gaussians as Phase A trains, exactly like any other splat scene.

The deformation MLP ψ

Each control node p_k needs a per-frame rigid motion: a rotation R_kⁱ and a translation t_kⁱ. Both come out of one small multilayer perceptron, ψ, queried once per node per frame — not once per gaussian. Querying it 1,024 or 2,048 times per frame is cheap. Querying it per-gaussian, tens of thousands of times, would erase the entire point of going sparse.

The math: from a handful of node motions to one gaussian's motion

Position first. For gaussian m, let S be the set of its K nearest control nodes (by distance in the canonical shape). Each node in S contributes its own rigid-motion guess for where μ^*_m should go, and those guesses get averaged, weighted by w_mk:

Read the bracket as "if node k were the ONLY thing moving this gaussian": rotate the gaussian's offset from the node (μ^*_m − p_k) by that node's rotation, add the node's own (unmoved) position back, then add the node's translation. Eq 11 is a weighted vote across K such guesses.

Orientation follows the same shape, but quaternions don't average the way vectors do — you compose (multiply) each node's rotation onto the gaussian's own canonical orientation q^*_m, weight each result, and sum:

⊗ is quaternion multiplication (composing two rotations into one). A weighted SUM of unit quaternions isn't itself a unit quaternion in general, so this raw result gets renormalized back to unit length before it's used as a rotation — you'll see that exact step in the worked example below.

Both equations lean on the same weight, w_mk, and that weight is where the "sparse but smooth" behavior actually comes from:

This is a Gaussian radial basis function (RBF): a node's vote fades smoothly with distance, controlled by its own radius o_k. The appendix marks o_k as learnable — the optimizer can widen a node's reach (large o_k, gentle falloff, influences gaussians farther away) or sharpen it (small o_k, steep falloff, only the very closest gaussians hear from it) independently for every single node.

There's a practical reason K stays small, too, beyond the smoothing argument above. Every one of G^*'s tens of thousands of gaussians needs to look up its K nearest control nodes, every single training iteration. With K = 3 or 4 and N_p in the thousands, that's a cheap K-nearest-neighbor query per gaussian. If instead every gaussian had to weigh in on ALL N_p nodes — the fully-dense limit, no sparsity at all — the per-gaussian cost would scale with N_p instead of K. Sparsity here is a compute decision as much as a regularization decision.

Worked example: one gaussian, four nearest nodes

Take one canonical gaussian at μ^* = (0.40, 0.25) in some local 2D patch of the canonical shape (units are arbitrary — treat them as fractions of the object's overall size). Its four nearest control nodes, with positions, raw distances, and each node's own learned radius o_k:

Two things worth stopping on. First: the weights sum to 0.2465+0.3848+0.1166+0.2521 = 1.0000, exactly as Eq 13's normalization guarantees. Second, and more interesting: node 4 is the single farthest of the four (d₄ = 0.1044, larger than every other distance here), yet it ends up with almost as much blending weight as node 1, which sits closer (d₁ = 0.0943). Why? Node 4's learned radius (0.09) is wider than node 1's (0.08) — its softer falloff buys back the distance it's giving up. That is the entire point of making o_k learnable: a node's reach is not fixed geometry, it's a parameter the optimizer can tune per node.

Now blend a position. Say ψ predicts these four small translations for the current frame (representing d_xyz — think "roughly a centimeter of local shift on a meter-scale object"): node 1: (0.010, −0.004), node 2: (0.006, −0.002), node 3: (0.015, 0.008), node 4: (−0.004, −0.001). Using the near-zero-init-simplified form of Eq 11 (rotations near identity, so each node's vote reduces to "my own translation"), the blended shift is ∑ w_mk t_k:

Notice the individual node translations range from −0.004 to +0.015 — a 5× spread — but the blended result moves the gaussian by only about (0.0055, −0.0011), a fraction of any single node's raw vote. That smoothing is Eq 13 doing exactly its job: no one node can yank a gaussian around on its own; it always has to out-vote its neighbors.

Now the rotation half. Suppose the same four nodes predict small twists (rotation about one axis, for illustration) of θ₁=6°, θ₂=−4°, θ₃=10°, θ₄=2°, and the gaussian's own canonical orientation is untwisted (q^*_m = identity, so q_kⁱ ⊗ q^*_m = q_kⁱ directly). Each quaternion is (cos(θ/2), 0, 0, sin(θ/2)):

Weight each quaternion and sum component-wise: raw = (0.2465·0.9986 + 0.3848·0.9994 + 0.1166·0.9962 + 0.2521·0.9998,  0.2465·0.0523 + 0.3848·(−0.0349) + 0.1166·0.0872 + 0.2521·0.0175) ≈ (0.9989, 0.0140). That's not a unit quaternion (its length is 0.99904, not 1), so Eq 12's last step renormalizes: divide both components by 0.99904 to get (0.9999, 0.01405). Converting back to an angle, sin(θ/2) ≈ 0.01405 → θ ≈ 1.6°.

Chapter 6: Geometry Losses

Two loss terms, one shared discipline: everything in this chapter grades the deformable canonical shape against Stage 2's own outputs — never against the raw video. That restriction is deliberate, and by the end of the chapter you'll know exactly why.

Chapter 5 gave the canonical shape a way to move: a handful of control nodes, blended by distance, driven by ψ. Nothing has told that machinery what motion is CORRECT yet. That's this chapter's job — and it's harder than "just compare pixels to the photo," because the thing you're comparing against isn't perfectly trustworthy either.

Remember what Stage 2 actually gives you: 32 independent per-frame reconstructions, each with its own noisy estimate of where the object sits relative to the camera. Chapter 4's causal warm start keeps their SHAPE consistent frame to frame. It does not promise their global POSE is consistent — each frame's object-to-camera transform Tⁱ is still its own independent guess. If you naively match the deformed canonical points against a drifting per-frame target, the optimizer will "fix" pose noise by warping the shape, because warping the shape is the only lever it has.

Chamfer distance: matching two point clouds without a template

Before Eq 4, the background piece: given two point sets A and B with no known point-to-point correspondence, the Chamfer distance asks, for every point in A, "how far is my nearest neighbor in B?" — and does the same the other way, from B back to A — then averages both directions:

Why nearest-neighbor matching, instead of just comparing point 1 to point 1, point 2 to point 2? Because there IS no shared indexing here. The canonical gaussians and a given frame's Stage-2 reconstruction were never the same array subsampled twice — they came from independently splatting two different diffusion samples, with no promise that gaussian #4,502 in one corresponds to anything in particular in the other. Chamfer distance sidesteps needing correspondence at all: it just asks each point to find whichever partner in the other set happens to be closest, right now, and scores THAT.

Bidirectional matters: A→B alone can't punish B for having extra points nowhere near A (every extra point in B just finds SOME nearest A point and stays silent); B→A alone can't punish A the same way in reverse. Concretely: if B secretly grew a fourth point way off to the side — a stray gaussian the reconstruction shouldn't have — the A→B sum wouldn't even notice, because none of A's three points would ever pick that stray point as their nearest neighbor. Only the B→A direction catches it, because that stray point still has to find ITS nearest neighbor in A, and whatever it finds will be far away. Together, both directions have to agree the clouds overlap well; either alone can be fooled by extra structure on the side that direction doesn't check.

Here's the trap. Chamfer distance has no notion of "these two clouds are the same shape, just shifted." Shift one cloud sideways by a few centimeters and the Chamfer distance goes up exactly as if the SHAPE had changed, even though nothing about the shape did. Watch this happen with real numbers.

Set A (canonical): a₁=(0, 0), a₂=(2, 0), a₃=(1, 1.732). Set B (a per-frame reconstruction: the same rough triangle, plus a small amount of shape noise, plus a deliberate global drift of about (0.6, 0.5)): b₁=(0.7, 0.6), b₂=(2.5, 0.4), b₃=(1.65, 2.1). All nine pairwise distances:

Bold marks each row's (and, here, each column's — the pairing happens to be symmetric) nearest neighbor. A→B mean squared distance = (0.922²+0.640²+0.747²)/3 = (0.850+0.410+0.558)/3 = 0.606. B→A comes out identical here, since the matching is symmetric. Total:

Now watch what a pure re-centering does. Centroid(A) = (1.0, 0.577); centroid(B) = (1.617, 1.033); the difference, (0.617, 0.456), is almost exactly the drift we injected. Shift every point in B by minus that difference: b₁′=(0.083, 0.144), b₂′=(1.883, −0.056), b₃′=(1.033, 1.644). Recompute distances to the SAME matched partners: d(a₁,b₁′)=0.166, d(a₂,b₂′)=0.129, d(a₃,b₃′)=0.094.

1.212 down to about 0.035 — a roughly 34× drop — and not one point in B moved relative to its neighbors. All that changed is where the whole cloud sits. That gap, 1.212 versus 0.035, IS the reason Eq 4 needs an alignment step baked in, not bolted on after the fact.

The interactive below runs an O(N²) brute-force nearest-neighbor search over just 6 points a side — fine for a toy. The released optimizer runs this same bidirectional search over two 4,096-point clouds, every single training iteration, via a batched GPU K-nearest-neighbor routine (pytorch3d.ops.knn_points, K=1 in each direction) rather than anything resembling a nested Python loop.

Eq 4: Chamfer with a per-frame escape valve

Read the two sets: {μ^*_m} is the deformed canonical gaussian means at frame i (Chapter 5's output). {μ_i^m} is frame i's own per-frame reconstruction means, straight from Stage 2 — but first passed through T_i^align, a per-frame, per-iteration LEARNABLE rigid alignment (the centroid shift above is a simplified stand-in for it; the real T_i^align is fit by gradient descent, jointly with everything else, to minimize exactly this loss). Every frame gets its own T_i^align, because every frame's pose drift is different.

The second readout above is the actual correction being applied — the vector T_align would add to every point in B before scoring it, computed here as the simple centroid-matching translation that (for pure translational drift, exactly what dragging simulates) is already very close to what a learned T_align converges to. The real released optimizer discovers this vector by gradient descent, jointly with everything else; a closed-form centroid match is this canvas's stand-in for that convergence.

Keep the drag toy's two lessons distinct. Drag WITHOUT auto-align and you're watching what an under-regularized Chamfer term does to a drifting frame: the number inflates, and an optimizer chasing that number has only the canonical shape left to blame. Drag WITH it on and you're watching what T_align is actually FOR: absorbing exactly that inflation before it ever reaches the shape gradient.

In the released code

Searching the actual Stage-3 training loop turns up no learnable per-frame transform gating the Chamfer term specifically. The bidirectional Chamfer (subsampled to 4,096 points per side) compares the deformed canonical points directly against each frame's raw result.ply xyz — in the same canonical/object-relative frame, no explicit alignment step in between. Two OTHER, related mechanisms do the alignment work Eq 4 describes on paper, just not inside the Chamfer term itself:

1. --smooth_transforms_window 5 — a FIXED, non-learnable moving-average low-pass filter, run ONCE at initialization over the raw per-frame object-to-camera transforms Tⁱ that Stage 2 already produced (with the rotation quaternions hemisphere-aligned before averaging). Hemisphere alignment matters because a quaternion q and its negation −q represent the EXACT same rotation; if consecutive frames' rotations happen to be stored on opposite sides of that double-cover, naively averaging q_t and q_t+1 can produce something close to zero — nonsense, not a rotation at all. Flipping one quaternion's sign before averaging (whenever the two are more than 90° apart in quaternion space) fixes this. The whole filter removes Stage-2's frame-to-frame pose JITTER before optimization ever starts — a cleanup pass, not a learned correction.
2. --optimize_per_frame_compose_transforms_app — a genuinely LEARNABLE per-frame (scale, rotation, translation) triple. But it lives entirely in the render and SDS path, gated on in Step B (Chapter 8), and never touches loss_chamfer, which is a Step-A-only term.

So the Chamfer term in the released pipeline ends up leaning on Chapter 4's causal warm start to have already kept consecutive frames' canonical poses roughly aligned, rather than on any explicit per-frame re-alignment inside Stage 3 itself. The interactive above teaches Eq 4's IDEA faithfully; just don't go looking for a tensor named T_align in the checkpoint.

One more code detail worth flagging while we're here: the Chamfer term doesn't run over every gaussian in G^* against every point in that frame's target — both sides get randomly subsampled to 4,096 points before the nearest-neighbor search runs. Nearest-neighbor lookups are Chamfer's expensive part; naively, cost scales with the product of both cloud sizes. Subsampling to a fixed 4,096 keeps every training iteration's cost constant regardless of how many gaussians G^* currently has, without meaningfully changing what the loss measures — 4,096 points is still a dense sample of a gaussian cloud's overall shape.

The other half of L_rec: rendering from a random camera

Why sample cameras on a whole SPHERE, rather than just near the input viewpoint? Because the canonical shape has to be right from angles nobody scored it from yet — the entire point of this pipeline is a 4D asset viewable from anywhere, and the belly nobody filmed still needs a consistency signal pulling it toward Stage 2's own (independently, per-frame) reconstructed belly, not just toward whatever the input camera happened to see. Random spherical sampling is what makes L_mv a full-coverage loss instead of a front-view-only one.

In the released code, L_render is actually four terms, not two:

• L1 + D-SSIM (weighted λ_dssim=0.2) — the paper's two terms, unchanged.
• 0.1×LPIPS — a learned perceptual-similarity term, scoring whether two images look alike to a network trained on human similarity judgments, not just pixel-by-pixel.
• A Charbonnier-smoothed depth term anchoring against Stage 2's own rendered depth from the same random camera. The Charbonnier loss, √(x² + ε²), behaves like a smoothed absolute-value: nearly linear (robust to outliers, like L1) for large errors, but smoothly curved near zero instead of L1's sharp corner there, which keeps gradients well-behaved as the depth error shrinks toward convergence.
• An outside-mask transparency penalty, pushing any gaussian rendered OUTSIDE the object's silhouette toward zero opacity, cleaning up stray floaters the optimizer might otherwise grow.

Cameras sit on a sphere at orbit distance 3.0 by default (with a handful of DAVIS-specific overrides in the repo — e.g. parkour: 1.0, hockey: 4.0 — because different source videos frame the object at very different scales); --rc_batch_size controls how many such random cameras get sampled and averaged per iteration.

Between Chamfer and this random-camera render loss, Phase A now has everything it needs to fuse Stage 2's temporally-consistent-but-uncorrelated per-frame gaussians into one coherent, deformable canonical shape — without a single glance at the actual input video. Chapter 7 is where that changes.

One nuance worth being precise about: the released Step A recipe does include an --optimize_canonical_color flag, so canonical gaussian color isn't frozen during Phase A either. The point stands regardless of that detail — whatever color signal reaches the gaussians in Phase A still flows through L_mv, graded against Stage 2's own render, never against a real photograph. The distinction this whole section is building toward isn't "does color ever update in Phase A," it's "what does color get COMPARED against," and the answer never changes until Chapter 8.

One thread to hold onto going into Chapter 7: everything in this chapter compared a render to ANOTHER render, or a point cloud to another point cloud — never once to the pixels the camera actually recorded. That restraint is deliberate, not an oversight, and the next chapter explains exactly why touching the real photo has to wait.

Chapter 7: Occlusion Mask

A note before diving in: this is the shortest chapter in the lesson in terms of new machinery — one equation for detection, one for compositing — and the longest in terms of what it takes to make that machinery survive contact with real pixels. Both halves matter.

Chapter 6 gave Phase A a plausible SHAPE for the whole animal, including the belly no camera ever pointed at. But every loss you've seen so far only ever compared a render to ANOTHER render — canonical vs. Stage 2's own reconstruction. Nothing has looked at the real photo yet.

That has to change eventually: the point of this entire pipeline is a 4D asset that matches what actually happened in the video, not just a shape that's internally consistent with itself. But looking at the real photo is dangerous if you do it carelessly. A hand passes in front of the animal for six frames. If you naively supervise every pixel against the raw photo, you'll teach the optimizer that the animal's shoulder is shaped like a human hand for those six frames — a real occluder, mistaken for real geometry.

Before Chapter 8 can safely use the photo, you need to know exactly WHICH pixels in it are trustworthy, and stitch together one reference image that mixes real pixels where they're safe with a plausible substitute everywhere they aren't. That's this chapter, in two equations.

Naive photometric supervision against the raw video actually breaks in two separate ways, and it's worth being precise about which one this chapter fixes. First: the camera only ever sees roughly a third of the animal at once (Chapter 0's coverage problem, still unsolved by anything so far), so photometric supervision is inherently SPARSE — partial-observation gradients can corrupt geometry that Chapter 6's Chamfer and render-loss terms already got right, undoing their regularization. Chapter 8 fixes that half, by freezing ψ and adding a denser, appearance-only set of control nodes so coarse motion can't be disturbed. Second, the problem THIS chapter fixes: even within the region the camera DOES see, some of those pixels are lying, because something else in the scene is standing in front of the animal. Chapter 7 exists to sort the trustworthy pixels from the lying ones before Chapter 8 ever touches them.

Notice, too, what this problem is NOT. It isn't "segment the hand out of the frame" — SAM3 already did that back in Stage 1, and Stage 1's mask already excludes the hand from the object mask entirely. The question here is narrower and more specific: given that the photo mask already says "not object" at those pixels, is that BECAUSE something is blocking the view, or because the reconstruction's own silhouette is simply wrong there? Those two situations need opposite treatment, and nothing about a single frame's mask, on its own, tells you which one you're looking at.

Eq 7: a depth test, AND a disagreement test

Two equations, one job: decide, per pixel, per frame, whether the real photo is safe to learn from directly.

Four ingredients, all of which you've already met:

• D_i^scene — the REAL photo's depth at frame i, from Depth Anything 3 (the same monocular depth estimator that conditioned Stage 2, back in Chapter 3). It's an estimate of how far every pixel in the actual footage is from the camera, whatever is really there — animal, hand, background.
• D_i^πc — the CURRENT 3D asset's own rendered depth, from π_c, the input camera (not a random sphere camera like Chapter 6 — specifically the one the footage was actually shot from). How far the reconstructed surface THINKS it is from the camera at that pixel.
• M_i — SAM3's mask on the real photo, unchanged since Stage 1. Does the segmenter think the object is at this pixel?
• M_i^πc — the current asset's own rendered silhouette from π_c. Does the RECONSTRUCTION think the object is at this pixel?

Every one of those four signals is either measured directly from the real footage (D_scene, M_i) or produced by rendering an asset (D^πc, M^πc) — specifically, as the next section makes precise, a single snapshot of the asset taken right where Chapter 6's Phase A finished, not something continuously re-rendered as Step B trains on top of it.

The XOR, (M_i ⊕ M_i^πc), finds every pixel where the photo mask and the render mask DISAGREE about whether the object is there. Since Phase A already recovered a full, plausible 360° shape, the render will confidently say "yes, object" in plenty of places the photo's segmenter says "no" — sometimes because something is physically blocking the camera's view there, sometimes just because the render is wrong. The XOR alone can't tell those two cases apart.

That's what the depth test is for. (D_i^scene < D_i^πc) asks a narrower, physical question: is something in the REAL photo sitting nearer to the camera than where the render believes its own surface is? Only a genuine foreground occluder satisfies that — a render error that hallucinates geometry which isn't really there does NOT, because the real photo at that pixel shows whatever's actually behind (farther away, or equal), never something artificially closer. AND the two conditions together, and you get exactly the pixels worth treating with caution: the render and the photo disagree, AND the reason is a real object blocking the view.

Try dropping the depth test and keeping only the XOR. Every single frame, the render's silhouette edge and the photo mask's silhouette edge will disagree by a pixel or two purely from antialiasing and slightly different thresholding — nowhere near an occluder, just numerical noise along a boundary. XOR alone would flag thousands of harmless edge pixels every frame for the histogram-matching treatment, quietly degrading Eq 8's composite along the ENTIRE silhouette, every frame, everywhere. The depth test is what keeps Eq 7 selective: only pixels where something is confirmed physically nearer than the reconstructed surface survive it, and stray edge noise almost never satisfies that — an edge pixel's depth reads roughly the same as its neighbor's, not meaningfully closer.

Eq 8: stitching a reference image you can trust everywhere

Eq 7 answers "which pixels." Eq 8 answers the next question: given that answer, what single image do you actually hand Chapter 8?

⊙ is element-wise multiply — each mask picks out its own region of pixels, and the two regions get added together into one composite. Where M_i is true (the real photo shows the object, unoccluded), just use the real pixels, I_i, directly — nothing beats ground truth where you actually have it. Where M_i^occ is true (Eq 7 flagged a genuine occluder), the real pixel is untrustworthy by construction, so paste in the RENDER's own color instead — but color-corrected first (the "histogram-matched render," walked through below), because the raw render and the real photo tend to agree on shape while disagreeing on saturation and white balance, and a raw, uncorrected patch would bake a visible seam right into the one region Chapter 8 needs to trust most.

One honest scope note: Eq 7 is a genuinely PIXEL-wise test in the real pipeline — hundreds of thousands of independent boolean evaluations per frame, one per rendered pixel. This canvas evaluates the same logic at the coarser granularity of individual gaussian blobs (a few dozen, not hundreds of thousands) purely so the drag interaction stays legible at this size. The boolean logic is identical either way; only the resolution it's applied at differs.

The bottom readout evaluates Eq 7's own ingredients live, at wherever you've currently dragged the hand: whether the photo mask currently sees the object there, whether the render's silhouette always does, and whether the (simplified, always-true-when-overlapping) depth test passes. Drag fully off the animal and watch M_i flip back to "yes" with nothing flagged — the third row of the truth table below, live.

Watch the % readout as you drag slowly across the animal's outline: it should rise smoothly, not jump in steps, as more of the fixed gaussian set falls inside the occluder's radius. If it ever looks like it's snapping rather than sliding, that's the coarse-blob-granularity scope note above showing through — a real pixel-resolution mask would read smoother still.

Nothing here is faked for display: the % readout, the boolean-state readout, and Panels A/C/D all draw from the exact same occlusion() computation.

Two things to actually try. First, drag the hand fully OFF the animal, onto empty background — Panel D goes dark (nothing flagged), because with no XOR disagreement in the object region there's nothing for the depth test to even evaluate. Second, at 0% match strength in the I_full view, the pasted-in patch reads as a visibly wrong tint against the surrounding real pixels — drag the slider up and watch the seam disappear. That seam is exactly what Eq 8's histogram matching exists to erase.

Reading the table below: each row is one PIXEL's situation at one frame — not a whole-image classification, and not a whole-object classification. Different pixels in the same frame can land in different rows simultaneously; that's exactly why Eq 7 has to run per pixel rather than once per frame.

(If you predicted Panel B would never react to the drag — correct. That's the whole point of "always solid": the render doesn't know a hand exists, because nothing about a hand was ever part of what it reconstructed.)

Rows 1 and 3 are the easy half, and worth naming explicitly precisely because they're easy: whenever M_i and M_i^πc already AGREE — both say "background" (row 1) or both say "object" (row 3) — the XOR is false before the depth test ever runs. Eq 7 short-circuits: there's no disagreement to explain, so nothing gets flagged, and it doesn't matter in the slightest what the two depth values happen to be. The depth comparison only ever matters on the DISAGREEMENT rows.

Row 2 and row 4 are the pair worth sitting with, because they look identical on the mask alone: both have M_i=no, M_i^πc=yes, so both trigger the XOR. The depth test is the ONLY thing separating "flag this" from "leave it alone." In row 2, the hand really is in front of the animal, so the real photo's depth at that pixel reads near — nearer than the render's own believed surface. In row 4, nothing is actually there; the real photo's depth at that pixel reads whatever's genuinely behind the animal (background, or simply level with the reconstructed surface) — never nearer. Same mask disagreement, opposite depth relationship, opposite outcome.

Traced as literal boolean algebra, rows 2 and 4 diverge at the very last step:

Note what "— (short-circuits)" means concretely: for rows 1 and 3, XOR already evaluates to F, and F AND anything is F, so the depth values never even need to be read for those pixels to be correctly excluded. Only rows 2 and 4 — where the two masks actually disagree — make the depth comparison do any work at all.

One honest limitation worth naming here, since it will matter again in Chapter 10: Eq 7 is only as good as its two upstream inputs. If SAM3's mask itself under- or over-segments the occluder in Stage 1, that error propagates straight through — a missed sliver of the hand simply never gets flagged, and no amount of clever depth-test logic downstream can recover pixels the mask never told it to look at. Chapter 7 is deterministic and correct GIVEN its inputs; it has no mechanism for noticing when those inputs are themselves wrong.

Training a dedicated occlusion-detection network was never really an option here anyway: it would need LABELED occlusion masks for in-the-wild footage — ground truth for "is the camera's view of the object blocked at this pixel, in this frame" — and no such labels exist for arbitrary Pexels or DAVIS clips at the scale this pipeline needs. Eq 7 sidesteps the labeling problem entirely by deriving occlusion from signals that are already available for free: a segmenter that was going to run anyway (Stage 1), a monocular depth model already in the loop for Stage 2's conditioning, and the render pipeline's own depth and alpha channels, which come for free with every differentiable rasterization pass.

From boolean logic to actual pixels: three fixes the code adds

Eq 7 as written above is exact. Turning it into a clean, usable mask over real, noisy sensor data takes three more practical steps that the equation doesn't show. None of the three changes WHAT gets flagged in principle — each one exists purely to keep noise in the depth or mask signals from corrupting that decision.

1. Rescale the depths before comparing them. Monocular depth estimators like Depth Anything 3 are scale-ambiguous: they recover accurate RELATIVE distances (this pixel is twice as far as that one) but not an absolute metric scale, on their own. By the time you're deep into Stage 3's optimization, the reconstructed asset's own rendered depth has drifted from whatever scale DA3 originally assumed back in Stage 2 — every loss term in Chapters 5–6 has been quietly reshaping the geometry since then. So the code re-anchors DA3's occlusion-time depth to the CURRENT render's scale, using the MEDIAN (not the mean — resistant to the odd wildly-wrong depth pixel at a silhouette edge) over the region where both signals agree the object is visible:

# vis: pixels where the photo mask, the render mask, AND a valid
# rendered depth all agree — the one region both depths can be trusted.
vis = m_sam_bool & m_rend & (sam3d_d > 1e-3)
s = torch.median(sam3d_d[vis]) / torch.median(da3_depth[vis]).clamp_min(1e-6)
da3_aligned = da3_depth * s

# Eq 7, translated directly: depth test AND xor-disagreement AND
# restricted to where the render actually believes there's an object.
m_occ_bool = (da3_aligned < sam3d_d) & (m_rend ^ m_sam_bool) & m_rend

Every pixel of DA3's photo depth gets multiplied by that single per-frame factor before it's compared to the render's depth anywhere in the image — including inside the occluded region itself, which is exactly where you need a correct comparison and exactly where you couldn't have computed s directly (that region is, by definition, in dispute).

A toy illustration of the arithmetic (invented numbers, to make the mechanism concrete — not a real logged frame): suppose the trusted vis region's median depth reads 0.58 in the render's own units but 2.30 in DA3's raw units. Then s = 0.58 / 2.30 ≈ 0.252. Now take a pixel under the hand, where DA3's raw (unscaled) reading is 2.10 and the render's own believed surface depth is 0.60. Naively comparing 2.10 to 0.60 would say the hand is FARTHER than the object — backwards, because the two numbers were never on the same scale to begin with. Rescale first: 2.10 × 0.252 ≈ 0.529. Now compare 0.529 (rescaled scene) to 0.60 (render): 0.529 < 0.60, correctly passing the depth test — the hand really is closer.

Carry that same pixel through the rest of Eq 7. Say, at that pixel, SAM3's photo mask reads M_i = 0 (the hand is there, not the animal) and the render's own silhouette reads M_i^πc = 1 (the reconstruction believes the animal's surface is there). XOR(0, 1) = 1 — they disagree. AND that with the depth test result just computed (also 1, since 0.529 < 0.60): 1 AND 1 = flagged. This single pixel walks the exact path row 2 of the truth table describes, with real intermediate numbers attached to every step.

2. Dilate the trusted region out, then morphologically clean the flagged region.

m_occ_np = m_occ_bool.cpu().numpy() & ~binary_dilation(m_sam_np, iterations=2)
m_occ_np = binary_dilation(binary_erosion(m_occ_np, iterations=1), iterations=2)

The first line grows the SAM mask by 2 pixels and subtracts that grown region out of the flagged mask — so no pixel within 2px of the real visible-object boundary can ever be marked occluded, which keeps noisy, anti-aliased edge pixels from being mistaken for a discovered occluder. Concretely: a pixel sitting exactly 1px inside the SAM mask's true edge is, by construction, well within a 2px dilation of that same edge — so it gets subtracted out unconditionally, no matter what the depth test at that pixel says. The second line is a morphological "open": erode by 1px first (an isolated 1-pixel fleck of false-positive flagging — a stray depth glitch — has no 1px-thick neighborhood to survive on, so it vanishes), then dilate by 2px (the larger, coherent occluded region that DID survive the erosion gets regrown, and slightly padded back out). Noise gets erased; the real occluder survives.

3. Feather the edge. A hard 0/1 cutoff at the mask's boundary would bake a second visible seam right where the occlusion mask itself ends. Instead, a distance transform (scipy.ndimage.distance_transform_edt) measures each flagged pixel's distance to the nearest UNflagged pixel, and converts that into a soft weight that ramps from 0 to 1 across a 20px band — pixels deep inside the flagged region get full weight, pixels right at its edge get a fractional blend. Same idea as the histogram-matching slider you just dragged: never let a hard edge exist where nothing hard exists in reality.

A toy illustration of the ramp (the real falloff shape isn't specified beyond "distance-transform-based," so treat this as illustrative, not exact): a pixel sitting 2px inside a flagged region's edge, with a 20px feather band, gets a blend weight around 2/20 ≈ 0.10 — mostly still deferring to the real photo just outside the flag. A pixel 15px deep into the same region gets roughly 15/20 = 0.75, weighted heavily toward the color-matched render. By the time you're ≥20px from any edge, the weight has saturated to 1.0 and the pixel is treated as fully, unambiguously occluded.

The two constants here — a 2px dilation and a 20px feather — trade off in opposite directions, and neither is free. Grow the dilation past 2px and you start throwing away real, trustworthy pixels right at the object's true boundary, shrinking how much of the animal the render loss ever gets to see. Shrink the feather band well below 20px and the composite edge sharpens back toward a hard cutoff, reintroducing exactly the visible seam Eq 8's histogram matching exists to erase. Both numbers were tuned, not derived from first principles — they're a compromise between "don't discard good data" and "don't leave an artifact for the render loss to latch onto."

Histogram matching, the actual mechanism (Eq 8's "matched render"). The code doesn't build explicit histograms. It sorts. For each color channel independently, sort the SOURCE pixels (the render's own colors, restricted to the region where you can already verify them — M_i's visible area) and the TARGET pixels (the real photo's colors, same visible region) separately. Then, for any query pixel, find its RANK among the sorted source values via binary search, and replace its value with whatever sits at that SAME rank in the sorted target array:

src_ref = sam3d_norm[m_sam_bool]; dst_ref = gt_norm_f[m_sam_bool]
for c in range(3):
    src_sorted, _ = src_ref[:, c].sort(); dst_sorted, _ = dst_ref[:, c].sort()
    ranks = torch.searchsorted(src_sorted, all_pixels[:, c].clamp(src_sorted[0], src_sorted[-1]))
    all_pixels[:, c] = dst_sorted[ranks.clamp(0, dst_sorted.shape[0]-1)]

Read that in words: "the k-th brightest render pixel becomes exactly as bright as the k-th brightest real pixel," per channel. That IS histogram matching — sort-and-lookup is a compact, GPU-friendly equivalent to building and inverting a cumulative histogram by hand. Fit only where both signals are verifiable (the visible region), then applied to the ENTIRE render, including the occluded pixels where you can't verify anything — extrapolating a color-correction mapping learned where you can check it to where you can't, but need it.

Notice the direction the fitting runs: SOURCE is the render's colors, TARGET is the real photo's colors, both measured only where they can be cross-checked. Get the two swapped and you'd be correcting the trustworthy real pixels to match the untrustworthy render instead — exactly backwards.

A tiny hand-worked pass through that same code, on invented small numbers. Suppose the visible region's SORTED render-red values (the source) are [0.40, 0.55, 0.70], and the visible region's sorted real-photo-red values (the target) are [0.30, 0.50, 0.68]. A query pixel in the occluded region has render-red = 0.55. torch.searchsorted finds where 0.55 sits in the sorted source array — at index 1 (it's the middle value, exactly matching src_sorted[1]). The matched output is whatever sits at that SAME index in the sorted target array: dst_sorted[1] = 0.50. The render's mid-brightness red pixel becomes exactly as bright as the real photo's mid-brightness red pixel — not because anyone computed a color offset or a gain factor, just because both distributions got sorted and the ranks were matched up.

In the released code, Eq 8's clean two-region partition is actually built as a short sequence, not one formula evaluated in a single pass. First: ref_img = GT_photo × Mi + white × (1 − Mi) — a photo-on-white cutout, not yet touching the occluded region at all. Then, and only then: ref_img = matched_render × occ_w + ref_img × (1 − occ_w), where occ_w is the SAME soft, feathered weight from the 20px distance-transform band above — not a hard 0/1 mask like the truth table's M_occ. The feathering IS the compositing step here, not a separate polish pass tacked onto a hard boundary afterward.

One more thing worth knowing about WHEN all of this runs: the entire pipeline above — depth rescale, dilate, erode, feather, histogram fit — is computed ONCE per frame, before Step B's training loop even starts, not recomputed every training iteration. The occlusion mask itself doesn't change as the gaussians train; only the render loss's TARGET (built once, from this precomputed mask) does. That's a meaningful cost saving: distance transforms and per-channel sorts are comparatively expensive operations you very much do not want inside a tight optimization loop.

Worth flagging where D_i^πc and M_i^πc actually come from, mechanically: nothing new. They're the same depth and alpha channels every differentiable 3D Gaussian Splatting renderer already produces alongside its RGB output — render the current asset from π_c once, and depth, alpha, and color all fall out of that one rasterization pass. Eq 7 doesn't need a second, specialized render; it reuses output the pipeline was already computing.

Rendering D_i^πc and M_i^πc at all presupposes you already know exactly where π_c is — the camera's pose and focal length for this specific frame. That's not new information conjured here; it's the same obj_transform.json (scale, translation, rotation, focal) that Stage 2 already wrote out per frame back in Chapter 3, reused as-is.

And the I_full composite you're building above doesn't only feed Chapter 8's SDS conditioning. In the released code, the SAME precomputed composite also serves as the TARGET for Chapter 8's occlusion-aware render loss — one computed image, reused for two different loss terms, rather than two separate compositing passes that could quietly drift out of agreement with each other.

Stack the three fixes in order and the pipeline is: rescale, so the two depths are comparable; combine into the boolean mask; clean the boolean mask up (dilate-out, then open); feather its edge into a soft weight. Each step consumes the previous one's output.

I_i^full is the payoff of this whole chapter: one image, for one frame, that's real everywhere it can afford to be and honestly-substituted everywhere it can't. Chapter 8 conditions a view-synthesis prior on exactly this image — never on the raw, occluder-corrupted photo, and never on the render alone.

One last thread before the quiz. Everything in this chapter has been about the SINGLE input camera, π_c. Chapter 8's SDS term will introduce a second kind of camera — a randomly sampled NOVEL view, used to ask the appearance prior what the unseen side of the animal should look like. Keep those two roles distinct going in: π_c is where you decide what to trust; the novel views are where you spend that trust.

That's the whole chapter: two equations, three practical fixes, and one image handed forward.

Chapter 8: Painting the Unseen

Phase A (Chapters 5–6) earned you something real: a canonical shape G⋆ that deforms plausibly through every frame, fit against stage 2's own per-frame reconstructions. That shape is correct where the camera looked and correct-by-inference everywhere else — the sparse control nodes and their skinning weights don't know the difference between "the flank the camera filmed" and "the belly it never saw." Geometry is done.

Texture is not. Every render loss so far has compared against stage 2's own gaussians, and stage 2's gaussians were themselves only ever trained to look right from roughly where the camera stood. Nothing in Phase A has ever asked "what color is the far side of this rhino?" Phase B has to answer that question using nothing but a generic prior for "what backs of objects tend to look like," and it has to do it without wrecking the geometry Phase A just finished earning.

Eq 6: the appearance loss, in one line

Phase B's objective replaces Phase A's Chamfer-and-multi-view loss with:

Read the two terms as two different jobs graded on two different sets of pixels. L_render is the occlusion-aware photometric loss from Chapter 7 — it grades pixels the camera actually saw (or can be honestly reconstructed via the histogram-matched I_full). L_SDS (this chapter's new term) grades pixels nobody saw, using a generative prior instead of a ground-truth pixel to compare against. One loss for "match reality," one loss for "make something plausible up." They are never asked to agree on the same pixel — that's the whole design.

Why ψ has to be frozen for this to work

A score-distillation gradient is noisy by construction — it comes from a generic diffusion prior that has never seen this specific rhino, only millions of unrelated objects. If that gradient could push the deformation network, one bad SDS step could warp the animal's actual silhouette in the name of "looking more plausible" from some random new angle. Freezing ψ makes that mechanically impossible: no matter how loud or wrong the appearance gradient gets, the shape parameters simply have no gradient path to receive it.

Every drag attempt on a grey node gets refused — it shudders and tells you why. Every drag on a purple node moves nothing but a soft ripple of hue and brightness across the nearby blobs; the silhouette underneath never bends. That is Phase B's entire design compressed into something your hand can feel: geometry is a closed door, color is the only room left open.

The duplicate-control-nodes trick

"ψ is frozen" doesn't mean geometry is completely locked — the paper is more careful than that. Phase A's coarse motion needs to stay put, but small per-frame corrections that help appearance land correctly (a limb settling half a centimeter, a joint's rotation firming up now that color gradients are pulling on it) are still allowed. The mechanism: duplicate the control nodes.

Take the exact same control-node positions Phase A converged on. Instead of routing them through ψ again, give each node its own small, directly-optimizable per-frame SE(3) — no MLP, no shared weights across frames, just a free parameter per node per frame. Only these new per-frame transforms, plus the canonical gaussians' own color and opacity, get gradients in Phase B. The coarse motion Phase A already found stays exactly where it was; the new nodes only ever absorb small appearance-fitting adjustments on top.

# train_lift4d_scgs.py, node_delta step (Phase B)
with torch.no_grad():
    d_xyz_base = self.deform_node_base(canonical_xyz, fid)   # frozen ControlNodeWarp, ψ's output

d_xyz_delta = self.delta_node_warp(canonical_xyz, fid)        # DeltaNodeWarp, trainable this phase
d_xyz = d_xyz_delta + self._compute_node_delta_base(canonical_xyz, fid)  # = d_xyz_delta + d_xyz_base

The paper's own phrase for this is a "denser set of duplicate control nodes" — and it's worth being precise about what "denser" means here, because it isn't spatial. The node count and positions are unchanged from Phase A. What's denser is temporal: instead of one shared MLP curve producing a smooth trajectory for every node across all frames, each node now gets an independent free vector at every single frame — far more parameters in time, at the same spatial resolution. That's exactly the freedom you'd want for absorbing small per-frame appearance-fitting jitter without touching the coarse trajectory underneath.

Why condition on I_full, not the raw photo

Chapter 7 built I_full — real pixels where the camera saw them, histogram-matched reconstruction pixels stitched in under the occluder. It would be tempting to skip that step and just hand Stable Zero123 the raw frame I_i: fewer moving parts, one less thing to compute.

It would also be worse. The raw photo has a hole in it exactly where the prior needs its cleanest signal — wherever something is occluding the object, I_i shows the occluder, not the object. A view-conditioned diffusion prior anchors everything it imagines about unseen angles on the one clean image it's given; feed it a photo with a hand in the way and the prior has to either hallucinate around the hand or (worse) let the hand's colors leak into its guess for the object's far side. I_full removes that ambiguity before conditioning ever happens — the prior always gets a complete, occlusion-free reference of exactly this object's colors, which is precisely why the paper reports conditioning on I_full "substantially improves novel-view quality" over conditioning on I_i directly.

Eq 9: pixel-space score distillation

Now the actual mechanism that paints the unseen regions. For a randomly sampled novel camera π:

Unpacked, symbol by symbol: render the current deformed-and-colored canonical gaussians from novel camera π to get Î_i^π. Encode that render to a latent z, add noise to get z_t, then ask the frozen Stable Zero123 diffusion model to denoise z_t back to a clean guess ẑ — conditioned on I_full and steered by classifier-free guidance. D is the diffusion decoder, mapping ẑ back to a pixel-space image. The loss is a squared pixel error between your render and that decoded target, plus L_p (LPIPS, a perceptual distance) between the same two images. ω_t is uniform — no fancy timestep reweighting, just an average over sampled noise levels.

In the released code: the DDIM loop, exactly

The codebase doesn't implement the classic epsilon-prediction SDS gradient (a sds_loss method exists in stable_zero123_guidance.py but is never called from the training loop). What actually runs is a short, fully no_grad DDIM denoise producing a detached clean target, exactly matching Eq 9's "decode first, then compare" form:

# stable_zero123_guidance.py + train_lift4d_scgs.py, Phase B, per SDS step
sdsrgb_xyz_in = sdsrgb_deformed.detach()          # geometry detached — ONLY color/opacity get gradient
rendered = render(sdsrgb_xyz_in, live_colors, novel_az, novel_el, dist=3.8)

ddim_target, w = zero123_guidance.ddim_target(
    rendered.detach(), elevation_deg, azimuth_deg, distance,
    ddim_steps=5, guidance_scale=3.0)   # fully no_grad; a real denoising loop, not one gradient step

loss_sds_rgb = 0.1 * F.mse_loss(rendered_norm, ddim_target.detach()) \
             + 0.1 * lpips(rendered_norm, ddim_target.detach())

Five denoising steps, guidance scale 3.0, camera placed at a fixed conditioning distance of 3.8 from the object. The two 0.1's in that last line come from different places, worth not conflating: the MSE weight is --lambda_sds_rgb, a CLI flag the README's Step B recipe happens to set to 0.1; the LPIPS weight is a fixed 0.1 baked into the loss function itself, not exposed as a flag at all. The geometry going into that render is .detach()'d before rendering even starts — not just "no gradient flows back," but literally disconnected from the autograd graph, so there is zero possibility of a stray gradient reaching ψ through this path. set_active_frame(frame_idx) swaps in that frame's precomputed I_full embeddings before each call, so the conditioning is genuinely per-frame, not one global reference for the whole video.

One more paper↔code delta worth flagging precisely: the paper states the diffusion timestep is sampled uniformly from [0.2, 0.5]. The released defaults (sds_min_step/sds_max_step) are 0.3/0.6 instead — a real, if modest, shift toward noisier (higher-timestep) denoising targets than the paper's own stated range. The crop fed into this whole pipeline is 256×256, matching the appendix's stated SDS crop resolution.

When this all switches on

Recall Eq 10 from Chapter 6 — the whole pipeline's loss is a hard curriculum switch, not a blend:

N_rec = 10,000, out of 20,000 total iterations. Every mechanism in this chapter — the frozen ψ, the duplicate delta nodes, the occlusion masking, the DDIM-target SDS — is off for the entire first half of training and switches on all at once at iteration 10,000. There is no gradual hand-off; k crosses N_rec and the optimizer starts receiving a completely different objective on the next step.

Chapter 9: The Scoreboard

Every mechanism this lesson has built so far was a bet: causal propagation instead of independent per-frame denoising, sparse nodes instead of one transform per gaussian, a frozen ψ instead of letting everything move at once, an occlusion mask and pixel-space SDS instead of a naive photometric loss. Every bet was justified by argument. Does any of it actually help, once you strip away the diagrams and the analogies?

Four ablations, three benchmarks, and one comparison against six competing methods answer that with numbers instead of intuition.

Does the causal trick actually matter? Five configurations, head to head

Start with the piece most likely to feel optional on first read — the causal warm-start from Chapter 4 — and see what the paper's own numbers say about removing it.

All five rows below are the same Lift4D pipeline, evaluated on the Consistent4D synthetic benchmark (7 input videos, 4 ground-truth novel views each, 32 frames — the one benchmark with true novel-view ground truth, which is what makes LPIPS and FVD computable at all). Each row removes exactly one piece. Switch the metric, then tap or hover a row to see what the paper says breaks.

Two numbers are worth holding in your head before the table below: batch-wise (no causal warm-start) costs 0.120 LPIPS / 627.90 FVD, and no L_reg costs 0.122 LPIPS / 794.82 FVD — noticeably worse on FVD despite starting from the SAME causally-consistent per-frame geometry. Removing L_SDS entirely is the single worst ablation by a wide margin (0.170 / 1242.32 / CLIP 0.848), confirming that the generative prior is doing real, load-bearing work, not decoration.

The dashed vertical line running through the ladder is a reference, not decoration: it marks the best external competitor's score on whichever metric you're viewing (DM4D's 0.128 on LPIPS, PAD3R's 645.09 on FVD — both read straight off the "Against the field" table just below). Watch how many rows cross to the wrong side of it. On LPIPS, "No L_SDS" (0.170) and "L_rec only" (0.160) both land worse than DM4D's best external number. On FVD, "No L_reg" (794.82) joins them past PAD3R's 645.09. These ablations don't just cost Lift4D some ground against its own full configuration — they cost it its lead over the field entirely, on that metric.

One more piece of hand arithmetic, comparing the ablations to each other rather than to the field: the causal warm-start's LPIPS contribution is 0.120−0.116 = 0.004. L_SDS's LPIPS contribution is 0.170−0.116 = 0.054 — roughly 13× larger. That doesn't mean Chapter 4's trick matters less; it means the two mechanisms operate on different scales of the problem. The causal warm-start is a precision fix — it keeps per-frame geometry from drifting by small amounts every frame. L_SDS is the ONLY signal touching entire regions of the object that would otherwise have no color information at all. Losing precision costs a little; losing an entire information source costs a lot.

Against the field

Internal comparisons are a lower bar than they sound — a method can beat every stripped-down version of itself and still lose to the field.

Internal ablations answer "does removing this piece hurt." A field comparison answers a harder question: "does keeping every piece actually beat everyone else's best attempt at the same problem."

The ablations show what each piece is worth internally. This table shows what the whole assembled pipeline is worth against six other published methods, same benchmark, same protocol:

BANMo is evaluated synthetic-only in the paper — it doesn't appear in the in-the-wild numbers below. Lift4D leads all three metrics; the closest competitor per metric differs (DM4D on FVD, PAD3R on CLIP), which is itself informative — no single baseline is Lift4D's nearest neighbor across the board.

One naming note before the family breakdown: DM4D in this table is short for DreamMesh4D. The paper's baselines list also names a broader DreamScene4D family alongside it, grouped together in its discussion of prior work rather than broken out as separate numbered rows in this specific comparison — DreamMesh4D is the member of that family the paper actually benchmarks head to head.

Chapter 1 named two failure families for the problem Lift4D solves, and this table is a good place to see them by name. L4GM is the feedforward family — a network trained end-to-end to predict 4D directly, bottlenecked by scarce 4D training data. STAG4D, DM4D, PAD3R, V2M4, and BANMo are all optimization methods, Lift4D's actual peer group — each fits a representation at test time, the same way Lift4D does, just with a different prior doing the fitting (PAD3R and V2M4 use an image-to-3D prior for initialization only, then hand off to video-diffusion-style refinement; STAG4D and DM4D lean on video diffusion priors throughout). Lift4D's argument isn't "optimization beats feedforward" in general — L4GM actually beats STAG4D and V2M4 on FVD. It's that decoupling which prior drives which phase beats every single-prior optimization method tried, feedforward or not.

The arithmetic, done by hand: against DM4D (the closest FVD competitor), Lift4D's 592.44 is a (688.84−592.44) / 688.84 = 14.0% reduction. Against PAD3R (the closest CLIP competitor), 0.950 / 0.942 − 1 = a 0.85% relative CLIP gain — small in isolation, but CLIP is already near its ceiling for every method in this table (0.81–0.95), so small gaps there are harder won than the same-sized gap on LPIPS or FVD. On LPIPS against PAD3R: (0.137−0.116) / 0.137 = a 15.3% reduction — roughly the same relative size as the FVD gain, which is a useful sanity check that the improvement isn't an artifact of one metric's particular scale.

AdamW doesn't run at one learning rate for the whole 20,000 steps, either — canonical gaussian positions decay from position_lr_init 1.6e-4 down to position_lr_final 1.6e-6, a 100× taper the README's per-phase recipes further compress into whatever step budget each phase is given (10,000 / 20,000 / 5,000 depending on the run). Feature (color), opacity, scaling, and rotation each get their own fixed rate: 0.0025, 0.05, 0.001, and 0.001 respectively. Both appearance channels (color, opacity) move faster than both shape channels (scaling, rotation) — opacity fastest of all at 0.05, fifty times scaling or rotation's rate — matching this whole chapter's story that geometry is the parameter you protect and appearance is the one you let move freely.

Three released knobs exist purely to trade that 30 minutes against quality, and they're worth knowing exist before you assume the number is fixed: the released code notes Phase B captures most of its gain by roughly 6,000 iterations into that phase, so --ddim_steps 3 (down from the default 5) trims Chapter 8's DDIM loop for faster SDS at a small quality cost; --knn_batch_size 512 lowers the peak VRAM the local-rigidity term needs; and --rc_batch_size 1 (versus the README's default of 2 random cameras per step) further reduces memory but is flagged in the repo's own notes as measurably degrading results — a rare case of the codebase explicitly telling you which speed knob isn't free.

In the wild: no ground truth, different metrics

Consistent4D is synthetic, which means it ships genuine held-out novel-view frames — the only reason LPIPS and FVD are computable at all, and worth being precise about why. LPIPS is a per-frame pixel-aligned distance: it needs the exact same camera pose and the exact same instant rendered twice, once by Lift4D and once by the ground-truth renderer, so it can diff two images that should line up pixel-for-pixel. FVD (Fréchet Video Distance) is looser but no less demanding of ground truth — it runs a pretrained video classifier over a whole clip, collapses that into a feature distribution, and measures the statistical distance between the distribution of Lift4D's rendered clips and the distribution of the REAL held-out novel-view clips. No pixel-for-pixel alignment needed, but it still needs an actual reference set of real videos from the withheld viewpoints to build that second distribution against.

Pexels and DAVIS are real footage; nobody filmed the rhino's other side, so there is no ground-truth novel view to diff against, pixel-aligned or distributional. The paper switches to two metrics built for exactly that situation:

The two in-the-wild sets aren't picked to be easy. Pexels' 10 clips are chosen specifically for variety — a mix of deformable, rigid, and occluded subjects under diverse lighting, each with its own SAM3-produced mask and Depth Anything 3 scene depth rather than a curated benchmark's clean annotations. DAVIS' 8 clips are a standard, independently-curated video-segmentation set the paper repurposes rather than a set built for this task at all. Neither one gives Lift4D a home-field advantage.

Read the empty cells as informative, not missing. LPIPS and FVD are dashed out for Pexels/DAVIS because both require a ground-truth novel-view video to diff against, and in-the-wild footage has none — the same reason the paper states FVD explicitly as Consistent4D-only. CLIP-T (text-alignment score) is dashed for Consistent4D simply because the paper's main comparison table doesn't report it there, not because it's inapplicable.

The CLIP and CLIP-T numbers themselves follow one fixed protocol across every benchmark: render three fixed novel orbit views — 90°, 180°, and 270° away from the input camera — then take the cosine similarity between each rendered view's CLIP embedding and the corresponding input frame's embedding, averaged over every frame and all three views. CLIP-T swaps the reference from "the input frame" to "a text description," averaged the same way. Fixing the orbit angles matters: it means every method in the table is graded from the identical set of unseen viewpoints, so a higher score genuinely reflects better unseen-region reconstruction, not a luckier camera draw.

In the released code, the actual comparison videos you'd inspect by eye come from a separate rendering pass: save_orbit_video writes one video per orbit view, default --num_orbit_views 4, at azimuths 75°, −15°, −105°, 165° — a fixed 4-view rig chosen to match Consistent4D's own evaluation convention, and (when the benchmark's ground-truth multi-view set is found on disk) it also writes a side-by-side comparison video against it. This is separate machinery from the CLIP-scoring protocol above; think of it as the pipeline's own eyes-on sanity check.

EPE: measuring correctness with no ground truth at all

CLIP and CLIP-T cover appearance in the wild. Geometry still needs its own check.

That leaves one real question: how do you grade 4D reconstruction accuracy on video with no ground-truth 3D of any kind? The paper's answer is end-point error (EPE), and it leans on something Lift4D never explicitly trained for.

Run CoTracker3 — an off-the-shelf point tracker — on the raw input video with a 20×20 sampling grid, producing roughly 400 tracked 2D points per video. Take frame 0's tracked pixels, match each one to its nearest vertex on Lift4D's canonical gaussian cloud, then follow that geometric point through the deformation network ψ across every later frame and project it back into the input camera. EPE is the pixel distance between where the deformation says that point should be and where CoTracker3 actually tracked it.

By hand, for one point at one frame: say CoTracker3 tracked a spot on the rhino's shoulder to (125, 90) in that frame's pixel coordinates. Project the SAME gaussian, carried forward through ψ's learned deformation, into that camera and it lands at (120, 84). The end-point error for that single point-frame pair is the plain Euclidean distance between the two: √((125−120)² + (90−84)²) = √(25+36) = √61 ≈ 7.81. Do that for all ~400 tracked points, at every frame, and average — that average is the one number reported per video. A method whose deformation drifts off the real motion accumulates larger and larger point-frame errors as frames go on; a method that's actually tracking the same physical point stays close throughout.

On Pexels, Lift4D scores 0.072 EPE against the best baselines' 0.119 (PAD3R) and 0.136 (STAG4D) — V2M4 and DM4D trail further behind at 0.211 and 0.140, and L4GM doesn't report an EPE number at all on this benchmark. On DAVIS, Lift4D's 0.161 again leads a tighter field: STAG4D 0.189, V2M4 0.195, DM4D 0.197, PAD3R 0.205. For CLIP, Lift4D's in-the-wild lead is real but narrower — Pexels 0.780 against STAG4D's 0.757, DAVIS 0.715 against V2M4's 0.637.

Same hand arithmetic, applied here: on Pexels, Lift4D's tracking error against PAD3R's best is (0.119−0.072) / 0.119 ≈ 39.5% lower. On DAVIS, against STAG4D's best, it's (0.189−0.161) / 0.189 ≈ 14.8% lower — a real gap, but noticeably smaller than Pexels' gap. Both video sets run 77–100 frames, so length isn't the explanation; the paper doesn't say why DAVIS is the tighter field, but one honest guess fits the data in front of you — DAVIS's eight objects (bear, camel, rhino, two horsejump clips, libby, cows, dog) are mostly large, rigid-ish animals, exactly the regime where Chapter 4 said higher t0 helps every method's causal or quasi-causal consistency, baselines included, narrowing the gap Lift4D otherwise opens on more deformable Pexels content.

Worth a small caution before you over-read that narrower CLIP gap: CLIP embeddings are semantic, not pixel-precise — they were trained to match an image to a caption, not to notice that a texture is slightly blurred or a color slightly off. A method can render something recognizably "rhino-shaped and rhino-colored" and score respectably on CLIP even if the fine detail is soft or wrong. That is exactly why the paper never leans on CLIP alone — on Consistent4D it always sits beside LPIPS and FVD, the two metrics that actually notice pixel-level and distributional error. EPE plays a similar grounding role in-the-wild, where CLIP is the only appearance metric available: it forces the comparison back onto something CLIP can't paper over — whether the tracked geometry is actually where it should be.

One honest gap in every table above: none of it reports variance across the individual videos in each benchmark — 7 for Consistent4D, 10 for Pexels, 8 for DAVIS. A single mean hides whether Lift4D wins by a little on every video, or by a lot on a few and barely at all on the rest. That per-video breakdown isn't in the numbers this chapter draws from, so it isn't claimed here either — a useful habit to carry into reading any comparison table, this one included.

A concrete illustration of why that gap matters: if six of Consistent4D's seven videos scored a clean 0.100 LPIPS and one hard case scored 0.212, the mean is (6×0.100 + 0.212) / 7 = 0.116 — the exact number reported. The same 0.116 could just as easily come from all seven videos clustering tightly around it. Both stories are consistent with the one number this chapter (and the paper) actually reports.

Chapter 1's crossing curves were drawn illustrative and unitless on purpose — a promise that real numbers would come later. This is that promise kept: L_SDS's outsized ablation cost is the appearance-prior-necessity curve, made concrete. The causal warm-start's smaller cost is the geometry-prior-reliability curve, staying comparatively flat because geometry was never the fragile half of this pipeline to begin with.

One question ties the whole scoreboard together, and it's the one the quiz below asks directly.

Chapter 10: Limits & Connections

You now know how Lift4D turns a single, uncalibrated camera into a complete, deformable 4D object. This closing chapter asks the question every honest teardown owes its reader: where does it actually stop working?

Every trick in this lesson had a specific job: the causal warm-start bought temporal consistency for free, sparse control nodes made large non-rigid motion cheap to express, the occlusion mask told the appearance phase exactly which pixels to trust, and freezing ψ kept a noisy generative prior from ever touching geometry. None of that makes the pipeline infallible. Here is exactly where each piece still breaks, and why nothing downstream can quietly fix an upstream mistake.

It helps to say plainly what kind of system you've actually spent nine chapters learning, because it explains why the failure modes below look the way they do. Lift4D trains nothing. Every number in Chapter 9's scoreboard came from optimizing a fresh set of parameters — canonical gaussian attributes, control-node weights, per-frame deltas — against three networks that never update, for one specific input video, then throwing that optimized state away and starting over for the next video. There is no checkpoint that gets better with experience. Every strength and every weakness in this chapter is a property of the ORCHESTRATION, not of anything the pipeline has learned from seeing many videos before this one.

Cascade error: click a stage, watch what it poisons

Start here, because this one interactive answers the "where would a mistake actually hide" question for the whole pipeline at once.

Lift4D is a strict pipeline, not a loop — nothing late in the chain ever feeds back and corrects something early. There's no mechanism anywhere in this system that looks at Phase B's rendered output and goes back to re-run Phase A, let alone re-run segmentation. Whatever each stage hands to the next is final. Pick an injection point below and watch the corruption spread. Notice which boxes it touches, and just as importantly, which boxes it never can.

Read the pattern, not just the individual cases: the earlier the injection point, the more boxes turn pink. A bad segmentation mask corrupts five downstream stages. A bad SDS prior corrupts exactly one — the final asset — because by the time Stable Zero123 runs, geometry is already frozen and cannot be re-poisoned. Late mistakes are contained. Early mistakes are not.

Put a number on that asymmetry: fixing a segmentation-stage bug protects five stages downstream of it (Depth, Causal Latent, Geometry, Appearance, and the Final Asset). Fixing an SDS-stage bug protects exactly one. That isn't a coincidence of how this particular pipeline happens to be drawn — it's the direct, structural consequence of the concept Chapter 1 built the whole lesson around: geometry gets decided early and frozen, so anything downstream of "geometry decided" inherits it whether it's right or wrong, while appearance gets decided last, so its mistakes have nothing left downstream to poison.

What actually breaks in practice

The cascade explains WHERE a corruption spreads once it starts. It doesn't explain what actually tends to trigger one in the first place. Four specific, concrete answers:

Three of these come straight from what the paper itself says, in its limitations discussion and its appendix's failure-case notes — kept short and honest rather than padded out. The fourth is this lesson's own inference, clearly marked as such below.

• Thin objects and high frame rates. The paper's own stated limitation: this whole cascade is controlled by a single hyperparameter, t0, and performance is inherently tied to Causal SAM3D's per-frame prediction quality — errors propagate without any oversight mechanism to catch them. In practice this bites hardest on thin structures (a bird's legs, an insect's antennae) and high-frame-rate video, where per-frame pose estimates can jump between neighboring frames by more than Phase A's Chamfer and regularization terms can absorb.
• The t0 tension has no free lunch. t0 too high suppresses legitimate deformation — the warm-start pulls so hard toward the previous frame's clean latent that genuine motion gets smoothed away. t0 too low fails to prevent geometric flickering — not enough of the previous frame survives to keep the shape consistent. Rigid objects tolerate higher t0 well; anything that deforms a lot needs it lower, and the paper offers no automatic way to pick it — it's set by hand, per video.
• Stable Zero123's prior is generic, not this object's. The paper doesn't spell this out directly, but it follows straight from Phase B's design: Zero123 was never fine-tuned on your specific rhino, or your specific anything. If the object's category sits far from what Zero123 has actually seen, the "plausible unseen texture" it hallucinates may just be wrong — confidently wrong, not uncertain-looking wrong. Chapter 8's containment argument still holds (it can't break the shape), but it can absolutely paint an implausible pattern on the far side of an unusual object.
• Nothing watches the watcher. The paper states this as its central limitation, almost verbatim: the cascaded setup is controlled by t0, and performance is inherently tied to the initial SAM3D prediction's quality, with errors propagating without oversight. That phrase is worth sitting with — there is no confidence score, no automatic detector anywhere in this pipeline that flags "this frame's reconstruction looks unreliable, tune t0 differently here." A human watching the comparison video is still the mechanism that catches a bad run.

Three frozen priors, three jobs, three honest limits. One more pass through all of them, side by side, before the connections — the same table you could sketch from memory if someone asked you to explain this paper in sixty seconds.

Where to go next on this site

Nothing in this lesson was invented from scratch — every mechanism is a specific, named borrow from a tool the site already teaches in full. Ten places to go deeper, matched to exactly where each one showed up above:

Symbol cheat sheet

Every symbol below traces back to a specific chapter you've already read, grouped loosely by which phase of the pipeline it belongs to rather than alphabetically — scan it top to bottom and it retraces most of the lesson.

Seventeen symbols, one table, for the review pass before you teach this back to someone else.

Ten chapters of mechanism end in one question of judgment: given everything above, where would YOU spend an engineering week to move a real number?