ZHANG, LE MOING, KOPPULA, ROCCO … ZISSERMAN, SAJJADI (GOOGLE DEEPMIND) · arXiv 2025

D4RT: Reconstructing Dynamic Scenes One Query at a Time

Stop asking "what is the geometry of everything, everywhere, all at once?" Encode the video once, then simply ask the model where any point goes — at any time, from any camera. One interface gives you depth, point tracks, point clouds, and camera parameters. 200+ FPS, 9–100× faster than the field.

Prerequisites: a transformer has attention + a camera turns 3D points into pixels. We build the rest — queries, depth, pose, tracking — from zero.

Chapters

Simulations

Interface, 5 tasks

Chapter 0: Everything, Everywhere, All at Once — Is the Wrong Question

Point a camera at a street: a swan glides across a pond, a train rolls behind it, the camera itself drifts. Now ask the computer-vision question: reconstruct this. Recover the 3D shape of every surface, the motion of every moving thing, and where the camera was at every instant. This is 4D reconstruction — three dimensions of space plus one of time.

For decades the field answered with a single rigid ambition: compute the geometry of everything, everywhere, all at once. Build one dense 3D model of the whole scene. It sounds noble. It is also, the D4RT authors argue, fundamentally ill-suited to a world that moves.

Why? Because "the geometry of everything" assumes there is one geometry. In a dynamic scene there isn't. The swan is in one place at frame 1 and another at frame 30. A single point cloud either smears it across the pond or picks one moment and ignores the rest. So the field fractured the problem into specialists:

MegaSaM bolts together a mosaic of off-the-shelf models — one for relative depth, one for metric depth, one for motion segmentation — then runs slow test-time optimization to force the pieces to agree.
VGGT is a single feedforward network, but it grows a separate decoder head for each output: a depth head, a pose head, a point head. And it cannot put a moving point in correspondence with itself across time.
SpatialTrackerV2 handles motion, but only through costly multi-stage iterative refinement, and only tracks points seeded from one frame.

The pattern of failure. Every prior approach either (a) fuses many models with expensive optimization, (b) splits one model into task-specific heads, or (c) iterates slowly. None offers a single, flexible way to ask: where is this particular point, at this particular time, seen from this particular camera? That missing question is the whole paper.

Below: the old way computes a dense answer for every pixel of every frame, even the parts you never asked about. Watch how much of that grid is wasted work when all you wanted was a few points.

Dense decoding spends compute on — cells per frame. Most are never used downstream.

Why is "the geometry of everything, everywhere, all at once" a poor fit for dynamic scenes?

Because dense 3D needs too much GPU memory. Because a moving point has different geometry at different times, so a single static reconstruction either smears it or drops it — there is no one geometry to compute. Because cameras are always moving and can't be calibrated.

Chapter 1: Ask, Don't Compute

Here is the reframing that makes D4RT work. Instead of computing a giant structured output (a dense per-frame point map), the model becomes something you query. You hand it one small question and it returns one small answer:

The query. "Take the 2D point at pixel (u, v) in frame t_src. Tell me its 3D position at time t_tgt, expressed in the coordinate system of the camera at frame t_cam." The answer is a single 3D point P ∈ R³.

That's it. Five numbers in, three numbers out. The genius is in what those five numbers decouple. Notice that t_tgt (which moment in time we want the point's position) and t_cam (whose camera frame we want it expressed in) are independent indices. They need not be equal.

This is a full disentanglement of space and time. "Where was this point at frame 30, as seen from the camera at frame 1?" is a perfectly legal question. Prior methods couldn't even phrase it — they baked "time" and "camera" together, locking you to "where things are now, from where the camera is now."

And because each question is answered independently, you can ask one (a single point's depth) or a million (every pixel, every frame) with the same machinery. Sparse when you want speed, dense when you want completeness. The model never has to commit to one output shape.

Why this is the key insight. A query interface turns one model into every 4D tool at once. We will see in Chapter 5 that choosing which queries to ask — sweeping t_tgt, or sweeping (u,v), or fixing them all equal — is exactly what selects between point tracking, point clouds, and depth. The model doesn't change. Only the question does.

Hold onto a mental image: the video gets read once into a fixed memory, and then a tiny, fast reader pokes that memory with questions. Reading is expensive and happens once; poking is cheap and happens as often as you like. That asymmetry is the source of every speed number in the paper.

What does it mean that t_tgt and t_cam are separate indices in the query?

The model runs twice, once per index. They must always be equal for the math to work. You can ask for a point's position at one time but expressed in a different camera's frame — space (which camera) and time (which moment) are disentangled.

Chapter 2: Where This Comes From

D4RT didn't appear from nowhere. It's the dynamic, query-based descendant of two converging lines of work. Knowing them tells you exactly which problems D4RT inherited and which it fixed.

Line one — feedforward 3D reconstruction. The classical pipeline (COLMAP: Structure-from-Motion + Multi-View Stereo) recovers geometry by carefully matching features across views and solving for cameras and points. It is interpretable and accurate — and brittle and slow. The breakthrough was DUSt3R: a transformer that takes two uncalibrated images and directly regresses a 3D point map, no explicit matching needed. VGGT scaled this from pairs to whole sequences with a Vision Transformer and global attention.

But these models share limits D4RT calls out by name: many can't handle changing camera intrinsics within a video; several can only use the first frame as the camera reference; and crucially, they grow separate decoder heads per task (depth, pose, points) or stitch in separate models for sub-tasks. Worst of all, none gives correspondences for the dynamic parts of a scene.

Line two — point tracking, 2D to 3D. Tracking asks a different question: follow this point through occlusion and non-rigid motion. The field went from sparse tracks (Particle Video) to dense "track every pixel" methods, then lifted 2D tracks into 3D using depth and intrinsics. Recent models (SpatialTracker, St4RTrack, DPM, L4P) predict 3D tracks directly — but St4RTrack and DPM are stuck in DUSt3R's pairwise paradigm; SpatialTrackerV2 is multi-stage and slow; L4P needs different heads for sparse vs dense outputs of the same quantity.

The gap D4RT fills. Reconstruction methods give you geometry but not dynamic correspondence. Tracking methods give you dynamic correspondence but not a holistic scene. Both proliferate task-specific heads and reference-frame restrictions. D4RT's bet: a single query interface can deliver reconstruction and tracking, static and dynamic, sparse and dense, with one lightweight decoder and no per-task heads.

The architectural seed is the Scene Representation Transformer (SRT): encode a set of views into a single latent "scene representation," then render novel views by querying it. D4RT keeps SRT's encode-once-query-many spirit but swaps "render a pixel color" for "return a 3D position," and adds time.

What capability did prior feedforward reconstructors (DUSt3R, VGGT) lack that D4RT targets?

They couldn't run on a GPU. Correspondence for dynamic (moving) regions, plus they used separate per-task decoder heads. They couldn't estimate depth at all.

Chapter 3: One Query, Five Tasks

Let's make the query concrete and watch it become every 4D task. The query is the tuple

q = ( u, v, t_src, t_tgt, t_cam ) → P = D(q, F) ∈ R³

where (u,v)∈[0,1]² are normalized pixel coordinates in the source frame t_src, and t_tgt, t_cam ∈ {1…T} pick the target time and the camera reference. F is the encoded video (next chapter); D is the decoder.

Now the magic. By sweeping different fields over their ranges and fixing others, the same decoder produces wildly different structured outputs. This is the paper's Table 1, and it's worth internalizing as a recipe:

Task	u	v	t_src	t_tgt	t_cam
Point track	fixed	fixed	fixed	1…T	1…T
Point cloud	1…W	1…H	fixed	fixed	fixed
Depth map	1…W	1…H	t	=t_src	=t_src
Extrinsics	grid	grid	fixed	fixed	1…T
Intrinsics	grid	grid	1…T	fixed	fixed

Read the first row: fix a single pixel and its source frame, then sweep target time from 1 to T with the camera following along — you get that point's 3D trajectory, a point track. Read the second: sweep every pixel at one fixed camera reference — a full point cloud in one coordinate system, with no error-prone stitching of per-frame estimates. Read the third: ask each pixel for its own position in its own frame and keep only the Z value — a depth map.

Build a query below and watch which task it spells. Move the sliders; toggle whether a field is fixed or swept.

source pixel u (sweep all?)fixed

target time t_tgt (sweep 1…T?)sweep

camera t_cam (sweep 1…T?)sweep

This query produces: Point track

The insight, restated. There is no "depth network" and no "tracking network." There is one decoder and a language of queries. The tasks are not built in — they emerge from how you sample the query space.

To get a single point's 3D trajectory (a point track), which fields do you sweep?

Sweep (u,v) over all pixels; fix the times. Fix (u,v,t_src); sweep t_tgt=t_cam over 1…T. Fix everything and run the decoder once.

Chapter 4: Encode Once, Query Many

The architecture has exactly two parts, and the split between them is the efficiency story. Trace the data flow.

The encoder E. Input: a video V ∈ R^T×H×W×3 — T frames of RGB. It is tokenized with a spatio-temporal patch (the paper uses 2×16×16 — 2 frames deep, 16×16 pixels), then run through a Vision Transformer (ViT-g, 40 layers, ~1B parameters) with interleaved attention: local frame-wise self-attention (cheap, within a frame) alternating with global self-attention (expensive, across all frames and time). The local layers find structure within frames; the global layers establish dense correspondence across frames and reason about how time changes the scene.

Output: the Global Scene Representation F = E(V) ∈ R^N×C — N latent tokens, each a C-dimensional vector. This is the "memory." It is computed once. After that it never changes.

Aspect ratio, handled honestly. To support any aspect ratio, the video is resized to a fixed square before tokenizing — which would destroy the true shape. So the original aspect ratio is embedded into a separate token and fed to the transformer alongside the video tokens. The model learns to undo the squashing. A small decision that tells you the authors sweat the data-flow details.

The decoder D. A small (8-layer, ~144M parameter) cross-attention transformer — an order of magnitude lighter than the encoder. A query token is built by adding together: the Fourier-feature embedding of the continuous coordinates (u,v), the learned discrete embeddings of the three timesteps t_src, t_tgt, t_cam, and — this matters a lot — an embedding of the local 9×9 RGB patch centered at (u,v). That patch gives the query crucial local appearance context; the ablation shows it "dramatically improves performance."

Each query token cross-attends into F — it reads the fixed memory — and the result is mapped by a simple learned linear projection to the 3D point P. Watch the flow:

Encoder runs once (heavy). Decoder runs per query (light) — and all queries run in parallel.

Queries do NOT talk to each other. A deliberate, slightly surprising choice: there is no self-attention between queries. Each is decoded in total isolation. Why? Two reasons. (1) Training efficiency: you only need to decode a handful of queries to get a supervision signal, instead of a full dense map. (2) Robustness: the authors found that letting queries attend to each other caused major performance drops — correlated queries push the model out of distribution. Independence also gives trivial parallelism: a thousand queries are a thousand independent forward passes through a tiny network.

Why is computing F once (and reusing it) the source of D4RT's speed?

Because F is stored on disk. Because the decoder is also 1B parameters. The expensive 1B encoder runs once per video; every additional answer is just a tiny 144M decoder pass into the fixed F, and those passes are independent and parallel.

Chapter 5: The Decoder Interface (Showcase)

This is the paper's core figure, made playable. On the left is a tiny synthetic "video": a static ground plane (grey grid) with one moving object (the orange ball) arcing across it while the camera gently pans. On the right is what the decoder returns when you query it.

Pick a source point by clicking anywhere in the left frame. Then sweep t_tgt (which moment) and t_cam (whose camera), and switch the query mode to see the same interface become three different tools on the same scene.

the video — click to pick (u,v)

what the decoder returns (3D, top-down)

target time t_tgt6

camera frame t_cam0

Feel the disentanglement. In Track mode, set t_cam to a different value than t_tgt. The trajectory is the same physical motion but rigidly re-expressed in another camera's coordinates — the ground stays put while the whole track rotates/translates. That is "where was this point at time t_tgt, seen from camera t_cam" — a question only a disentangled interface can ask.

What's actually happening per mode (each is just a query pattern from Chapter 3's table):

Track: your one picked pixel, swept over all target times — the ball's points arc through space; a ground point sits still.
Point cloud: every pixel of the frame, all expressed in camera t_cam's frame at once — the whole scene in one coordinate system, no stitching.
Depth: every pixel asked for its own position, keep Z, color by distance — near is warm, far is cool.

In the showcase, switching from Track to Point-cloud mode changes only…

the model weights. the encoder. which queries we ask (sweep time for one pixel vs. sweep all pixels at one camera) — the decoder and F are identical.

Chapter 6: Getting Cameras Out of 3D Points

The query returns 3D point positions. But two of our five tasks — extrinsics (where the camera is) and intrinsics (the camera's focal length) — are camera parameters. The elegant part: D4RT never has a "camera head." It derives cameras from queried 3D points with classical geometry. The network gives points; arithmetic gives cameras.

Extrinsics via Umeyama. To get the relative pose between frames i and j, sample a grid of points and decode each one twice: once expressed in camera i's frame, once in camera j's frame. These are the same physical 3D points written in two coordinate systems. The transform between two such sets is a pure rigid motion (rotation + translation), and Umeyama's algorithm recovers it from a single 3×3 SVD. No optimization, no iteration.

Intrinsics via the pinhole model. Decode a grid of points in their own camera frame, giving 3D positions P=(p_x, p_y, p_z). A pinhole camera with principal point at the image center (0.5, 0.5) projects a 3D point to pixel (u,v) by u = 0.5 + f_x·p_x/p_z. Invert it for the focal length:

f_x = p_z(u−0.5)/p_x , f_y = p_z(v−0.5)/p_y

You get one focal-length estimate per grid point. Real predictions are noisy, so D4RT takes the median over the grid — robust to the occasional bad point. (Distorted lenses like fisheye just add a non-linear refinement on top.)

Work it yourself. Set a 3D point and the pixel it landed on; the focal length falls out. Drag the point and watch how a single estimate jitters — then see why the median over many points is the trustworthy answer.

pixel u0.75

point depth p_z5.0

point p_x1.25

Single estimate: f_x = — | Median over the grid: —

Why this design is beautiful. Cameras are not a separate prediction the network must learn — they are a consequence of getting points right. One well-trained point predictor, plus a few lines of linear algebra, yields depth, tracks, point clouds, and full calibration. Fewer heads to train, fewer ways to disagree.

How does D4RT get camera extrinsics without a pose head?

It runs gradient descent at test time. It decodes the same 3D points in two camera frames and recovers the rigid transform between them with Umeyama (a 3×3 SVD). It reads the camera from the video metadata.

Chapter 7: What the Model Learns From

Training is end-to-end. At each step, the model samples a batch of N queries (the paper uses 2048 per clip, oversampled on interesting regions for efficiency) and minimizes a weighted sum of losses. The primary signal is an L1 loss on the predicted 3D position. But two preprocessing steps on that loss reveal deep engineering wisdom.

Step 1 — normalize by mean depth. Both predicted and ground-truth point sets are divided by their respective mean depths. Why? Monocular 3D is only knowable up to scale — a small close scene and a huge far one can produce identical pixels. Normalizing removes the unknowable global scale so the loss measures shape, not an arbitrary size the model could never recover.

Step 2 — the sign(x)·log(1+|x|) transform. Far-away points have enormous coordinates; a raw L1 loss would let a single distant background point dominate the gradient and drown out the nearby foreground you actually care about. This log-style transform compresses large magnitudes while staying linear near zero — it dampens the influence of far points without ignoring them. Drag the far point below and watch a raw error of 40 shrink to a tame ~3.7 under the transform.

a point's coordinate x40

raw |x| = — → transformed = —

The auxiliary heads. On top of the decoder output, a few cheap linear projections add extra supervision — each is a different "view" of the same point that sharpens the representation:

L1 on 2D coords the point's location in image space — anchors the 3D answer to where it actually appears.
cosine on normals the 3D surface orientation — teaches local geometry, not just position.
BCE on visibility is this point even visible in the target frame? — essential for honest tracking through occlusion.
L1 on motion the point's displacement vector — directly supervises dynamics.
−log(c) confidence the model can down-weight points it's unsure about, and is penalized for low confidence.

Losses are only applied where ground truth exists. The training mixture spans 11 datasets — synthetic (Kubric, VirtualKITTI, PointOdyssey) and real (ScanNet, Waymo, Co3D) — and not all carry every label. Each loss term switches on only when its supervision is available. The whole thing trains in just over 2 days on 64 TPU chips, 500k steps, batch size 1 per chip, 48-frame clips at 256×256.

Why pass the 3D error through sign(x)·log(1+|x|) before the L1 loss?

To make the loss differentiable. To convert meters to pixels. To compress huge coordinates of far-away points so a single distant point can't dominate the gradient over the nearby foreground.

Chapter 8: Tracking Every Pixel, Cheaply

D4RT's lightweight, independent decoder unlocks something prior methods can't do well: track all pixels for a complete, hole-free 4D reconstruction. The naive way is brutal — tracking every pixel of every frame is O(T²HW) queries, and most are redundant: if a track already passed visibly through a pixel, you don't need to start a fresh track there.

Algorithm 1 exploits that redundancy with an occupancy grid G ∈ {0,1}^T×H×W. Start a track only from an unvisited pixel; as that full-video track sweeps through space-time, mark every pixel it visibly passes as visited; repeat until the grid is full. One track covers many cells, so you launch far fewer than the naive count.

Run it below. Each new track (a colored streak) marks the cells it crosses. Watch coverage race to 100% with a fraction of the naive queries — the paper measures a 5–15× speedup depending on how much the scene moves.

motion complexitymedium

coverage 0% | tracks launched 0 | vs naive —

Why only D4RT can do this. The algorithm is "feasible precisely because our decoder is both sparse and lightweight." Pure reconstructors (MegaSaM, π³) give no dynamic correspondence at all. Dense frame-level decoders (St4RTrack) are stuck in the naive O(T²HW) regime. Heavy sparse decoders (SpatialTrackerV2) pay too much per query to ever scale to all pixels. D4RT's cheap-per-query design is the enabling condition.

What does the occupancy grid in Algorithm 1 save you from?

Re-encoding the video for each track. Launching redundant tracks from pixels an existing track already visibly covered (the O(T²HW) naive cost). Storing the video in memory.

Chapter 9: The Receipts

The pitch is "unified and fast and accurate." Papers say that all the time; D4RT backs it with two kinds of numbers: a speed–accuracy frontier for camera pose, and raw 3D-tracking throughput.

Pose: accuracy vs. speed. On Sintel + ScanNet, accuracy is 1 − error averaged over three pose metrics; throughput is frames per second on an A100. D4RT sits at the top-right — most accurate and fastest: 200+ FPS, 9× faster than VGGT and 100× faster than MegaSaM, while beating both on accuracy. Toggle the methods and see the Pareto frontier; D4RT dominates the corner everyone wants.

← slower · faster → (FPS, log scale) | ↑ more accurate pose

3D tracking throughput. How many full-video tracks can each model produce while holding a target frame rate? At a 1-FPS budget D4RT yields ~40,180 tracks vs SpatialTrackerV2's 2,290 and DELTA's 5,770 — and at a demanding 60-FPS budget D4RT still does 550 while SpatialTrackerV2 manages 29. Overall 18–300× faster. (Each D4RT "track" is T independent decoder queries — the parallelism from Chapter 4 cashing out.)

On the accuracy side, D4RT sets a new state of the art on TAPVid-3D (DriveTrack, ADT, PStudio) across the headline tracking metrics, both with and without ground-truth intrinsics, and produces cleaner full-scene reconstructions than reconstruction-only methods (which smear or drop moving objects) and tracking-only methods (which leave holes behind moving things).

The honest caveat. The two ablations the paper foregrounds tell you where the performance lives: (1) the local 9×9 RGB patch in the query "dramatically improves" results — raw coordinates aren't enough, the decoder needs local appearance; and (2) adding self-attention between queries hurts — independence isn't just for speed, it's for accuracy. Remove either and the headline numbers soften.

D4RT's 18–300× tracking-throughput advantage comes mainly from…

a bigger model than the baselines. encoding once, then answering each track as T tiny, independent, parallel decoder queries — no iterative refinement. running at lower resolution than everyone else.

Chapter 10: Connections & Cheat Sheet

D4RT is one move applied ruthlessly: turn a structured-prediction problem into a query-answering problem. That move shows up across modern ML — NeRF queries a scene with a ray, SRT queries it for a pixel color, DETR queries an image with object slots, and now D4RT queries a video for a point's 4D position. When your output space is huge and variable-shaped, don't predict it all; build an interface to ask for any slice of it.

Where to go next on this site: the attention and transformer gleams (the encoder/decoder machinery), any depth-estimation or camera-geometry lesson (the pinhole and Umeyama steps), and the Reverbs/manifold material on coordinate frames if you want to feel rigid transforms in your hands.

The cheat sheet.

Query: q = (u, v, t_src, t_tgt, t_cam) → P = D(q, F) ∈ R³. (u,v): source pixel; t_src: source frame; t_tgt: target time; t_cam: camera reference. t_tgt ≠ t_cam allowed → space/time disentangled.
Encoder: F = E(V) ∈ R^N×C, computed ONCE. ViT-g, interleaved local+global attention, ~1B params.
Decoder: 8-layer cross-attention, ~144M. Query = Fourier(u,v) + embed(t_src,t_tgt,t_cam) + 9×9 patch. No self-attention between queries.
Tasks (Table 1): track = fix pixel, sweep t_tgt=t_cam. cloud = sweep pixels, fix t_cam. depth = t_src=t_tgt=t_cam, keep Z.
Cameras from points: extrinsics = Umeyama (3×3 SVD) on same points in two frames; intrinsics = f_x = p_z(u−0.5)/p_x, median over a grid.
Loss: L1 on P, normalized by mean depth, through sign(x)·log(1+|x|); aux = 2D L1 + normal cosine + visibility BCE + motion L1 + confidence.
Dense tracking: Alg 1 occupancy grid → 5–15× speedup; only start tracks from unvisited pixels.
Numbers: 200+ FPS pose (9× VGGT, 100× MegaSaM); 18–300× tracking throughput; SOTA on TAPVid-3D; trained ~2 days on 64 TPU.

The mastery test. You should now be able to: write the query tuple and say what each field means; explain why encode-once/query-many is fast; reproduce Table 1's recipe for track/cloud/depth from memory; derive the focal-length formula from the pinhole model and explain the median; explain why the loss normalizes by depth and log-compresses; and explain why queries are kept independent (speed and accuracy). If you can teach those back, you own D4RT.

The paper's one-sentence soul: a dynamic scene isn't a thing to compute — it's a thing to interrogate.