SAM 3D: 3Dfy Anything
in a Single Image

Chapter 1: The Key Insight

SAM 3D's breakthrough is not a clever new network layer. It is a recipe borrowed from LLMs: stop trying to find scarce real 3D data, and instead build a multi-stage training pipeline plus a self-improving data engine that manufactures it.

Two ideas that made the data engine possible

The LLM playbook, mapped to 3D

The payoff: a 5:1 human-preference win rate on real-world objects and 6:1 on full scenes over the previous best methods. Reframing 3D as an LLM-style data problem is the entire contribution.

Chapter 2: Flow Matching Background

Before the architecture, we need the engine that generates shapes. SAM 3D is a rectified flow matching model. If you've met diffusion, flow matching is its straighter, faster cousin — and you need it to understand how a voxel grid gets conjured from noise.

Manufacturing the need

We said reconstruction is sampling from a distribution. But how do you sample a complex object like "a 64³ voxel grid of a chair"? You can't just pick from a list. The trick of all modern generative models: learn to transport a simple distribution (Gaussian noise) into the complex one (real shapes).

Imagine noise at time t=0 and a real shape at t=1. Draw a straight line between a noise sample x₀ and a data sample x₁. A point on that line at time t is:

Here x₀ is pure Gaussian noise (the easy distribution), x₁ is a real shape latent (the hard distribution we want), and t ∈ [0,1] is a "how far along" knob — 0 means all noise, 1 means finished shape. Differentiate that line with respect to t and the velocity is gorgeously simple:

That constant x₁ − x₀ is the target velocity — the direction and speed you'd move to get from noise to data along the straight path. The network's only job is to predict that velocity at any (x_t, t), conditioned on the image. Call the network v_θ. The training loss is just regression:

Every symbol: E = average over random t and random noise/data pairs; v_θ = the network's predicted velocity; the squared term = how wrong that prediction is. Minimize it and the network learns the velocity field that flows noise to shapes.

Sampling: integrate the velocity

At inference you start from noise x₀ and walk forward in N steps of size Δt = 1/N, each step nudging x by the predicted velocity:

This is one function evaluation (NFE) per step. SAM 3D's base model uses 25 NFE; the distilled one uses 4. Play with the simulation below to see noise flow into a shape, and watch what too-few steps does to the result.

Chapter 3: The Architecture

SAM 3D is a two-stage cascade. A Geometry model first predicts a coarse shape and the object's layout. Its output is then handed to a Texture & Refinement model that adds high-resolution detail and color. Think: rough clay form first, then sculpt and paint.

The data flow, with actual shapes

Let's trace exactly what tensors move through the system — this is where "concept" becomes "realization".

The four conditioning token sets — and WHY four

SAM 3D feeds DINOv2 two pairs of (image, mask), giving four token sets. This is a deliberate engineering choice:

• Cropped object + its mask: a high-resolution, focused view of just the target. Detail.
• Full image + full mask: global scene context. This is where the "recognition" cue lives — seeing the whole kitchen tells you the half-occluded thing is a kettle, which tells you its 3D shape.

The optional pointmap: graceful degradation

SAM 3D can optionally take a coarse scene point map P (from an iPhone LiDAR or a monocular depth estimator). Here's the key failure-mode lesson: if no pointmap is provided, the model still predicts shape and texture fine — only metric layout (translation/scale) becomes less accurate, because absolute scale is genuinely unknowable from RGB alone. The model degrades gracefully rather than crashing.

Chapter 4: The Geometry Model & Mixture-of-Transformers

The Geometry model is the heart of the system. It is a 1.2B-parameter flow transformer that models p(O, R, t, s | I, M): coarse shape O (a 64³ voxel occupancy field), 6D rotation R, translation t, and scale s. It must generate two very different things — a big spatial voxel grid AND a tiny 12-number layout vector — in one model. How?

Mixture-of-Transformers (MoT): two streams, one attention

SAM 3D uses a Mixture-of-Transformers. Picture two parallel transformer "streams" — one specialized for geometry tokens, one for layout tokens. Each stream has its own weights (feed-forward, norms). But they share a single multi-modal self-attention layer where the two streams can read each other.

Notice two things in the sim. Low recognition strength (image barely conditions the flow) gives a blobby, generic shape — the model falls back to a prior, ignoring the photo. This is exactly what under-trained or no-context models do. Too few NFE leaves the shape noisy and the pose jittery — the integration hasn't converged. Both are real failure modes SAM 3D had to engineer around.

6D rotation: why not Euler angles or quaternions?

R is a 6D rotation representation (Zhou et al., 2019), not 3 Euler angles. Euler angles have discontinuities (gimbal lock) and the wraparound at 360° makes them a nightmare to regress — a network predicting 359° vs 1° gets a huge loss for a 2° error. The 6D form (two 3-vectors, then Gram-Schmidt orthogonalized into a rotation matrix) is continuous, so the loss surface is smooth and learnable. A small ablation in the paper confirms 6D beats alternatives for pose.

Chapter 5: Texture, Refinement & Decoding

The Geometry model gives a coarse 64³ voxel blob. That's the silhouette in 3D, but it's low-res and colorless. The Texture & Refinement model — a 600M-parameter sparse latent flow transformer — fixes both.

Why "sparse"? A crucial efficiency decision

A 64³ grid has 262,144 voxels. But most of them are empty — an object only occupies a thin shell of that cube. Running a dense transformer over all 262K positions would be enormously wasteful.

Two decoders, one latent space

The refined latent can be decoded two ways by a pair of VAE decoders that share the same encoder and latent space:

• D_m → mesh: watertight triangle mesh, good for game engines, 3D printing, editing.
• D_g → 3D Gaussian splats: point-cloud-like primitives, great for fast photorealistic rendering.

What degrades, and why

Texture quality depends on visible surface. The back of an occluded object is hallucinated from the learned prior — plausible, but not "true". The paper notes common failure modes the post-training stage specifically hunts down: floaters (stray disconnected voxels), bottomless meshes (no base), and broken symmetry. These are exactly the artifacts flow-matching's per-voxel objective doesn't penalize — which motivates the next two chapters.

Chapter 6: The Data Engine (the real contribution)

Architecture aside, the paper's genuine innovation is the model-in-the-loop (MITL) data engine — the flywheel that manufactures real-world 3D training data. This is the part you'd present in an interview as "what's actually new here".

The cold-start problem

On iteration zero, the model is synthetic-only and produces few good real-world meshes. So how do you bootstrap? An ensemble. Early on, candidate meshes come mostly from a suite of existing learned + retrieval-based models. As the model improves, it dominates — eventually producing ~80% of the annotated data. The crutch is kicked away once the model can walk.

The annotation pipeline: humans select, don't author

Best-of-N: the math of why selection works

Why ask for N=8 candidates instead of 1? Suppose any single model sample is "good enough" (r > α) with probability p. The probability that the best of N is good enough is:

Worked example: say p = 0.30 (a lone sample is good 30% of the time). With N=1 you get 30%. With N=8: P = 1 − (0.70)⁸ = 1 − 0.0576 = 94.2%. By spending 8 samples and one cheap human pick, the success rate jumps from 30% to 94%. SAM 3D further boosts effective N by pre-filtering candidates with a model, then having the human filter — a two-stage funnel.

Chapter 7: Multi-Stage Post-Training (SFT + DPO)

The base model has only ever seen (semi-)synthetic data. The domain gap to real photos is large. Post-training closes it, in a careful order.

SFT: noisy-then-clean curriculum

Supervised fine-tuning starts with the larger, noisier non-expert labels (MITL-3DO) to broadly adapt the model to real images, then finishes on the small, high-quality artist set (Art-3DO). Order matters: the artist data is precious, so you use it last to polish — it aligns outputs with artists' aesthetic sense and specifically suppresses floaters, bottomless meshes, and broken symmetry.

Why SFT alone isn't enough → DPO

Flow matching's objective is a per-voxel reconstruction loss. It has no notion of "humans find symmetric, closed shapes more pleasing". You can have low flow-matching loss and still output an ugly, asymmetric, hole-riddled mesh. The objective and human preference are misaligned.

So after SFT comes Direct Preference Optimization (DPO), borrowed straight from LLM alignment. Recall the D⁺/D⁻ pairs the data engine produced for free: D⁺ is the human-chosen shape, D⁻ is a rejected candidate. DPO nudges the model to raise the likelihood of preferred outputs and lower the rejected ones — without needing an explicit reward model.

Symbols: q_θ = model being trained; q_ref = frozen reference (the SFT model); σ = sigmoid; β = a temperature controlling how hard we push. The bracket is "how much more does our model prefer the chosen over the rejected, relative to the reference". Maximizing it = learning human aesthetic preferences. The paper finds this off-policy preference data is effective at eliminating undesirable outputs even after Art-3DO SFT.

The cascade ablation — every stage earns its keep

Table 4 of the paper shows near-monotonic improvement as each stage is stacked. Watch the metric climb:

Chapter 8: Distillation to Sub-Second Inference

The aligned Geometry model is great but slow: 25 function evaluations per reconstruction. For an interactive demo ("convert image to 3D" in a browser) you want sub-second. The final stage cuts NFE from 25 → 4.

Shortcut models: learn to take big steps

SAM 3D adapts the shortcut model idea (Frans et al., 2024). A normal flow model only knows how to take an infinitesimal step — that's why you need many. A shortcut model is additionally conditioned on the step size d, and trained so that one big step of size d lands in the same place as two small steps of size d/2 would. This self-consistency property lets it leap.

Where x' is where the first half-step lands. Read it as: "your big-step velocity should equal the average of two consistent half-steps." Train this and the model can take 4 giant strides instead of 25 baby ones — because the straight rectified-flow path makes large steps accurate in the first place (recall Chapter 2).

Chapter 9: Experiments & The Core Insight in Code

The headline: on the new SA-3DAO benchmark (1,000 artist-made meshes from real images), SAM 3D dominates. It's not mainly about clean synthetic objects — it's about messy real ones.

Reading the numbers

Note the honest detail in the paper: on the isolated synthetic ISO3D set, SAM 3D merely matches prior work on perceptual metrics. The win is entirely on hard, real, occluded inputs — which is exactly what the data engine was built for. The method does what it was designed to do, and not more — a credibility signal.

The core insight as runnable code

Strip away the 1.2B params and the engineering. The beating heart is: train a velocity field that flows image-conditioned noise into shape latents, and sample by integrating it. Here is that core, runnable:

import torch, torch.nn as nn

class VelocityField(nn.Module):
    """Predicts dx/dt = x1 - x0, conditioned on image features + time."""
    def __init__(self, latent_dim, cond_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim + cond_dim + 1, 512), nn.SiLU(),
            nn.Linear(512, 512), nn.SiLU(),
            nn.Linear(512, latent_dim),   # outputs a velocity, same shape as latent
        )
    def forward(self, x_t, t, cond):           # cond = DINOv2 image+mask tokens
        t = t.view(-1, 1)
        return self.net(torch.cat([x_t, cond, t], dim=-1))

def flow_matching_loss(model, x1, cond):
    """Rectified flow: regress the straight-line velocity x1 - x0."""
    x0 = torch.randn_like(x1)                  # noise (easy distribution)
    t  = torch.rand(x1.size(0), 1)             # random point along the path
    x_t = (1 - t) * x0 + t * x1                # point on the straight line
    target_v = x1 - x0                         # constant target velocity
    pred_v = model(x_t, t, cond)
    return ((pred_v - target_v) ** 2).mean()

@torch.no_grad()
def sample(model, cond, latent_dim, nfe=25):
    """Integrate noise -> shape latent. nfe=25 base, nfe=4 distilled."""
    x = torch.randn(cond.size(0), latent_dim)  # start at noise (t=0)
    dt = 1.0 / nfe
    for i in range(nfe):
        t = torch.full((cond.size(0), 1), i * dt)
        x = x + dt * model(x, t, cond)         # Euler step along velocity
    return x                                    # -> decode with VAE to mesh/splats

That's the whole idea in 30 lines. SAM 3D scales the network to 1.2B params, makes cond the four DINOv2 token sets, splits the latent into geometry+layout via MoT, and wraps it all in the data-engine flywheel. But this is the engine.

Chapter 10: Connections & Cheat Sheet

SAM 3D sits at the crossroads of generative modeling, the Segment Anything lineage, and the LLM training playbook.

The cheat sheet — every key symbol

When to reach for what

• Rectified flow over diffusion: when you'll want few-step / distilled sampling later. Straight paths distill cleanly.
• MoT over one transformer: when generating multiple modalities with different statistics (here: spatial voxels + low-dim pose) that still must inform each other.
• Sparse latent flow: when the signal occupies a thin shell of a large grid (3D shells, point clouds).
• MITL data engine: when authoring labels is expensive but selecting among proposals is cheap. Generalizes far beyond 3D.
• SFT then DPO: when your training loss can't express the quality you actually care about (aesthetics, symmetry, safety).

What the paper doesn't solve

• Absolute metric scale from RGB alone is unknowable — needs an optional pointmap. Without it, layout degrades.
• Occluded back surfaces are hallucinated from the prior, not measured. Plausible ≠ correct.
• Multi-object physical plausibility (objects interpenetrating, floating) is improved but not guaranteed.
• The flywheel needs an ensemble cold-start — you can't start from nothing; you need existing models to bootstrap candidates.