PoseDiffusion: Solving Pose Estimation via Diffusion

Chapter 1: The Key Insight

Here's the idea that changes everything: what if camera pose estimation was just denoising?

Think about what a diffusion model does for images. You start with pure noise and gradually refine it into a coherent picture. At each step, the model nudges the noisy image slightly closer to something that looks real. The final result emerges through many small corrections, not one giant leap.

Now think about what Bundle Adjustment does. You start with rough camera poses (from a noisy initialization) and iteratively refine them until all the cameras are geometrically consistent. Each iteration makes small adjustments. The final result emerges through many corrections.

The parallel is striking. Both processes are iterative refinements from noise to signal. PoseDiffusion makes this parallel literal.

The model learns the conditional distribution p(x|I) — the probability of camera parameters x given images I. At test time, sampling from this distribution produces camera poses. Because the distribution is assumed to be near-delta (i.e., there's essentially one right answer for a given set of images), any sample is a valid pose estimate.

The inference pipeline, end to end

Chapter 2: Background

Before we dive into PoseDiffusion's mechanics, let's nail down the three pieces of geometry it relies on.

Camera Extrinsics

A camera's extrinsic parameters describe where it is and which way it's pointing. Formally, extrinsics g = (R, t) consist of a rotation matrix R in SO(3) and a translation vector t in R³. Together, they define a rigid-body transformation from world coordinates to camera coordinates:

PoseDiffusion represents the rotation as a unit quaternion q in H (4 numbers) and keeps the translation as a 3-vector, giving 7 numbers per camera for extrinsics.

Camera Intrinsics

The intrinsic parameters describe how the camera projects 3D points onto its 2D sensor. The calibration matrix K maps a 3D camera-space point to a 2D pixel:

PoseDiffusion simplifies this to one degree of freedom: the focal length f. The principal point (p_x, p_y) is fixed at the image center, which is standard in SfM. The focal length is predicted as f = exp(f̂) to guarantee it's always positive. This adds 1 number per camera, for a total of 8 parameters per camera.

Epipolar Geometry

Given two cameras with known poses, there's a fundamental relationship between corresponding points. If a point p₁ in image 1 corresponds to a point p₂ in image 2, then the epipolar constraint says:

where F is the Fundamental Matrix computed from the two cameras' poses, and p̃ denotes homogeneous coordinates. This says: if you know one point in image 1, the corresponding point in image 2 must lie on a specific line (the epipolar line). If the poses are correct, all correspondences satisfy this constraint. If they don't, the poses are wrong.

Chapter 3: Diffusion for Poses

PoseDiffusion adapts DDPM (Denoising Diffusion Probabilistic Models) to the domain of camera parameters. Let's trace how the standard DDPM machinery maps onto this problem.

The Forward Process: Adding Noise to Cameras

Start with ground-truth camera parameters x₀. The forward process adds Gaussian noise over T steps:

After T steps, the cameras are indistinguishable from pure Gaussian noise. The variance schedule β₁, ..., β_T controls how fast this happens. PoseDiffusion uses T = 100 steps with a linear schedule from 10^-3 to 0.2.

A convenient closed-form lets us jump directly to any timestep:

where ᾱ_t = ∏_i=1^t (1 - β_i). This is essential for efficient training: we can sample any noise level without simulating the full chain.

The Reverse Process: Denoising Cameras

The reverse process is what we actually use at inference. Starting from pure noise x_T ~ N(0, I), we iteratively denoise:

The denoiser D_θ takes the current noisy cameras x_t, the timestep t, and image features I, and predicts the clean cameras. PoseDiffusion uses the "x₀ prediction" formulation: the network directly predicts the clean signal rather than the noise. This is empirically more stable for camera parameters.

What Makes This Different from Image Diffusion?

In image diffusion, each sample is a grid of pixels — high-dimensional, spatially structured. Here, each sample is a set of camera parameter vectors. There are key differences:

• Variable-length sets: PoseDiffusion handles an arbitrary number N of cameras. The denoiser processes them jointly through a transformer.
• Low dimension per camera: Just 8 numbers per camera, versus millions of pixels. But these 8 numbers must be geometrically precise.
• Conditional on images: The denoising is conditioned on image features, so it's really modeling p(x|I), not p(x).
• SE(3) structure: Camera parameters live on a manifold (rotations are in SO(3)), not in flat Euclidean space. PoseDiffusion handles this by working with quaternions and letting the noise schedule operate in the ambient R⁸ space.

Concrete diffusion setup

• Total diffusion steps T: 100 (much fewer than the 1000 typical for image diffusion — pose space is low-dimensional and doesn't need as many refinement steps)
• Noise schedule: Linear β from 10^-3 to 0.2. This is aggressive — by step 100, the signal-to-noise ratio is very low, ensuring x₁₀₀ is nearly pure Gaussian.
• Input dimension: 8 per camera × N cameras. For N=10, the total diffusion operates over an 80-dimensional space. Compare to image diffusion over a 32×32×4 = 4096-dimensional space. PoseDiffusion's space is 50x smaller.
• Quaternion normalization: After each denoising step, the quaternion portion of each camera's parameters is L2-normalized to unit length. This enforces the SO(3) constraint directly in the diffusion process.

Chapter 4: Geometry-Guided Sampling

The diffusion denoiser alone can produce reasonable poses, but it's a feed-forward neural network — and neural networks are notoriously bad at regressing precise geometric quantities like rotation angles and translation vectors. PoseDiffusion's secret weapon is Geometry-Guided Sampling (GGS): injecting classical epipolar constraints directly into the diffusion sampling process.

The Mechanism: Classifier Guidance for Geometry

Recall that in classifier-guided diffusion for images, you steer samples toward a desired class by adding the gradient of a classifier to the denoising step. PoseDiffusion does the same thing, but the "classifier" is replaced by an epipolar geometry likelihood.

At each denoising step t, the predicted mean μ_t-1 is adjusted:

The gradient ∇ log p(I|x) pushes the poses toward satisfying epipolar constraints from 2D correspondences. The scalar s controls guidance strength.

The Sampson Likelihood

The likelihood p(I|x) is modeled as a product of exponential distributions over pairwise Sampson Errors:

where e^ij is the Sampson Epipolar Error between cameras i and j, computed from 2D correspondences extracted by SuperPoint+SuperGlue. The Sampson error is clamped at ε = 10 to handle outlier correspondences robustly.

The gradient of log p(I|x) is just the negative gradient of the total Sampson error across all pairs. It tells the optimizer: "adjust these camera poses so that epipolar lines pass closer to the matched keypoints."

Implementation Details

• Applied late: GGS runs only during the last 10 of the 100 diffusion steps. Early in the process, poses are too noisy for geometric constraints to help.
• 100 iterations per step: At each diffusion step where GGS is active, 100 gradient-descent iterations adjust the mean.
• Adaptive strength: The guidance strength s is set so that the gradient norm doesn't exceed α · ||μ_t|| with α = 0.0001. This prevents geometric guidance from overwhelming the diffusion prior.

Why geometry guidance works so well

Neural networks are fundamentally bad at precise geometric regression. A network might learn that "cameras looking at the same object should face inward," but it can't learn that "this specific camera's optical axis must pass within 0.5 pixels of this specific point." That level of precision comes from explicit geometric computation.

GGS provides this precision by computing exact Sampson errors from 2D correspondences. The gradient of the Sampson error tells the optimizer exactly which direction to nudge each camera to improve geometric consistency. The diffusion prior provides the rough neighborhood; GGS provides the final precision within that neighborhood.

The late application (last 10 steps) is critical: early in denoising, cameras are too scattered for epipolar constraints to be meaningful (random camera pairs have no real geometric relationship). By step 10, the diffusion model has already placed cameras approximately right — GGS just needs to fine-tune within a small region, which is exactly what gradient descent excels at.

Chapter 5: The Architecture

The denoiser D_θ is a transformer that processes the noisy camera parameters jointly with image features. Let's trace the data flow.

Inputs: Three Streams per Camera

For each of the N cameras, the transformer receives a token built from three components:

• Noisy pose x_tⁱ: The 8-dimensional camera parameter vector at the current noise level. Projected through a linear layer to 96 dimensions.
• Diffusion time t: The scalar timestep, also projected to 96 dimensions.
• Image features ψ(Iⁱ): 384-dimensional DINO ViT-S/16 features for image Iⁱ, plus a 1-dimensional binary pivot flag (is this the canonical camera?). Total: 385 dimensions.

These three are concatenated into a single token per camera, then fed into the transformer.

The Transformer

The architecture is a standard transformer encoder with 8 layers, 4 attention heads, and feedforward dimension 1024. There's no decoder — all cameras attend to each other through self-attention. This is important: each camera can see every other camera's noisy pose and image features. The model learns to reason about multi-view consistency.

Output

The transformer's output tokens are passed through a 2-layer MLP (hidden dim 128, output dim 8) to produce the predicted clean camera parameters: log-focal-length f̂, quaternion q, and translation t for each camera.

The complete data flow with shapes

Feature Extraction: DINO

Image features come from DINO ViT-S/16, pretrained in a self-supervised fashion. The images are center-cropped and resized to 224x224, then features are extracted at three scales (1x, 1/2, 1/3) and averaged for multi-scale understanding. DINO's weights are fine-tuned during training — this lets the feature extractor adapt to the pose-estimation task.

Coordinate Frame Canonicalization

SfM datasets define poses in arbitrary scene-specific coordinate frames. To prevent the model from overfitting to these arbitrary frames, PoseDiffusion canonicalizes all poses relative to a randomly selected pivot camera. The pivot camera gets identity rotation and zero translation. A binary flag in the input tells the model which camera is the pivot. Translations are further normalized by their median norm to handle scale ambiguity.

Model size: why so small?

PoseDiffusion's transformer has only 8 layers, 4 heads, and processes at most ~20 tokens. Compare:

The tiny model works because: (1) the output space is 8N dimensions, not millions of pixels; (2) each token already carries rich 384-dim visual features from DINO; (3) the geometric reasoning required (relative camera placement) is compositionally simpler than photorealistic image generation. More parameters would likely overfit — CO3Dv2 has only ~37K scenes.

Chapter 6: Training

PoseDiffusion is trained with a remarkably simple objective: predict the clean cameras from noisy ones.

The Denoising Loss

At each training step, the model receives a batch of scenes with ground-truth cameras x₀ and images I. A random timestep t is sampled, noise is added to get x_t, and the denoiser predicts x₀:

That's it. No adversarial losses, no perceptual losses, no multi-task heads. Just an L2 loss between predicted and ground-truth camera parameters, averaged over all cameras in the scene and all timesteps.

Training Data

Two datasets provide the training signal:

• CO3Dv2: ~37,000 turntable-like videos of objects from 51 categories. Cameras annotated by COLMAP using 200 frames per video. Object-centric, mostly circular camera trajectories.
• RealEstate10K: ~57,000 YouTube clips of interiors and exteriors. Cameras annotated by ORB-SLAM2 + bundle adjustment. Scene-centric, mostly linear fly-through trajectories.

Training Details

• Optimizer: Adam, initial LR 0.0005, decayed 10x after 30 epochs
• Batch sampling: Each batch randomly samples 3-20 frames from a random scene. This variable count teaches the model to handle any number of cameras.
• Diffusion schedule: T = 100 steps, linear β schedule from 10^-3 to 0.2
• Training time: 2 days on 8 GPUs until convergence

Frozen vs. trained components

Total trainable: ~27M parameters. This is remarkably small — orders of magnitude fewer than image diffusion models. The small model size is possible because: (1) the output space is only 8 numbers per camera, not millions of pixels; (2) DINO provides rich visual features without needing a massive backbone; (3) the transformer only processes N tokens (number of cameras), not thousands of image patches.

Engineering decisions

Why diffusion for poses instead of direct regression? A regression network maps images directly to a single pose prediction. The problem: multi-view pose estimation is inherently ambiguous (especially with few views), and the loss landscape has many local minima. Diffusion provides: (1) iterative refinement from coarse to fine (the 100-step chain), (2) implicit ensembling (can sample multiple times for uncertainty), and (3) a natural injection point for geometric constraints (GGS). The PoseReg ablation confirms this: same architecture without diffusion scores 48.2 mAA vs 66.5 with diffusion.

Why DINO instead of CLIP or ResNet? DINO ViT-S/16 provides features trained with self-supervised objectives that emphasize spatial structure and object parts — exactly what you need for geometric reasoning. CLIP features emphasize semantic similarity, which is less useful for precise pose estimation. ResNet features lack the global receptive field that ViT's self-attention provides.

Why fine-tune DINO? Generic DINO features are good but not specialized for pose. Fine-tuning lets the feature extractor learn to emphasize viewpoint-discriminative information (silhouettes, parallax cues) over semantic content (object identity). This is one of the keys to PoseDiffusion's performance.

A key advantage of the diffusion formulation: the model is trained one step at a time. Unlike autoregressive methods that require backpropagation through the full generation chain, each training step only requires forward/backward through a single denoising step. This makes training tractable even for complex geometric reasoning.

Chapter 7: Results

PoseDiffusion is evaluated on two challenging real-world datasets with different characteristics. The results are compelling.

CO3Dv2: Object-Centric Scenes

Each scene is a turntable-like video of a single object. Cameras orbit the object at roughly constant distance. PoseDiffusion significantly outperforms all baselines in both sparse and dense settings.

Key observations: (1) The diffusion model alone (w/o GGS) already beats every baseline. (2) Adding GGS provides a further 10+ point boost in mAA(30). (3) The non-diffusion baseline PoseReg with the same architecture scores much lower, validating that diffusion itself — not just the architecture — is responsible for the gains.

RealEstate10K: Scene-Centric Views

These are fly-through videos of real interiors and exteriors — the domain where COLMAP traditionally excels. Yet PoseDiffusion still wins across all metrics and frame counts.

Novel View Synthesis

To test whether the estimated cameras are truly useful, the authors train NeRFs using PoseDiffusion's output. The NeRF rendering quality (PSNR) matches or exceeds NeRFs trained with COLMAP cameras — and crucially, replacing predicted focal lengths with ground-truth makes no difference, proving that the intrinsic estimation is highly accurate.

Generalization

Perhaps the most impressive result: a model trained on 41 CO3Dv2 categories transfers to 10 unseen categories with only a small accuracy drop (50.8 to 48.0 mAA). Even more remarkably, transferring from CO3Dv2 (object-centric, circular trajectories) to RealEstate10K (scene-centric, linear trajectories) — a huge domain shift — produces results comparable to PixSfM.

What degrades and when

• Fewer images (N): With N=3 frames, mAA(30) drops to ~42 (from 66.5 at N=10). With N=2, the problem becomes highly ambiguous — there are infinite configurations consistent with just a stereo pair. More images = more mutual constraints = better accuracy.
• Wider baselines: When cameras are spread far apart (wide-baseline), features share less overlap and the matching becomes harder. PoseDiffusion handles this better than COLMAP (which fails outright on many wide-baseline pairs) but still degrades. Extremely wide baselines (>120 degrees rotation) push error up significantly.
• Without GGS: mAA(30) drops from 66.5 to 56.0 on CO3Dv2 — a 10.5 point drop. The model still beats all baselines without GGS, proving the diffusion prior alone is valuable. GGS provides geometric precision that the neural network alone cannot match.
• Without diffusion (PoseReg baseline): Same architecture but trained as a direct regressor (no noise, no iterative denoising) scores only 48.2 mAA. The diffusion framework itself provides an 18.3 point improvement over regression.
• Domain shift: CO3Dv2 to RealEstate10K (object-centric circular → scene-centric linear) loses ~5-8 points. Unseen object categories within CO3Dv2 lose only 2.8 points (50.8 → 48.0), showing good within-domain generalization.

Chapter 8: The Bundle Adjustment Connection

PoseDiffusion's name includes "Bundle Adjustment" for a reason. The connection runs deeper than a surface analogy.

Classical Bundle Adjustment

Bundle Adjustment (BA) is the gold standard for refining camera poses. Given initial camera estimates and 3D point estimates, BA minimizes the reprojection error: the sum of squared distances between observed 2D keypoints and the projected positions of the estimated 3D points. It's a nonlinear least-squares optimization, typically solved with Levenberg-Marquardt.

where p_jⁱ is the observed 2D position of point j in camera i, and π is the projection function.

Diffusion as Implicit Bundle Adjustment

PoseDiffusion mirrors BA in several ways:

• Iterative refinement: BA iterates Levenberg-Marquardt steps. PoseDiffusion iterates denoising steps. Both converge from a rough initialization to a precise solution.
• Geometric constraints: BA explicitly minimizes reprojection error. GGS explicitly minimizes Sampson epipolar error. Both enforce multi-view geometric consistency.
• Joint optimization: BA optimizes all cameras jointly. The transformer's self-attention reasons about all cameras jointly.

But PoseDiffusion also has advantages over classical BA:

• Learned prior: BA has no prior over plausible camera configurations. PoseDiffusion has a strong learned prior from training on thousands of scenes. This prior is especially valuable when correspondences are sparse or unreliable.
• No explicit 3D points: BA requires triangulating 3D points, which is a chicken-and-egg problem (you need poses to triangulate, but you need 3D points for BA). PoseDiffusion skips 3D point estimation entirely.
• Single-step training: BA requires unrolling the full optimization to backpropagate gradients. PoseDiffusion trains one step at a time.

Chapter 9: Connections

PoseDiffusion sits at the intersection of classical geometry and deep generative modeling. Let's map where it connects to the broader landscape.

Relation to VGGSfM

VGGSfM (2024) extends the PoseDiffusion idea: it adds differentiable Bundle Adjustment on top of the diffusion-predicted poses, and jointly estimates 3D structure. Where PoseDiffusion skips 3D points entirely, VGGSfM uses them as an additional refinement signal. VGGSfM achieves even higher accuracy, validating the "diffusion initialization + classical refinement" paradigm.

Relation to Rooms from Motion

Rooms from Motion tackles indoor scene reconstruction from panoramic images. Like PoseDiffusion, it faces the challenge of wide baselines (rooms have very different viewpoints). Both methods show that learned priors can rescue reconstruction when classical matching fails.

Relation to COLMAP

COLMAP is the workhorse SfM pipeline that PoseDiffusion aims to replace. PoseDiffusion's training data is itself generated by COLMAP (on CO3Dv2) or ORB-SLAM (on RealEstate10K) — the student learns from the teacher, then surpasses it in the hardest cases. This is a recurring pattern in learned geometry: use classical methods to generate training data, then train a model that handles the failure modes better.

Relation to Classical SLAM

SLAM systems (ORB-SLAM, VINS-Mono) solve pose estimation in real-time. They use fast, approximate methods (PnP, essential matrix) and rely on temporal continuity. PoseDiffusion operates on unordered image sets, handles wider baselines, but is much slower. Future work could combine the best of both: SLAM for real-time tracking, PoseDiffusion for offline refinement.

Cheat Sheet