RADIO-ViPE — Veanors

Chapter 0 — The Problem

You strap a webcam to a robot and tell it: "go pick up the red mug on the shelf." To do this, the robot needs three things simultaneously:

Where am I? — Track its own position in 3D (odometry/SLAM).
What's the 3D shape of this room? — Build a geometric map.
Where is "the red mug"? — Ground free-form language onto 3D locations.

Existing systems force you to choose two out of three. Geometric SLAM (ORB-SLAM3) gives you 1 + 2 but knows nothing about "mugs." Open-vocabulary scene understanding (ConceptGraphs, HOV-SG) gives you 2 + 3 but requires pre-calibrated posed RGB-D — you need an expensive depth sensor, known camera parameters, and an offline processing pass. No system does all three online from a raw phone video.

The gap. No prior method simultaneously achieves: (a) online operation, (b) open-vocabulary semantic grounding, (c) robust odometry, (d) accurate mapping, (e) dynamic scene robustness, and (f) calibration-free operation. RADIO-ViPE is the first to unify all six.

But there's a deeper problem. Real environments move. People walk through your frame. Someone pushes a chair. A door swings open. Traditional SLAM treats every pixel as belonging to a rigid world — when that assumption breaks, the entire map corrupts. And it breaks constantly in any space with humans.

The simulation above shows why this matters. Static features (blue) give consistent reprojections across frames. Moving objects (orange) create impossible geometric constraints — the optimizer tries to explain their motion as camera motion, corrupting the entire trajectory. The system needs to know which points to trust.

What we need. A system that: (1) ingests raw monocular RGB — no depth sensor, no calibration, no pose initialization. (2) Builds a 3D map annotated with language-aligned features so you can query "red mug" at runtime. (3) Handles dynamic objects gracefully without manual class lists. (4) Runs online at 8-10 FPS.

Why can't we just run an open-vocabulary model (like CLIP) on each frame independently and skip the SLAM entirely?

CLIP is too slow to run per-frame Without 3D geometry, you can't answer "where" questions — you'd know there's a mug but not its metric position in the room CLIP only works on cropped objects, not full scenes

Chapter 1 — The Key Insight

RADIO-ViPE's breakthrough is a single architectural decision: don't just use semantics as a post-processing layer — inject them directly into the geometry optimization itself.

Previous approaches have a two-stage pipeline: first do SLAM (geometry only), then paste semantic labels onto the resulting point cloud. This means semantics can't help geometry, and geometry can't help semantics. If the SLAM drifts, the semantic map drifts with it. If an object moves, the SLAM has no semantic signal to detect that motion.

Tight coupling. RADIO-ViPE fuses vision-language embeddings with geometric constraints inside the bundle adjustment itself — in the initialization (semantic flow), in the optimization (embedding similarity energy term), and in the factor graph connections (embedding-based co-visibility). Semantics and geometry co-optimize. This is the "tightly coupled" in the title.

The second insight is about robustness. Instead of trying to detect and mask dynamic objects (which requires knowing what's dynamic a priori), RADIO-ViPE lets the optimization itself figure it out. A pixel that gives consistent semantic embeddings across multiple views over time is probably static. A pixel whose embeddings fluctuate wildly is probably moving. This temporal stability signal controls how much the optimizer trusts each pixel — via an adaptive Barron loss kernel that smoothly transitions from full trust (ℓ₂) to near-rejection (Cauchy).

The diagram above shows the three-way coupling. Traditional SLAM has one energy term (photometric). RADIO-ViPE adds a semantic energy term that penalizes embedding inconsistency across views, and wraps both in an adaptive kernel that downweights dynamic regions. All three interact in one joint optimization.

Why "RADIO"? RADIO stands for "Reduce All Domains Into One" — it's an agglomerative foundation model that distills multiple teacher networks (DINOv2, CLIP, SAM) into a single student. This gives RADIO-ViPE features that are simultaneously good at: spatial discriminability (from DINOv2), language alignment (from CLIP), and segmentation (from SAM). One backbone, three capabilities.

What's the key difference between "tight coupling" and the previous approach of running SLAM first, then adding semantics?

Tight coupling is faster at inference time In tight coupling, semantic consistency is a term in the geometric energy function — semantic and geometric errors are optimized jointly, so each modality improves the other Tight coupling uses a different SLAM backbone

Chapter 2 — Background: Factor Graph SLAM

Before diving into RADIO-ViPE's method, we need to understand what it's building on. The core framework is factor graph optimization — specifically, dense bundle adjustment as formulated in DROID-SLAM.

Factor Graphs in 30 Seconds

A factor graph G = (V, E) represents a joint optimization problem. Variable nodes V are the unknowns — camera poses T_i ∈ SE(3), per-pixel disparity maps d_i, and camera intrinsics K. Factor nodes (edges E) encode constraints between variables — "the pixel at position u in frame i, when reprojected into frame j, should land near position v."

// The classic photometric SLAM energy:
E_photo = Σ_(i,j)∈E Σ_u w(u) · ‖Ω_ij^prior − Ω_ij(u)‖²

// Where:
Ω_ij^prior = μ_ij − u     // predicted flow from geometry
μ_ij = Π_j(T_j T_i⁻¹ · Π_i⁻¹(u, d_i(u)))     // reproject u into frame j
Ω_ij(u)     // optical flow network's prediction of where u went
w(u)     // per-pixel confidence from the flow network

In words: for every pair of connected keyframes, for every pixel, measure how well our current geometry (poses + depths) explains the observed optical flow. Minimize the total discrepancy using Gauss-Newton.

What DROID-SLAM gives us

DROID-SLAM (Teed & Deng, 2021) made this framework practical by: (a) learning the flow prediction with a neural network that also predicts per-pixel confidence w(u), (b) operating at 1/8 resolution for speed, and (c) using iterative correlation volume lookups for sub-pixel flow. The baseline ViPE system extends DROID-SLAM to work without calibration by jointly optimizing intrinsics.

The calibration-free twist

ViPE bootstraps camera intrinsics from uniformly sampled frames using GeoCalib — a foundation model that estimates focal length and principal point from single images. These are then co-optimized within bundle adjustment. This means RADIO-ViPE requires zero setup — point a phone camera at a scene and go.

The interactive factor graph above shows variable nodes (poses, depths) and the constraints connecting them. In standard SLAM, edges encode only photometric consistency. RADIO-ViPE will add semantic edges — but we need to understand the foundation model first.

Why dense (not sparse)? Traditional feature-based SLAM (ORB-SLAM) matches sparse keypoints. Dense SLAM optimizes every pixel. Dense is more expensive but gives: (a) better depth maps for reconstruction, (b) robustness in textureless regions where sparse features fail, and (c) a natural place to attach dense semantic embeddings.

In the photometric energy E_photo, what does w(u) represent and why is it important?

The pixel's RGB weight — brighter pixels contribute more The flow network's per-pixel confidence — it downweights pixels where the optical flow prediction is uncertain (e.g., occluded regions, textureless patches) The camera's exposure weight — compensates for auto-exposure changes

Chapter 3 — The Full Pipeline

Let's trace a frame through the entire RADIO-ViPE system, from raw pixel to semantic 3D map. This is the paper's Figure 2 made concrete.

Geometry Semantics Fusion Output

Step by step:

1. Frame arrives — Raw RGB at arbitrary resolution. No depth. No pose. No intrinsics.

2. Camera initialization (first few frames only) — GeoCalib estimates [f_x, f_y, c_x, c_y] from uniformly sampled frames. These are later refined in bundle adjustment.

3. Keyframe selection — Compute weighted dense optical flow between this frame and the last keyframe. If relative motion exceeds a threshold → this frame becomes a keyframe and enters the factor graph.

4. Multi-modal feature extraction — Run RADSeg (built on RADIO) in sliding-window mode over the keyframe. Output: dense embeddings at (H/8, W/8) spatial resolution. Then PCA-compress from full dimensionality to D=256.

5. Depth estimation — Run a monocular depth foundation model (MoGe or UniDepth). Convert to inverse depth (disparity). Downsample to match flow resolution (1/8).

6. Semantic flow initialization — Augment the photometric flow prior with a semantic correspondence term: for each pixel, find its best match in the other frame using RADIO embedding cosine similarity. Blend with photometric flow using confidence-weighted fusion (Eq. 1).

7. Bundle adjustment — Jointly optimize poses, depths, and intrinsics by minimizing E_total = γ_photo·E_photo^ark + γ_embed·E_embed + E_reg. Factor graph connectivity includes both geometric proximity AND embedding-based co-visibility links.

8. Non-keyframe poses — Estimated in parallel via photometric alignment to their two nearest keyframes (no per-frame depth needed).

9. Open-vocabulary query — At any time, a text query is encoded with SigLIP's text encoder, then matched against the decompressed RADIO features of all 3D map points via cosine similarity.

Data flow summary.
Input: RGB frame H×W×3
→ RADSeg: H/8 × W/8 × D_full → PCA: H/8 × W/8 × 256
→ Depth model: H/8 × W/8 × 1 (disparity)
→ Flow network: H/8 × W/8 × 2 (flow) + H/8 × W/8 × 1 (confidence)
→ Bundle adjustment: {T_i ∈ SE(3), d_i ∈ R^H/8×W/8, K ∈ R⁴}
→ 3D map: N points × (xyz + 256-dim embedding)

Speed budget. The full pipeline runs at 8-10 FPS on an RTX 4090. The bottleneck is the foundation models (RADSeg + depth), not the optimization itself. PCA compression (from ~1000+ dims to 256) is critical — without it, the memory for storing per-pixel embeddings across all keyframes would be prohibitive.

Why does the system convert depth to inverse depth (disparity) before optimization?

Inverse depth is numerically more stable — nearby objects have finite disparity while far objects approach zero, avoiding the infinite-depth singularity and giving roughly uniform uncertainty across depths It's just a convention from DROID-SLAM with no practical benefit The depth model outputs disparity directly

Chapter 4 — Dense Visual Feature Extraction

The semantic backbone is what gives RADIO-ViPE its superpowers. But why RADIO specifically? Why not just CLIP? Let's trace the data flow through the embedding pipeline to understand.

The RADIO Foundation Model

RADIO (Reduce All Domains Into One) is an agglomerative model — it distills multiple specialist teachers into one student:

DINOv2 — gives spatially discriminative features (good for matching, dense correspondence)
CLIP/SigLIP — gives language-aligned features (good for open-vocabulary recognition)
SAM — gives segmentation-aware features (good for object boundaries)

The student inherits all three capabilities. For SLAM, this is perfect: we need spatial discriminability (for flow/matching), language alignment (for open-vocabulary queries), and boundary awareness (for clean object maps).

RADSeg: Dense Spatial Embeddings

RADSeg extends RADIO to produce dense per-pixel embeddings (rather than patch-level). It uses a sliding-window approach with self-attention refinement across overlapping regions. The output for each keyframe is:

Z_i ∈ R^D×H/8×W/8 // dense embedding map, D ≈ 1024 before PCA

PCA Compression: The Key Efficiency Trick

Storing 1024-dim embeddings per pixel per keyframe is expensive. With 100 keyframes at 60×80 resolution, that's 100 × 4800 × 1024 × 4 bytes ≈ 1.9 GB just for embeddings. RADIO-ViPE applies PCA to compress to D=256:

Z_i^PCA = W_PCA^⊤ · (Z_i − μ) ∈ R^{256×H/8×W/8}

// Crucial: PCA is applied in the ENCODER feature space, not
// the language-aligned space. This preserves spatial structure
// needed for geometric reasoning.

The simulation above shows how PCA dimensionality affects semantic segmentation quality (mIoU). The paper's ablation (Fig. 4) shows D=256 loses less than 1% mIoU compared to full-dimensional — the sweet spot between memory and accuracy.

When is PCA computed? Not offline! RADIO-ViPE waits until the bundle adjustment initialization period accumulates enough keyframes, then computes PCA components from that buffer. This means the PCA basis adapts to the specific scene — a clever trick that ensures the 256 dimensions capture scene-relevant variance.

Worked Example: Memory Budget

Let's trace the numbers for a 60-second clip at 30 FPS (selecting ~1 keyframe per 5 frames = 360 keyframes):

// Without PCA (D=1024):
360 keyframes × (480/8 × 640/8) pixels × 1024 dims × 4 bytes = 7.1 GB

// With PCA (D=256):
360 keyframes × (480/8 × 640/8) pixels × 256 dims × 4 bytes = 1.8 GB

// 4× reduction — fits in GPU memory for joint optimization

Why not compress in language space? The paper explicitly notes: PCA is performed on the encoder features, not the language-aligned SigLIP space. Why? The encoder space has the spatial discriminability from DINOv2 — essential for matching and correspondence. Language alignment is recovered at query time by projecting back through a learned mapping.

RADIO-ViPE computes PCA during the BA initialization period rather than using a pre-computed basis. What advantage does this give?

It's faster than loading a pre-computed basis from disk The PCA basis adapts to the specific scene's feature distribution, capturing the 256 most informative dimensions for THIS environment rather than a generic average It avoids needing to store a large PCA matrix in the model weights

Chapter 5 — Joint Vision-Language-Geometric Bundle Adjustment

This is the core technical contribution. RADIO-ViPE doesn't just use semantic embeddings — it optimizes through them. The bundle adjustment energy has three terms that are jointly minimized.

Term 1: Dense Photometric Flow (from DROID-SLAM)

For each edge (i, j) in the factor graph, reproject every pixel from frame i into frame j using current geometry estimates, then compare against the optical flow network's prediction:

E_photo = Σ_(i,j)∈E Σ_u w(u) · ‖Ω_ij^prior − Ω_ij(u)‖²

// Geometric reprojection:
μ_ij = Π_j(T_j · T_i⁻¹ · Π_i⁻¹(u, d_i(u)))

// Prior flow = geometric prediction minus pixel position:
Ω_ij^prior = μ_ij − u

// This gets compared against flow network output Ω_ij(u)

Term 2: RADIO Embedding Similarity (NOVEL)

This is new. For each edge, after reprojecting pixel u from frame i to its landing position in frame j, look up the RADIO embedding at that position (using bilinear interpolation because the landing point won't be at integer coordinates), then measure cosine similarity:

// Bilinear interpolation of target embedding:
Ẑ_j(P_i,j(u)) = Σ_{(l,m)∈N(P_i,j(u))} w_lm · Z_j^lm

// Cosine similarity between source and projected target:
cs_ij(u) = Z_i(u)^⊤ · Ẑ_j(P_i,j(u)) / (‖Z_i(u)‖ · ‖Ẑ_j(P_i,j(u))‖)

// Convert to photometric-scale residual:
r_embed(u) = λ_embed · √(2(1 − cs_ij(u))) // λ=2

// Full embedding energy:
E_embed = Σ_(i,j)∈E Σ_u w(u) · r_embed(u)²

The simulation above demonstrates the embedding similarity computation. When geometry is correct, a pixel in frame i reprojects to the same semantic region in frame j — high cosine similarity, low residual. When geometry is wrong OR the object moved, the reprojection lands on a semantically different region — low similarity, high residual. The optimizer then adjusts poses/depths to maximize cross-view semantic consistency.

Why √(2(1−cos))? This is the angular distance formula — it maps cosine similarity ∈ [−1,1] to a distance ∈ [0,2]. With λ_embed=2, the residual magnitude matches typical photometric flow residuals (a few pixels). This keeps the two energy terms balanced without manual tuning of relative weights.

Term 3: Depth Regularization

E_reg = α_disp · Σ_u ‖d_i(u) − d_i^prior(u)‖²

// d_i^prior = disparity from the foundation depth model
// α_disp = 1.0 — prevents drift while allowing multi-view refinement

Factor Graph Connectivity: Semantic Co-Visibility

Standard SLAM connects keyframes that are geometrically nearby (within a temporal window). RADIO-ViPE adds semantic edges: compute a global descriptor per keyframe (mean-pool its RADSeg embeddings + ℓ₂-normalize), then for each new keyframe, find all non-recent keyframes with cosine similarity above threshold η. These get bidirectional edges.

This means: if the robot revisits a semantically similar area (loop closure!), the factor graph gets a direct constraint even if the frames are temporally distant. And it's free — the embeddings are already in the buffer.

The full objective:
E_total = γ_photo · E_photo^ark + γ_embed · E_embed + E_reg

Optimized over: {T_i} ∈ SE(3), {d_i} ∈ R^H×W, {K} ∈ R⁴
Method: Gauss-Newton (linearize, solve normal equations, update)

If a pixel in frame i reprojects to a DIFFERENT object in frame j (e.g., it was occluded by a moving person), what happens to the embedding similarity term for that pixel?

Nothing — the photometric term handles this case The cosine similarity drops (different object = different embedding), creating a large residual r_embed that either pulls the geometry to fix the reprojection or, via the robust kernel, gets downweighted as a dynamic outlier The system removes that pixel from optimization entirely

Chapter 6 — Temporally Consistent Adaptive Robust Kernel

Here's the problem the previous chapter left unresolved: when a person walks through the frame, both the photometric and embedding terms see large residuals for those pixels. The optimizer could try to "explain" the person's motion by warping the camera pose — corrupting the entire map. We need a way to ignore dynamic pixels without knowing in advance which pixels are dynamic.

Step 1: The Temporal Stability Field

The key observation: a moving pixel gives inconsistent embeddings across different viewpoints over time, while a static pixel gives consistent ones. Compute the temporal mean and variance of cosine similarity for each pixel across all edges connected to keyframe i:

// Mean similarity across all connected keyframes:
c̄s_i(u) = (1/|N(i)|) · Σ_j∈N(i) cs_ij(u)

// Variance across time:
σ_i²(u) = (1/|N(i)|) · Σ_j∈N(i) (cs_ij(u) − c̄s_i(u))²

// Stability field — high = static, low = dynamic:
S_i(u) = c̄s_i(u) · (1 − σ_i²(u)) ∈ [0, 1]

Think about what this captures:

Static wall: high mean similarity (always matches), low variance → S ≈ 1
Walking person: low mean (different appearance when they move), high variance → S ≈ 0
Pushed chair: medium mean (matched before it moved), HIGH variance (matches some frames but not others) → S ≈ 0

The "quasi-static" detection. A chair that was moved partway through the session has high variance (it matches keyframes from before the move but not after). The variance term catches this even though the mean might still be moderate. This is what makes RADIO-ViPE robust to "agent-displaced scene elements" — not just actively moving objects.

Step 2: Three-Regime Barron Loss Mapping

The Barron loss ρ_α is a single parametric family that recovers different robust losses depending on α:

α = 2 → ℓ₂ loss (fully quadratic, full trust)
α = 1 → Huber loss (linear tails, moderate robustness)
α → 0 → Cauchy loss (very heavy tails, near-rejection of outliers)

Map stability S to shape α via piecewise-linear mapping:

α_i(u) = {
  2                   if S_i(u) ≥ θ_s = 0.75 // static → full ℓ₂
  1 + (S−θ_m)/(θ_s−θ_m) if θ_m ≤ S < θ_s      // movable → Huber region
  α_dyn + (S/θ_m)(1−α_dyn) if S < θ_m = 0.35 // moving → Cauchy region
}

The interactive simulation above lets you explore how the stability field S maps to the Barron loss shape α, and how this affects the per-pixel weight in the optimization. Drag the slider to see how a pixel transitions from "fully trusted" to "nearly ignored" as it becomes more dynamic.

Step 3: Apply to the Photometric Energy

E_photo^ark = Σ_(i,j)∈E Σ_u w_ark(E_photo(u), α_i(u)) · E_photo(u)

// Adaptive weight from Barron loss derivative:
w_ark(r, α) = (1 / max(r, ε)) · (∂ρ_α(r) / ∂r)

The weight w_ark smoothly downweights pixels based on BOTH their residual magnitude AND their temporal stability. A pixel on a moving person gets α near 0 (Cauchy), so even moderate residuals produce tiny weights. A pixel on a static wall gets α=2 (ℓ₂), so it contributes fully regardless of residual.

Why not just mask dynamic pixels out entirely? Three reasons: (1) The stability field is continuous — there's no clean binary boundary between "static" and "dynamic." (2) Partially dynamic pixels (edge of a moving person overlapping a wall) should contribute partially. (3) Keeping all pixels in the optimization (with appropriate weights) provides better gradient conditioning for Gauss-Newton than dropping rows from the Jacobian.

Worked Example

// A pixel on a static wall observed from 5 keyframes:
cs values: [0.92, 0.89, 0.91, 0.88, 0.90]
c̄s = 0.90, σ² = 0.00016
S = 0.90 × (1 − 0.00016) = 0.8999 → α = 2 (full ℓ₂ trust)

// A pixel on a walking person:
cs values: [0.31, 0.15, 0.42, 0.08, 0.22]
c̄s = 0.236, σ² = 0.0136
S = 0.236 × (1 − 0.0136) = 0.233 → α ≈ −0.3 (deep Cauchy → nearly ignored)

A sofa is static for the first 200 frames, then someone pushes it at frame 200. By frame 250, what does the stability field S look like for pixels on the sofa?

S ≈ 1 because the sofa was static for most of the sequence S ≈ 0 because the mean similarity drops (pre-move frames mismatch post-move geometry) AND variance is high (matches some views but not others) S is undefined because the system only looks at the most recent keyframe

Chapter 7 — Open-Vocabulary Grounding

The final piece: now that we have a 3D map where every point carries a 256-dim RADIO embedding, how do we answer "where is the red mug?"

Query-Time Pipeline

// 1. Encode the text query with SigLIP text encoder:
q = SigLIP_text("red mug") ∈ R^D_sigLIP

// 2. For each 3D map point p with stored embedding z_p:
// Decompress from PCA space back to full encoder space:
z_full = W_PCA · z_p + μ

// 3. Project into SigLIP language-aligned space via learned head:
z_lang = f_proj(z_full) ∈ R^D_sigLIP

// 4. Cosine similarity → relevance score:
score(p) = z_lang^⊤ · q / (‖z_lang‖ · ‖q‖)

// 5. Threshold or top-k → localized 3D region

This is "open-vocabulary" because the text query can be anything — no pre-defined class list. You can ask for "red mug," "the chair near the window," "something to sit on," or even "potential trip hazards." The RADIO/SigLIP embedding space captures semantic similarity broadly.

The simulation above shows a schematic 3D point cloud colored by relevance to a query. High-scoring regions (warm colors) cluster around the queried object. The system returns the 3D centroid and bounding region.

Why SigLIP not CLIP? SigLIP uses a sigmoid loss instead of softmax contrastive loss, which scales better to large batch sizes and produces better-calibrated similarity scores. RADIO was trained with SigLIP as one of its teachers, so the embedding space naturally aligns with SigLIP's text encoder.

Real-Time Considerations

The projection f_proj(z_full) is a small MLP — negligible cost. The expensive part is the cosine similarity computation over all map points. For a typical room with ~100K points, this is a single batched matrix-vector multiply — fast on GPU.

The paper's qualitative results (Fig. 5) show queries like "teddy bear," "painting on wall," "blue cushion" correctly localizing objects in Replica scenes, despite never training on object categories.

The grounding hierarchy. You can query at different granularities: object-level ("the lamp"), region-level ("the desk area"), material-level ("wooden surfaces"), or even functional ("things I can use as a stepping stool"). The embedding space supports all of these because RADIO's teachers collectively encode appearance, function, and context.

Why does RADIO-ViPE perform PCA in encoder space but query in SigLIP language space? Why not compress directly in language space?

Language space is higher dimensional so PCA would be slower The encoder space preserves spatial discriminability (from DINOv2) needed for matching and correspondence during SLAM, while language alignment is only needed at query time and can be recovered via projection SigLIP's space is already compressed so PCA isn't needed

Chapter 8 — Experiments & Results

RADIO-ViPE is evaluated on two axes: SLAM robustness (can it track accurately despite moving objects?) and semantic segmentation quality (are the open-vocabulary labels correct?).

SLAM on TUM-RGBD Dynamic Sequences

The TUM-RGBD benchmark has sequences where people walk, sit, and move objects in front of the camera. The metric is Absolute Trajectory Error (ATE) in centimeters — lower is better.

Key results from Table II:

// Average ATE (cm) across 8 dynamic sequences:
DynaMON (2024):       1.76 cm
RADIO-ViPE:         1.90 cm // embedding term only, no robust kernel
ViPE + SAM (2025):    2.17 cm
RADIO-ViPE^ark:      1.63 cm // full system with adaptive kernel → BEST

The ablation is revealing. RADIO-ViPE without the adaptive kernel (1.90 cm) is already better than ViPE+SAM (2.17 cm) — the embedding similarity term alone helps. Adding the adaptive robust kernel drops it to 1.63 cm — best among all methods. The kernel doesn't just help; it's the difference between competitive and state-of-the-art.

Semantic Segmentation on Replica

The Replica benchmark evaluates open-vocabulary 3D segmentation. Unlike competing methods, RADIO-ViPE operates without ground-truth depth, pose, or calibration:

// f-mIoU (without background), higher is better:
RayFronts (offline, calibrated): 62.03%
RADIO-ViPE^GT (w/ GT depth+pose): 52.24%
RADIO-ViPE (fully online, no cal): 50.63%

// Gap: GT vs estimated geometry costs only ~1.6% f-mIoU
// This proves: the SLAM quality is good enough for semantics

The small gap between RADIO-ViPE^GT and RADIO-ViPE confirms that the geometry estimation is reliable — removing all geometric supervision (depth, pose, calibration) barely degrades semantic performance.

What the numbers mean for deployment

The deployment story. Previous open-vocabulary methods (ConceptGraphs, HOV-SG, etc.) require: calibrated stereo/RGB-D cameras + external odometry + offline processing. RADIO-ViPE requires: one webcam. It gives up ~10% absolute mIoU vs the best offline method but gains: real-time operation, no calibration, no depth sensor, and dynamic robustness. For a robot with a cheap camera, this is transformative.

Computational Cost

// Runtime breakdown (RTX 4090):
RADSeg inference:      ~50 ms/keyframe
Depth model:           ~30 ms/keyframe
Flow + BA iteration:    ~20 ms/keyframe
Total:                 ~100-125 ms → 8-10 FPS

The paper shows RADIO-ViPE (no kernel) at 1.90 cm ATE and RADIO-ViPE^ark at 1.63 cm. Why does the adaptive kernel help SLAM accuracy when its primary purpose is handling dynamic objects?

The kernel speeds up convergence so the optimizer runs more iterations By downweighting dynamic pixels, the optimizer no longer tries to explain their motion as camera motion — the remaining static pixels give cleaner, unbiased pose estimates The kernel adds more training data from static regions

Chapter 9 — Connections & Cheat Sheet

Equation Cheat Sheet

// 1. Semantic Flow Initialization (Eq. 1):
Ω_prior(u) := β · Ω_prior(u) + (1−β) · Ω_sem(u)
// β = photometric confidence; Ω_sem from RADIO cosine similarity

// 2. Photometric Flow Energy (Eq. 3):
E_photo = Σ w(u) · ‖Ω_ij^prior − Ω_ij(u)‖²

// 3. Embedding Cosine Similarity (Eq. 5):
cs_ij(u) = Z_i(u)⊤ · Ẑ_j(P_i,j(u)) / (‖Z_i(u)‖ · ‖Ẑ_j(P_i,j(u))‖)

// 4. Embedding Residual (Eq. 6):
r_embed(u) = 2 · √(2(1 − cs_ij(u)))

// 5. Temporal Stability Field (Eq. 8):
S_i(u) = c̄s_i(u) · (1 − σ_i²(u))

// 6. Three-Regime α Mapping (Eq. 9):
θ_s=0.75 (static), θ_m=0.35 (moving), α_dyn≤0

// 7. Full Objective (Eq. 12):
E_total = γ_photo · E_photo^ark + γ_embed · E_embed + α_disp · E_reg

Key Design Decisions

PCA in encoder space (not language space) — preserves spatial discriminability for correspondence while language alignment is recoverable at query time.
D=256 compression — 4× memory reduction with <1% mIoU loss.
Scene-specific PCA — computed during BA initialization from the keyframe buffer, not pre-trained.
Continuous α mapping — avoids hard thresholds; the Barron loss smoothly transitions between regimes.
Semantic co-visibility edges — free loop closure detection using already-computed embeddings.
λ_embed=2 — calibrates embedding residuals to photometric magnitude.

Limitations (from the paper)

Background segmentation: Performance drops for structural classes (wall, floor) — these are semantically uniform and hard to segment open-vocabulary.
GPU requirement: 8-10 FPS needs an RTX 4090. Edge deployment requires model distillation or quantization.
PCA initialization: Needs a few seconds of video before PCA is stable. The very first frames operate without the embedding term.

Related Lessons

Gleams: Modern SLAM — visual-inertial SLAM, keyframe selection, loop closure fundamentals
Gleams: Multi-View Geometry — epipolar geometry, essential matrix, bundle adjustment basics
Gleams: Optical Flow — dense flow estimation, correlation volumes, RAFT architecture
Veanors: MASt3R-SLAM — real-time dense SLAM with 3D reconstruction priors
Veanors: VGGT — one-pass geometry from unposed images
Veanors: DUSt3R — pointmap regression for uncalibrated reconstruction
Veanors: GAS Semantic Mapping — monocular video to semantic floorplans

The bigger picture. RADIO-ViPE sits at the convergence of three trends: (1) foundation models making monocular depth/flow reliable, (2) agglomerative models (RADIO) unifying multiple capabilities in one backbone, and (3) robust optimization handling real-world messiness. As foundation models improve, the geometry estimation gets better for free — the architecture is designed to benefit from better components without redesign.

What is the single most important architectural decision that makes RADIO-ViPE possible (something that didn't exist in prior work)?

Using optical flow networks for correspondence The tight coupling of multi-modal (vision+language) embeddings directly inside the bundle adjustment energy function, making semantics and geometry co-optimize Running at 1/8 resolution for speed