GAS v2

Chapter 1: The Shared Backbone Revolution

GAS v1 ran three completely separate neural networks on every video frame. GO-SLAM used its own CNN encoder. Faster R-CNN used a ResNet-50 backbone with FPN. SAM used a ViT-H image encoder. Each model independently extracted features from the same image, tripling the compute.

In 2025, a single model family dominates visual feature extraction: DINOv2. Meta's self-supervised ViT, trained on 142M curated images, produces features so general that they serve as the backbone for nearly every downstream vision task.

DINOv2 is everywhere

Look at what uses DINOv2 internally:

• VGGT — initializes its ViT encoder from DINOv2
• MASt3R / MASt3R-SLAM — DINOv2 ViT backbone for stereo matching
• Depth Anything V2 — DINOv2 encoder + DPT decoder
• Grounding DINO 1.5 — uses a Swin/ViT backbone trained with DINO-style objectives
• SAM 2 — Hiera backbone (evolved from MAE/ViT lineage)

This is the shared backbone revolution. Instead of running three unrelated networks, GAS v2 can extract DINOv2 features once and route them to multiple task heads. One forward pass through a ViT-G produces features rich enough for geometry estimation, object detection, segmentation, and depth prediction simultaneously.

How feature sharing works

DINOv2 ViT-G processes an image into a grid of 1024-dimensional patch tokens (e.g., 16x16 = 256 tokens for a 224x224 image, more for higher resolutions). These tokens encode both local appearance (texture, edges, color) and global semantics (object identity, scene category). Task heads are lightweight networks that read these tokens and produce task-specific outputs:

• Geometry head — predicts pointmaps (3D coordinates per pixel)
• Detection head — predicts bounding boxes + class labels
• Segmentation head — predicts per-pixel masks
• Depth head — predicts metric depth per pixel

Chapter 2: 3D Geometry: VGGT or MASt3R-SLAM

GO-SLAM was GAS v1's geometry backbone. It worked, but it required camera intrinsics (the K matrix: focal length, principal point), sometimes stereo pairs, and ran a complex neural SLAM loop. In 2025, two alternatives have made calibrated cameras optional.

Option A: VGGT (offline, batch)

VGGT (Visual Geometry Grounded Transformer) won CVPR 2025 Best Paper. Feed it a set of unposed images — no camera intrinsics, no ordering, no calibration — and it outputs in a single forward pass:

• Camera poses for every image
• Dense pointmaps (3D coordinates per pixel)
• Depth maps
• Camera intrinsics (estimated, not required as input)

# VGGT: single forward pass, all geometry
from vggt.models import VGGTModel

model = VGGTModel.from_pretrained("facebook/vggt-1b")
model.eval().cuda()

# images: (B, N, 3, H, W) - batch of N unposed images
predictions = model(images)

poses = predictions["extrinsic"]     # (B, N, 3, 4) camera-to-world
points = predictions["pointmap"]     # (B, N, H, W, 3) 3D per pixel
depth = predictions["depth"]         # (B, N, H, W) metric depth
intrinsics = predictions["intrinsic"] # (B, N, 3, 3) estimated K

No calibration. No sequential processing. No SLAM loop. Just images in, geometry out.

Option B: MASt3R-SLAM (real-time, streaming)

MASt3R-SLAM is real-time dense SLAM built on top of MASt3R's learned stereo matching. It processes frames sequentially as they arrive, maintains a running map, and handles loop closures. Unlike VGGT (which processes a batch at once), MASt3R-SLAM is designed for live operation.

# MASt3R-SLAM: real-time streaming
from mast3r_slam import MASt3RSLAM

slam = MASt3RSLAM(config="configs/base.yaml")

for frame in video_stream:
    result = slam.process_frame(frame)
    pose = result.pose        # current camera pose
    pointmap = result.points  # current dense 3D map
    if result.loop_closure:
        print("Loop closure detected!")

When to use which

Both approaches eliminate GAS v1's most fragile dependency: the camera intrinsics matrix K. GO-SLAM would produce distorted geometry if K was wrong by even a few pixels. VGGT estimates K. MASt3R-SLAM uses learned priors that are robust to approximate intrinsics. This single change eliminates hours of calibration debugging.

Chapter 3: Open-Vocabulary Detection: Grounding DINO

GAS v1's detection story was the most painful part of the entire project. To detect 10 object classes (door, chair, table, toilet, sink, sofa, bed, TV, refrigerator, exit sign), the authors had to:

1. Collect images from 5 different datasets (ScanNet, Matterport3D, ADE20K, NYU Depth V2, COCO)
2. Convert all annotations to COCO format (different datasets use different schemas)
3. Balance class distributions (some classes had 50x more samples than others)
4. Fine-tune Faster R-CNN with ResNet-50-FPN backbone
5. Tune confidence thresholds per class

All of this to detect 10 classes. Want to add "bookshelf"? Start over: find training data, re-annotate, re-train.

Grounding DINO: zero-shot, open-vocabulary

Grounding DINO detects any object described in natural language. No training data. No fine-tuning. No COCO format conversion. You just describe what you want to find.

# Grounding DINO: detect anything from text
from groundingdino.util.inference import load_model, predict

model = load_model(
    "groundingdino/config/GroundingDINO_SwinB.py",
    "weights/groundingdino_swinb.pth"
)

# Detect ANY objects via text prompt
text_prompt = "door . chair . table . exit sign . bookshelf . window"
boxes, logits, phrases = predict(
    model=model,
    image=image,
    caption=text_prompt,
    box_threshold=0.3,
    text_threshold=0.25
)
# boxes: (N, 4) in [cx, cy, w, h] normalized
# phrases: ['door', 'chair', 'table', ...] matched labels

How Grounding DINO works

Grounding DINO fuses a vision transformer (Swin-B or Swin-L) with a text encoder (BERT). It uses cross-modality attention to match text tokens to image regions:

1. The text encoder processes your prompt into text features
2. The image encoder extracts multi-scale visual features
3. Cross-attention layers fuse text and image features bidirectionally
4. A detection head predicts boxes where text-image attention is highest

The dot-separated prompt format ("door . chair . table") tells the model to detect each concept independently. Each detected box comes with the matching text phrase and a confidence score.

Practical tips

• Threshold tuning: Start with box_threshold=0.3 and lower to 0.15-0.2 for recall-critical applications (semantic mapping needs high recall)
• Prompt engineering: Use specific terms. "exit sign" works better than "exit". "office chair" vs "dining chair" if you need the distinction
• Speed: Grounding DINO runs at ~8 FPS on an RTX 4090 with Swin-B. For real-time, consider YOLO-World as a lighter alternative

Chapter 4: Grounded SAM 2 — Detect + Segment + Track

In GAS v1, detection and segmentation were separate steps with separate models. Faster R-CNN produced bounding boxes. Each box was then fed to SAM as a point prompt to generate a mask. For video, masks were merged across frames using IoU matching — a brittle process that lost objects when viewpoints changed significantly.

Grounded SAM 2 combines Grounding DINO and SAM 2 into a single system that detects, segments, and tracks objects across video frames. One call replaces three separate steps from v1.

The three stages in one

1. Detect — Grounding DINO finds objects from text prompt → bounding boxes
2. Segment — SAM 2 generates pixel-precise masks from boxes → instance masks
3. Track — SAM 2's memory attention propagates masks across video frames → consistent object IDs

# Grounded SAM 2: detect + segment + track
import supervision as sv
from grounded_sam2 import GroundedSAM2Pipeline

pipeline = GroundedSAM2Pipeline(
    grounding_model="gdino_swinb",
    sam2_checkpoint="sam2_hiera_large.pt",
    device="cuda"
)

# Single frame: detect + segment
results = pipeline.predict(
    image=frame,
    text_prompt="door . chair . table . exit sign",
    box_threshold=0.3
)
masks = results.masks       # (N, H, W) binary masks
labels = results.labels     # ['door', 'chair', ...]
boxes = results.boxes       # (N, 4) bounding boxes

# Video tracking: propagate masks with memory
tracker = pipeline.create_video_tracker()
for i, frame in enumerate(video_frames):
    if i == 0:
        # Initialize with detections on first frame
        tracked = tracker.init_frame(frame, results)
    else:
        # Propagate masks using SAM 2 memory bank
        tracked = tracker.propagate(frame)
    # tracked.masks has consistent IDs across frames

Why this matters for semantic mapping

GAS v1 had to solve two hard problems in post-processing: (1) which masks across different frames correspond to the same object? and (2) how do we merge their 3D projections? Grounded SAM 2 solves problem (1) directly — every mask comes with a consistent track ID. Problem (2) becomes trivial: just aggregate all 3D points that share the same track ID.

Chapter 5: Metric Depth: Depth Anything V2

GAS v1 relied on GO-SLAM's depth estimation or, when available, RGB-D sensor data. Both approaches have limitations: GO-SLAM's depth is tied to its SLAM quality, and RGB-D sensors (like Intel RealSense) are expensive, fragile, and limited in range.

Depth Anything V2 produces metric depth maps from any single RGB image. No stereo pair. No depth sensor. No camera calibration. Just one image → one depth map with real-world scale.

# Depth Anything V2: metric depth from any image
from depth_anything_v2.dpt import DepthAnythingV2

model = DepthAnythingV2(
    encoder="vitl",  # vitb, vitl, or vitg
    max_depth=20.0   # indoor scenes: 20m max
)
model.load_state_dict(torch.load("checkpoints/depth_anything_v2_metric_indoor_vitl.pth"))
model.eval().cuda()

depth_map = model.infer_image(image)  # (H, W) in meters
# depth_map[100, 200] = 3.7 means 3.7 meters from camera

Why metric depth matters

Depth Anything V1 produced relative depth — it knew that object A is farther than object B, but not how far either one actually is. For mapping, you need metric depth: real distances in meters. Depth Anything V2 added a metric depth head trained on synthetic data (Hypersim, Virtual KITTI) that predicts absolute distances.

Back-projection without K

GAS v1 used the equation P = d · K^-1 · [u, v, 1]^T to project pixels into 3D. This requires camera intrinsics K. With VGGT, K is estimated automatically. But even without VGGT, Depth Anything V2 + estimated field of view gives you approximate 3D:

# Approximate back-projection without exact K
import numpy as np

def backproject(depth, fov_deg=60, H=480, W=640):
    # Estimate focal length from field of view
    f = W / (2 * np.tan(np.radians(fov_deg / 2)))
    cx, cy = W / 2, H / 2
    # Create pixel coordinate grid
    u, v = np.meshgrid(np.arange(W), np.arange(H))
    # Back-project to 3D
    x = (u - cx) * depth / f
    y = (v - cy) * depth / f
    z = depth
    return np.stack([x, y, z], axis=-1)  # (H, W, 3)

Chapter 6: 3D Scene Graphs: ConceptGraphs

GAS v1's merging step was simple: project object masks into 3D, check IoU between point clouds, and merge those above a threshold. This is purely geometric — it knows nothing about what the objects are in a semantic sense. Two spatially overlapping point clouds get merged even if one is a chair and the other is a table leg behind it.

ConceptGraphs (Gu et al., 2023) replaces this with a fundamentally different representation: a 3D scene graph where each node is an object with both geometric and semantic properties.

What is a 3D scene graph?

A scene graph is a data structure where:

• Each node is an object with: 3D position (centroid), 3D bounding box, semantic label, CLIP embedding, and an open-vocabulary text description
• Each edge encodes a spatial relationship: "chair is near table", "lamp is on desk", "door leads to hallway"

This is dramatically richer than a point cloud. You can query the scene graph with natural language: "Where is the nearest exit?" "Which room has a refrigerator?" "Is there a fire extinguisher near the kitchen?"

How ConceptGraphs builds the graph

1. SLAM — get camera poses and depth (VGGT or MASt3R-SLAM in v2)
2. Open-vocab detection — Grounding DINO detects objects per frame
3. Feature extraction — CLIP encodes each detected region into a semantic embedding
4. 3D lifting — back-project detections using depth to get 3D object positions
5. Graph merging — merge observations of the same object using both geometric proximity AND CLIP feature similarity

Integration with GAS v2

ConceptGraphs slots perfectly into the GAS v2 pipeline:

• VGGT provides poses + depth (replacing the RGB-D sensor ConceptGraphs originally required)
• Grounding DINO provides open-vocabulary detections
• SAM 2 provides precise masks with temporal tracking
• CLIP embeds each masked region for semantic matching

The output is a 3D scene graph that can be queried in natural language — the ultimate semantic map.

# ConceptGraphs-style scene graph construction
import clip
import numpy as np

class SceneGraph:
    def __init__(self, merge_threshold=0.7):
        self.nodes = []  # list of ObjectNode
        self.merge_threshold = merge_threshold

    def add_observation(self, position_3d, clip_feature, label):
        # Check against existing nodes
        for node in self.nodes:
            spatial_dist = np.linalg.norm(node.centroid - position_3d)
            semantic_sim = np.dot(node.clip_feature, clip_feature)
            if spatial_dist < 0.5 and semantic_sim > self.merge_threshold:
                node.merge(position_3d, clip_feature)
                return
        # New object
        self.nodes.append(ObjectNode(position_3d, clip_feature, label))

    def query(self, text):
        # Natural language query
        text_feat = clip.encode_text(text)
        scores = [np.dot(n.clip_feature, text_feat) for n in self.nodes]
        return self.nodes[np.argmax(scores)]

Chapter 7: 3D Gaussian Splatting for Scene Representation

GAS v1 represented scenes as meshes (from GO-SLAM) and point clouds (from mask back-projection). These are the traditional representations for 3D mapping. But in 2024-2025, a new representation swept the field: 3D Gaussian Splatting (3DGS).

What are 3D Gaussians?

Instead of triangular meshes or discrete points, 3DGS represents a scene as millions of tiny 3D Gaussian blobs. Each Gaussian has:

• Position (μ) — where in 3D space
• Covariance (Σ) — shape and size (anisotropic ellipsoid)
• Opacity (α) — how transparent
• Color (spherical harmonics) — view-dependent appearance

Rendering is done by "splatting" these Gaussians onto the image plane — no ray marching, no neural network evaluation per ray. This makes rendering extremely fast: 100+ FPS at high resolution.

Why Gaussians for semantic mapping?

The key innovation for our purposes is feature splatting. Each Gaussian can carry not just color but arbitrary feature vectors — including CLIP embeddings, semantic labels, or instance IDs. You can render the scene from any viewpoint and get:

• Photorealistic RGB images
• Semantic segmentation maps
• Instance maps
• CLIP feature maps (for open-vocabulary queries)

Integration with GAS v2

# Semantic Gaussian Splatting pipeline
# 1. Initialize from VGGT pointmaps
pointmaps = vggt_model(images)["pointmap"]  # (N, H, W, 3)
poses = vggt_model(images)["extrinsic"]     # (N, 3, 4)

# 2. Initialize Gaussians from VGGT points
gaussians = initialize_gaussians(pointmaps, images)

# 3. Attach semantic features (from Grounding DINO + CLIP)
for i, frame in enumerate(frames):
    detections = grounding_dino.predict(frame, "door . chair . table")
    clip_features = clip.encode_regions(frame, detections.boxes)
    assign_features(gaussians, clip_features, poses[i])

# 4. Optimize with photometric + semantic loss
optimize_gaussians(gaussians, images, poses, semantic_labels)

# 5. Query: render semantic map from any viewpoint
semantic_img = render(gaussians, query_pose, mode="semantic")

Chapter 8: The Complete GAS v2 Pipeline

Let's put it all together. Here's the full GAS v2 data flow, from raw video to semantic map.

The data flow

1. Video → Frames — extract frames at 2-5 FPS (not every frame — too redundant)
2. Frames → VGGT — single forward pass produces camera poses, pointmaps, depth maps, and estimated intrinsics
3. Frames → Grounding DINO — open-vocabulary detection with text prompt describing target objects
4. Boxes → SAM 2 — generate pixel-precise masks and propagate with video tracking for consistent IDs
5. Masks + Depth → 3D Lifting — back-project masked regions using VGGT depth (or Depth Anything V2 fallback)
6. 3D Objects + CLIP → ConceptGraphs — build scene graph with semantic merging
7. Scene Graph → Output — render as 2D floorplan, 3D scene graph, or Gaussian splat scene

Input/output summary

Input: RGB video from any camera (phone, GoPro, webcam, robot). No depth sensor. No calibration pattern. No training data.

Output (choose one or more):

• 2D semantic floorplan (backward compatible with GAS v1)
• 3D scene graph queryable in natural language
• 3D Gaussian splat scene with semantic features
• Per-frame semantic segmentation maps

Chapter 9: Project Setup: From Zero to Running

This chapter is a practical guide. By the end, you'll have a working GAS v2 pipeline on your machine.

Hardware requirements

• Minimum: NVIDIA RTX 3090 (24GB VRAM) — runs each model sequentially. At 24GB you will be tight: expect a max of ~64 frames at ≤504px max side.
• Recommended: NVIDIA RTX 4090 (24GB) or A100 (40/80GB) — faster inference, can fit multiple models
• Cloud alternative: Modal gives you an A10G (24GB) at ~$1.10/hr, billed by the second. Full pipeline cost for a 55-second video is under $0.10. No local GPU required. See the Mirdan build log for the modal run recipe.
• CPU-only: Not practical. VGGT alone takes ~5 minutes per batch on CPU vs ~2 seconds on GPU
• Apple Silicon: Blocked. Grounding DINO's official CUDA op has no MPS fallback; VGGT and SAM 2 both have CUDA-preferred paths. Use the transformers GDINO port + MPS for a degraded-but-working dev loop, or go to the cloud.

Step 1: Environment

# Create conda environment
conda create -n gas-v2 python=3.11 -y
conda activate gas-v2

# PyTorch with CUDA
pip install torch torchvision torchaudio --index-url \
    https://download.pytorch.org/whl/cu121

# Core dependencies
pip install numpy scipy opencv-python pillow tqdm
pip install supervision  # Roboflow's annotation/viz toolkit

Step 2: Install models

# VGGT (3D geometry) — from source
pip install git+https://github.com/facebookresearch/vggt.git

# Grounding DINO — use transformers' port (pure PyTorch, no nvcc)
pip install "transformers>=4.44"
# At load time: AutoModelForZeroShotObjectDetection.from_pretrained(
#   "IDEA-Research/grounding-dino-base")

# SAM 2 (segmentation + tracking) — from source
pip install git+https://github.com/facebookresearch/sam2.git

# Rerun (for visualization)
pip install "rerun-sdk>=0.20"

# Open3D (for PLY + Poisson mesh export)
pip install "open3d>=0.18"

# CLIP (semantic features for ConceptGraphs — if you extend dedup)
pip install open-clip-torch

# IMPORTANT: re-pin torch LAST, after the above installs.
# SAM 2's deps will otherwise drag torch to a CUDA-13 wheel that breaks nvrtc.
pip install --upgrade --force-reinstall \
    torch==2.5.1 torchvision==0.20.1 \
    --index-url https://download.pytorch.org/whl/cu124

Step 3: Download weights

# Create weights directory
mkdir -p weights && cd weights

# VGGT (auto-downloaded on first use via HuggingFace)
# Grounding DINO
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth

# SAM 2
wget https://dl.fbaipublicfiles.com/segment_anything_2/sam2_hiera_large.pt

# Depth Anything V2 (optional)
wget https://huggingface.co/depth-anything/Depth-Anything-V2-Metric-Indoor-Large/resolve/main/depth_anything_v2_metric_indoor_vitl.pth

Step 4: Capture data

• iPhone (best for indoor): Use the Record3D app for synchronized RGB + LiDAR depth. Export as MP4 + depth frames
• GoPro / webcam: Record at 1080p, 30fps. Walk slowly, overlap views by 60-80%. Avoid fast rotations
• Tips: Move smoothly. Cover every corner. Look at each object from 2-3 angles. A 60-second walk-through of a room is usually sufficient

Step 5: Code structure

# Recommended project layout
gas-v2/
  main.py          # Orchestrator: load video, run pipeline
  slam.py          # VGGT or MASt3R-SLAM wrapper
  detect.py        # Grounding DINO + SAM 2 wrapper
  segment.py       # Mask processing, tracking
  merge.py         # ConceptGraphs scene graph builder
  visualize.py     # 2D floorplan + 3D rendering
  config.yaml      # Model paths, thresholds, prompt
  weights/         # Downloaded model weights
  data/            # Input videos
  output/          # Results

Step 6: Run the pipeline

# main.py - simplified orchestrator
from slam import run_vggt
from detect import detect_and_segment
from merge import build_scene_graph
from visualize import render_floorplan

# 1. Extract frames
frames = extract_frames("data/apartment.mp4", fps=3)

# 2. 3D geometry
geometry = run_vggt(frames)  # poses, depth, pointmaps

# 3. Detect + segment
objects = detect_and_segment(
    frames, prompt="door . chair . table . sofa . bed . window"
)

# 4. Build scene graph
scene_graph = build_scene_graph(objects, geometry)

# 5. Render output
render_floorplan(scene_graph, output="output/floorplan.png")
print(f"Found {len(scene_graph.nodes)} objects")

Chapter 10: Evaluation & Benchmarking

How do you know if your GAS v2 pipeline actually works? You need benchmarks with ground truth, metrics to measure quality, and baselines to compare against.

Datasets

• Replica (Facebook Research) — 18 photorealistic indoor scenes with dense ground-truth meshes, semantic labels, and camera trajectories. GAS v1 used this. Direct comparison possible
• ScanNet (Dai et al., 2017) — 1,513 real-world RGB-D scans with 3D semantic labels. The standard benchmark for 3D scene understanding
• ScanNet++ (Yeshwanth et al., 2023) — higher quality scans with DSLR images. Better for evaluating modern high-resolution models

Metrics

Each stage of the pipeline has its own evaluation metric:

• Pose accuracy: Absolute Trajectory Error (ATE) — average distance between estimated and ground-truth camera positions. VGGT achieves ~2cm ATE on Replica
• Detection: mAP@50 — mean average precision at IoU threshold 0.5. Grounding DINO: ~48 mAP@50 on COCO (zero-shot) vs Faster R-CNN fine-tuned: ~55 mAP@50 (but only on trained classes)
• Segmentation: mIoU — mean intersection-over-union across classes. SAM 2 with Grounding DINO prompts: ~72% on ScanNet semantic segmentation
• Mapping completeness: percentage of ground-truth objects successfully detected and placed in the scene graph. Depends heavily on video coverage and detection threshold

GAS v1 vs v2: expected comparison

Running evaluation

# Evaluate on Replica dataset
from eval import evaluate_pipeline

results = evaluate_pipeline(
    pipeline=gas_v2,
    dataset="replica",
    scenes=["office_0", "room_0", "apartment_0"],
    metrics=["ate", "mAP", "mIoU", "completeness"]
)

for scene, m in results.items():
    print(f"{scene}: ATE={m['ate']:.3f}m, mAP={m['mAP']:.1f}, mIoU={m['mIoU']:.1f}%")

Chapter 11: What's Next: Embodied Foundation Models

GAS v2 builds a semantic map. But a map is only useful if something acts on it. The next frontier is closing the loop: explore, map, reason, and act.

OK-Robot: pick-and-place from semantic maps

OK-Robot (Liu et al., 2024) demonstrates the end-to-end vision: give a home robot a natural language command ("bring me the coffee mug"), it uses an open-vocabulary semantic map to find the mug, plans a path to it, and picks it up. The semantic map is essentially a GAS v2-style scene graph.

SayPlan: LLM planning on 3D scene graphs

SayPlan (Rana et al., 2023) feeds a 3D scene graph (like ConceptGraphs output) directly to an LLM for task planning. The LLM reasons about spatial relationships: "To get to the kitchen, go through the hallway, turn left at the bathroom." The scene graph provides the grounding that prevents the LLM from hallucinating spatial facts.

The convergence

We're watching three fields merge into one:

• SLAM + foundation models = spatial intelligence (what GAS v2 builds)
• LLM planning on scene graphs = spatial reasoning (what SayPlan demonstrates)
• VLA execution = spatial action (what pi-0 and OK-Robot execute)

The pipeline is becoming: perceive (GAS v2) → reason (LLM + scene graph) → act (VLA). Each piece is a foundation model. Each piece is zero-shot. The system works in any environment without environment-specific training.

Related Veanors lessons

• VGGT → — the geometry backbone, CVPR 2025 Best Paper
• MASt3R-SLAM → — real-time dense SLAM with learned priors
• DUSt3R → — the predecessor to VGGT and MASt3R
• SAM 2 → — segment and track anything in video
• GAS v1 → — the original pipeline this lesson reimagines
• DINOv2 → — the universal visual backbone
• Depth Anything V2 → — monocular metric depth
• 3D Gaussian Splatting → — the new scene representation
• Theory of Space → — can foundation models build spatial beliefs?