Foundation-Model Visual Place Recognition

Chapter 1: VPR as Retrieval

Strip away the vision and VPR is nearest-neighbour search in a vector space. Every database image i is reduced to a global descriptor g_i ∈ ℝ^d. The query image becomes g_q. We rank the database by similarity and return the top-K. If a returned image lies within a tolerance (typically 25 m) of the query's true position, it is a correct retrieval.

Because descriptors are L2-normalized (‖g‖ = 1), cosine similarity reduces to a plain dot product, and "most similar" equals "smallest Euclidean distance." So the global descriptor is everything: the search is trivial; the descriptor is the science.

The metric: Recall@K

VPR is scored with Recall@K: the fraction of queries for which at least one of the top-K retrieved images is a true match. Recall@1 is the strict version (the single best match must be correct); Recall@5 and Recall@10 are more forgiving, which matters because a downstream geometric verifier can re-rank a short list. A method that gets the right place into the top-5 ninety-five times out of a hundred is, in practice, excellent.

where 𝟙[·] is the indicator (1 if true, 0 if false), τ is the localization tolerance (25 m is standard for outdoor driving benchmarks), and pos is ground-truth position.

A worked Recall@K

Suppose a query's true place is database image #42, and #42 sits at GPS coordinate p. We compute cosine similarity to all database descriptors and sort:

Recall@1 = 0 for this query (rank 1 is a false match — a different street that looks similar). Recall@5 = 1 (a true match appears by rank 2). This is the typical story: the right place is usually in the short list, so Recall@5 and @10 are high while Recall@1 is the demanding number that separates strong methods from weak ones.

Chapter 2: DINOv2 Local Features

The shared engine of all three papers is DINOv2 (Oquab et al., 2023): a Vision Transformer self-supervised — no labels — on a curated 142M-image dataset, with a discriminative self-distillation objective (student/teacher DINO loss plus an iBOT masked-patch loss). The payoff for VPR is empirical and decisive: DINOv2 patch tokens behave like a dense, semantically meaningful, condition-robust local feature, without ever being trained for places.

From image to patch grid: the real data flow

Take a 224×224 RGB image. The ViT-L/14 backbone splits it into non-overlapping 14×14 patches, giving a 16×16 = 256 patch grid (plus one CLS token). Each patch is linearly embedded and passed through the transformer. The output we use is the last-layer patch tokens:

So one image becomes a set of N = 256 local descriptors, each D = 1024-dimensional. This is the raw material. (AnyLoc uses larger input crops, giving more patches; SALAD uses 322×322, giving N = 16×16... the exact grid depends on the crop, but the structure is identical: a grid of D-dim patch tokens.)

Which layer and which token?

AnyLoc ran a careful probe: which backbone, which layer, and which facet (the raw token, or the internal query/key/value projections) give the best VPR features? The headline findings:

A worked patch dot product

Take two tiny D = 3 patch tokens (real ones are 1024-dim; the arithmetic is identical). Patch a = [0.6, 0.0, 0.8] from a query window of a brick wall; patch b = [0.5, 0.1, 0.85] from the same wall at dusk; patch c = [-0.7, 0.7, 0.1] from foliage. L2-normalize, then dot:

cos(a, b) = (0.6·0.5 + 0 + 0.8·0.85) / (1 · √(0.25+0.01+0.7225)) = 0.98/0.991 ≈ 0.989 — same surface, near-identical even across the day→dusk shift. cos(a, c) = (-0.42 + 0 + 0.08) / (1 · 0.99) ≈ -0.343 — a wall patch and a foliage patch are far apart. The features cluster by what the patch is, not by lighting. This is the property the aggregator will exploit.

python
# DINOv2 patch tokens are the raw material for foundation-model VPR
import torch

dinov2 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2.eval()
for p in dinov2.parameters(): p.requires_grad_(False)   # FROZEN — no VPR training

img = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    out = dinov2.forward_features(img)
    patch_tokens = out['x_norm_patchtokens']   # [1, 256, 1024]

# patch_tokens[0] is a SET of 256 local descriptors, D=1024.
# Aggregating this set into ONE global vector is the whole problem.
print(patch_tokens.shape)   # torch.Size([1, 256, 1024])

Chapter 3: AnyLoc — VLAD on Frozen Features

AnyLoc's recipe is almost provocatively simple: take the frozen DINOv2 local descriptors and aggregate them with VLAD (Vector of Locally Aggregated Descriptors, Jégou et al., 2010) — a classical, unsupervised aggregation from the pre-deep-learning era. No triplet loss, no place labels, no training. And it generalizes everywhere NetVLAD does not.

VLAD, derived from first principles

VLAD answers: how do I turn a variable-size set of N local descriptors into one fixed-size global vector that preserves their distribution? The idea: build a small vocabulary of K cluster centers (a "visual vocabulary," learned once by k-means on a pile of descriptors), then for each cluster, record the sum of residuals — how the descriptors assigned to that cluster deviate from its center.

Let x_i ∈ ℝ^D be local descriptor i (i = 1..N), and c_k ∈ ℝ^D be cluster center k (k = 1..K). Hard-assign each descriptor to its nearest center; let a_ik = 1 if x_i's nearest center is c_k, else 0. Then the VLAD descriptor stacks K residual sums:

The full VLAD vector has dimension K·D. AnyLoc uses K = 32 clusters with D = 1536 (its chosen DINOv2 facet), giving a 32·... — actually a large vector that is then intra-normalized (each V_k block L2-normalized separately) and L2-normalized overall. Intra-normalization is the crucial trick: it stops one populous, bursty cluster (e.g. a huge patch of sky) from dominating the descriptor.

A fully worked VLAD (tiny example)

Take D = 2, K = 2, N = 3 descriptors. Centers c₁ = [0, 0], c₂ = [4, 4]. Descriptors x₁ = [0.2, 0.1], x₂ = [-0.1, 0.3], x₃ = [4.1, 3.9].

Assign: x₁, x₂ are nearest to c₁; x₃ is nearest to c₂. So a₁₁=a₂₁=1, a₃₂=1, all else 0.

Residual sums:
V₁ = (x₁−c₁) + (x₂−c₁) = [0.2, 0.1] + [-0.1, 0.3] = [0.1, 0.4].
V₂ = (x₃−c₂) = [4.1−4, 3.9−4] = [0.1, -0.1].

Intra-normalize: ‖V₁‖ = √(0.01+0.16) = 0.412, so V₁ → [0.243, 0.970]. ‖V₂‖ = √0.02 = 0.141, so V₂ → [0.707, -0.707].
Stack & global L2: V = [0.243, 0.970, 0.707, -0.707], ‖V‖ = √(0.059+0.941+0.5+0.5) = √2 = 1.414, final V = [0.172, 0.686, 0.500, -0.500].

Notice what intra-normalization bought us: cluster 2 had only one descriptor and a tiny residual, yet after intra-norm it contributes as strongly as cluster 1's two-descriptor residual. Each region of feature space gets an equal vote, regardless of how many patches landed there. That is exactly why sky and pavement don't drown out the distinctive facade.

Universal vs map-specific vocabulary

AnyLoc made one more domain-spanning observation. The k-means vocabulary can be built once on a diverse mix of images (a "universal" vocabulary, the AnyLoc-VLAD-DINOv2 config) and reused everywhere; or it can be built per-domain ("map-specific") for a small extra gain. The universal vocabulary is what makes AnyLoc truly training-free and plug-and-play — drop it on aerial imagery it has never seen, and it works.

AnyLoc also showed that the DINOv2 feature space, after this aggregation, exposes interpretable domain clusters: aerial, urban, indoor, and sub-surface images occupy distinct regions, which is why a single vocabulary can serve them all without collapsing them together.

python
# AnyLoc = frozen DINOv2 locals + classical hard-assignment VLAD
import numpy as np

def vlad(X, C):                  # X: [N, D] locals, C: [K, D] centers
    N, D = X.shape; K = C.shape[0]
    # hard assignment: nearest center per descriptor
    d2 = ((X[:, None, :] - C[None]) ** 2).sum(-1)   # [N, K]
    a = d2.argmin(1)                                 # [N]
    V = np.zeros((K, D))
    for i in range(N):
        V[a[i]] += X[i] - C[a[i]]                  # residual sum
    V = V / (np.linalg.norm(V, axis=1, keepdims=True) + 1e-9)  # intra-norm
    V = V.flatten()
    return V / (np.linalg.norm(V) + 1e-9)            # global L2 -> [K*D]

Chapter 4: SALAD — Optimal-Transport Assignment

AnyLoc proved frozen features are enough. SALAD (Izquierdo & Civera, CVPR 2024) — Sinkhorn Algorithm for Locally Aggregated Descriptors — asks: the aggregation is still a 14-year-old hard k-means assignment; what if we make the assignment itself soft, learned, and globally balanced? Their answer reframes "which descriptor goes to which cluster" as a problem of optimal transport, and it sets a new state of the art with a tiny trainable head on top of (mostly frozen) DINOv2.

The reframing: assignment is a transport plan

VLAD's hard assignment a_ik ∈ {0, 1} is a brutal commitment: descriptor i goes entirely to its single nearest cluster. A descriptor that sits between two clusters is forced to pick one, discarding information. Soft assignment (as in NetVLAD) lets a_ik ∈ [0, 1] spread across clusters — but row-wise softmax has no notion of balance: a single overwhelming cluster can soak up every descriptor.

SALAD instead treats the N×K assignment matrix as a transport plan P, where P_ik is "how much mass of descriptor i is sent to cluster k." It is the soft assignment with two-sided constraints:

The row constraint says every descriptor's mass is conserved (it must go somewhere). The column constraint is the new idea: each cluster has a capacity, so no single cluster can monopolize the descriptors. This two-sided balance is exactly what hard VLAD and plain softmax lack, and it is what optimal transport gives you.

The score matrix and the dustbin

A small learned head turns each local descriptor into a score against each cluster, producing the cost/score matrix S ∈ ℝ^N×K (high score = good match). SALAD adds one more column — a "dustbin" — a (K+1)-th destination that absorbs uninformative descriptors (dynamic objects, sky, motion blur) so they don't pollute the aggregation. This is the same trick SuperGlue uses for unmatched keypoints, borrowed into VPR.

Solving for a balanced P given S is the optimal-transport problem. There is no closed form, but an entropy-regularized version has a beautifully simple iterative solver — Sinkhorn — which we derive in the next chapter. For now: SALAD feeds the patch tokens through a tiny MLP to get S, runs a handful of Sinkhorn iterations to get a balanced assignment P, then aggregates exactly like VLAD (residuals weighted by P instead of hard a), discards the dustbin, and L2-normalizes. The aggregation is differentiable end-to-end, so the head learns assignments that maximize retrieval — but the heavy DINOv2 backbone stays (mostly) frozen.

Why two-sided balance helps retrieval

Consider an image where 60% of patches are sky. Hard VLAD or softmax dumps them all into the "sky" cluster, whose huge residual block (even after intra-norm) reflects almost nothing place-specific. SALAD's column constraint caps the sky cluster's intake and forces the model to spread informative patches across clusters, so the descriptor's capacity is spent on the parts of the scene that actually distinguish this place. Empirically this — plus the dustbin and the learned scores — is what lifts Recall@1 past AnyLoc and the older trained methods.

Chapter 5: Sinkhorn, Derived

Now the core mechanism. We want the balanced assignment P that best matches the learned scores S, subject to fixed row marginals r and column marginals c. This is entropy-regularized optimal transport, and Sinkhorn's algorithm solves it with nothing but row and column normalizations.

The optimization problem

Optimal transport finds the plan P that maximizes total matched score (equivalently minimizes a cost) while respecting both marginals. The raw OT problem is a linear program — expensive and non-differentiable at the vertices. Cuturi (2013) added an entropy regularizer H(P) = −∑_ik P_ik(log P_ik − 1), which makes the objective strictly convex, the solution smooth (great for backprop), and — the magic — admits a closed-form structure:

Here ε > 0 is the regularization strength. The Lagrangian (one multiplier per row constraint, u_i, and one per column, v_k) has gradient zero exactly when:

Read that carefully: the optimal P is the kernel matrix G (the exponentiated, temperature-scaled scores) scaled by a positive vector a on the rows and a positive vector b on the columns. We only need to find the scaling vectors a and b that make the marginals come out right. That is what Sinkhorn iterations do.

The algorithm: alternate row and column normalization

Start with the kernel G = exp(S/ε). Then repeatedly: rescale rows so each row sums to its target r_i; rescale columns so each column sums to its target c_k. Each step fixes one marginal and breaks the other slightly; iterating drives both to convergence (guaranteed for positive G).

and read off P_ik = a_i G_ik b_k. In practice this is done in log-space for numerical stability — instead of multiplying by a and b you add log-scalings, and the row/column normalizations become log-sum-exp subtractions. SuperGlue and SALAD both run the log-space variant for just a handful of iterations.

A fully worked Sinkhorn (2×2, by hand)

Let the kernel be G = [[1, 0], [0, 1]]-ish... use a concrete non-trivial case. Scores give kernel G = [[4, 1], [1, 2]] (already exponentiated for simplicity). Targets: uniform rows r = [1, 1], columns c = [1, 1] (we want a doubly-balanced plan; here we normalize to total mass 2).

Iteration 1 — row normalize (each row should sum to 1; here r_i=1 with 2 rows, total 2, but we'll target row-sum 1):
Row 0 sum = 4+1 = 5 → scale 1/5: [0.8, 0.2]. Row 1 sum = 1+2 = 3 → scale 1/3: [0.333, 0.667].
P = [[0.8, 0.2], [0.333, 0.667]].

Column normalize (each column should sum to 1):
Col 0 sum = 0.8+0.333 = 1.133 → scale 1/1.133: [0.706, 0.294]↓. Col 1 sum = 0.2+0.667 = 0.867 → scale 1/0.867: [0.231, 0.769]↓.
P = [[0.706, 0.231], [0.294, 0.769]].

Iteration 2 — row normalize again:
Row 0 sum = 0.937 → [0.753, 0.247]. Row 1 sum = 1.063 → [0.277, 0.723].
Column normalize: col 0 = 1.030 → [0.731, 0.269]; col 1 = 0.970 → [0.255, 0.745]. P = [[0.731, 0.255], [0.269, 0.745]].

After just two passes the plan is close to doubly-balanced (rows ≈ 0.986/1.014, columns within 1%) and it keeps the structure of G — entry (0,0) was the largest score and stays the largest mass (0.731), while the plan now respects both marginals. A few more iterations and the row/column sums are 1 to machine precision. That alternating normalize-rows / normalize-columns is the entire algorithm.

python
# Sinkhorn: balanced soft assignment by alternating normalization
import numpy as np

def sinkhorn(S, r, c, eps=0.6, iters=20):
    G = np.exp(S / eps)          # kernel: exponentiated scaled scores
    b = np.ones(S.shape[1])
    for _ in range(iters):
        a = r / (G @ b + 1e-9)    # row scaling: hit row marginals r
        b = c / (G.T @ a + 1e-9)  # col scaling: hit col marginals c
    return a[:, None] * G * b[None, :]   # P = diag(a) G diag(b)

# SALAD: S from a tiny head on frozen DINOv2 locals, last col = dustbin
P = sinkhorn(scores, r=row_targets, c=col_targets)
P = P[:, :-1]                        # drop the dustbin column
# aggregate residuals weighted by P, then L2-normalize -> global descriptor

Chapter 6: Results & Ablations

What did these design choices actually buy? The benchmarks that matter for VPR are MSLS (Mapillary Street-Level Sequences, including the hard MSLS-Challenge split), Pitts250k/Pitts30k (Pittsburgh street view), Nordland (extreme seasonal change, summer↔winter on a railway), Tokyo 24/7 (day/night), SPED, and AmsterTime. They probe different failure modes: viewpoint, season, illumination.

AnyLoc: generalization without training

AnyLoc's headline is qualitative and decisive. On structured, urban benchmarks that NetVLAD was trained for, AnyLoc is competitive. On out-of-distribution domains — aerial, indoor, underwater, night driving — where trained baselines collapse, AnyLoc roughly doubles the recall of prior methods. The paper reports up to a ~4× improvement in the very hardest unstructured cases. The lesson: trained-from-scratch VPR descriptors are sharp on their home turf and blind off it; frozen-foundation descriptors are broadly capable everywhere.

SALAD: state of the art with a light head

SALAD takes the standard MSLS / Pitts / Nordland / Tokyo24-7 benchmarks and sets a new state of the art, surpassing both AnyLoc and the strong trained baselines (e.g. MixVPR, EigenPlaces) — with a compact descriptor and a training recipe so light it converges in well under an hour on a single GPU, because only the small aggregation head (and a light backbone touch) is trained. The qualitative ablation findings the paper emphasizes:

A worked Recall@1 comparison

Imagine 1000 Nordland queries (summer query, winter database — the same track six months apart). A NetVLAD-style model trained on urban summer driving might get the exact winter frame at rank 1 for ~150 of them (Recall@1 ≈ 15%): winter strips the trees, dumps snow, and the trained texture features fall apart. A frozen-DINOv2 method holds structural cues (track curvature, signal masts, tunnel mouths) that survive the season, lifting many more queries to a correct rank-1. The delta is largest exactly where the appearance gap is largest — which is the whole reason foundation features matter.

Why the compact descriptor still wins

It is counterintuitive that SALAD's smaller descriptor beats AnyLoc's much larger VLAD vector. The resolution: descriptor dimensions are not all equally informative. AnyLoc's hard assignment and untrained centers waste capacity on redundant or sky-dominated blocks; SALAD's learned, balanced assignment makes every dimension earn its keep. This is a recurring theme in retrieval — a well-trained 4k-dim descriptor routinely beats a poorly-structured 32k-dim one. Quality of the assignment, not raw dimensionality, sets the ceiling.

Chapter 7: MegaLoc — One Retrieval Model

AnyLoc and SALAD focus on VPR. But "retrieve a similar image of a place" is one of three closely related retrieval tasks the robotics/vision stack actually needs:

Historically each task had its own specialist model trained on its own dataset. MegaLoc (Berton & Masone, 2025) asks: can one model be strong at all three? Its recipe is, again, frozen-foundation features plus a strong aggregation head — but the key contribution is the training mixture: a megamix of multiple datasets (street-view VPR data, landmark datasets, and localization-style data) trained with a multi-similarity / contrastive objective and careful mining, so the single descriptor is simultaneously good at coarse place recall, instance-level landmark matching, and the retrieval stage of a localization pipeline.

Why one model can serve all three

The tasks differ in their tolerance (25 m for VPR vs same-instance for landmarks) and their downstream consumer (a human-readable match vs a geometric pose solver), but they share the core need: a descriptor where "same physical place/object" lands close and "different" lands far. Training on a mixture forces the descriptor to satisfy all three notions of "same" at once. The result is a model that, at the time, matched or beat the per-task specialists on their own benchmarks — a single retrieval front-end for an entire localization stack.

Chapter 8: Assignment Explorer

This is the payoff. The single mechanism that ties the whole arc together is how patches get assigned to clusters before aggregation — hard (AnyLoc/VLAD) vs balanced optimal transport (SALAD/Sinkhorn). Below, drive both policies on the same scene and watch the resulting global descriptor change.

From assignment to descriptor, end to end

Once you have the assignment P (hard or balanced), the rest is mechanical and identical to VLAD: for each cluster k, sum the residuals of its assigned patches weighted by P_ik, intra-normalize each block, concatenate, L2-normalize. The descriptor you get out is the thing that gets stored in the database and compared by cosine at query time. The only thing the three papers really change is the assignment policy and where the cluster definitions / scores come from.

Set P to a one-hot hard assignment and c to k-means centers and you have AnyLoc. Set P to a Sinkhorn-balanced soft plan with a learned head and a dustbin and you have SALAD. Train the head on a megamix and you have MegaLoc. One equation, three papers.

Chapter 9: Limits & Connections

Foundation-model VPR is a clear win, but it is not magic. Knowing where it breaks is what separates using it from understanding it.

Limitations

The two-stage localization pipeline

In practice, foundation-VPR is the first stage of a hierarchical localizer. Stage 1 (retrieval, this lesson) narrows a 100k-image database to a top-K shortlist in milliseconds. Stage 2 (local matching, e.g. SuperPoint+SuperGlue or LightGlue, then PnP+RANSAC) does the heavy geometric work on just those K candidates to recover precise pose. The global descriptor doesn't need to be perfect — it needs to get the right place into the top-K, where the geometric verifier finishes the job. That is why Recall@5/@10 is the operationally relevant number.

Connections

Cheat sheet