Classification & Aggregation VPR

Chapter 1: VPR as Retrieval

Before we change how we train, let's nail how VPR is used at inference, because that constraint shapes every design decision. At test time there is no classifier and no labels — only nearest-neighbor search in descriptor space.

Formally, an image I is passed through a backbone CNN f (a ResNet, frozen-then-finetuned) to produce a feature map of shape [C, H, W] — C channels over an H×W spatial grid. An aggregation module g collapses that grid into a single vector and an L2 normalization makes it a unit vector:

Here D is the descriptor dimension — 512 is the canonical choice (compact enough that 10M descriptors fit comfortably in memory, expressive enough to separate places). Because every descriptor is unit-norm, the dot product between two descriptors equals their cosine similarity:

At inference, the query descriptor is compared against every database descriptor by this dot product, and the database images are ranked by similarity. The top-ranked image's GPS is the predicted location. Performance is reported as Recall@N (R@N): the fraction of queries for which at least one of the top-N retrieved images is within a threshold distance (commonly 25 m) of the true location.

Let's do a tiny worked example. Suppose D = 3 and we have unit descriptors:

The match m1 wins because its descriptor points in almost the same direction as the query. Notice m2 is geographically nearby but faces a different way — its descriptor is only mildly similar. That single fact (same GPS, different appearance) is the wedge EigenPlaces will later drive into.

Chapter 2: GeoClassification — VPR Without Mining

CosPlace's central move: stop training on pairs/triplets and instead train on a classification proxy. The label space of VPR is continuous — every image has a (latitude, longitude, heading). CosPlace discretizes that continuous space into a finite set of classes, then trains the backbone to classify which class an image belongs to. The classifier head is thrown away at the end; what we keep is the backbone that learned descriptors which separate the classes.

How is a class defined? By a cell in geographic space. The world is divided into a grid of square cells of side length M meters (e.g. M = 10 m). The heading (compass) is divided into bins of α degrees (e.g. α = 30°). A class is one (cell, heading-bin) pair — all images taken roughly from the same spot, facing roughly the same direction.

The heading bin matters a great deal. Two photos taken at the same GPS but facing opposite directions share no visual content — making them the same class would force the network to map two unrelated scenes to one label, which is impossible and harms training. Splitting by heading keeps each class visually coherent: same place, same direction, genuinely similar pictures.

Worked example. Suppose an image is at east = 327 m, north = 154 m (relative to a city origin), heading = 71°, with M = 10 m and α = 30°:

• east cell index = ⌊ 327 / 10 ⌋ = 32
• north cell index = ⌊ 154 / 10 ⌋ = 15
• heading bin = ⌊ 71 / 30 ⌋ = 2 (the 60–90° wedge)
• class = (32, 15, 2)

Now training is a plain classification problem with cross-entropy: feed an image, predict its class, backprop. No mining, no triplets, no per-step database search. Each image's label is fixed for the whole run — computed once from its metadata. This is the property that lets CosPlace train on San Francisco eXtra Large (SF-XL), a dataset of tens of millions of images covering the entire city.

Chapter 3: The CosFace Large-Margin Loss

This is the mathematical heart of CosPlace, borrowed from face recognition (Wang et al., CosFace / Large Margin Cosine Loss, 2018). We derive it from ordinary softmax in four steps, because each step removes a specific weakness.

Step 0: ordinary softmax cross-entropy

A linear classifier has a weight vector W_j per class j and a bias b_j. The logit for class j on a descriptor x (the backbone output before normalization) is z_j = W_j^Tx + b_j. The softmax loss for an example with true class y is:

Step 1: drop the bias, factor the logit into magnitude and angle

Set b_j = 0. A dot product equals the product of norms times the cosine of the angle between the vectors:

where θ_j is the angle between the descriptor x and the class direction W_j. The problem: the loss can be reduced by inflating the norms ‖W_j‖ or ‖x‖ instead of improving the angle — but at test time, only the angle (cosine) is used. The training objective and the test metric disagree.

Step 2: normalize away the magnitudes

Force ‖W_j‖ = 1 (normalize each class weight) and ‖x‖ = s (rescale every descriptor to a fixed radius s). Now the logit is purely the cosine, scaled by s:

The constant s (the scale, e.g. s = 30) is needed because cosines live in [−1, 1]; without scaling, the largest possible logit gap is 2, and softmax can never become confident. Multiplying by s widens the usable range of logits so the softmax can sharpen.

Step 3: subtract a margin on the true class

Here is the CosFace idea. We make the network's job harder for the correct class by subtracting a fixed margin m from its cosine before the softmax:

To classify the example correctly, the network can no longer settle for cosθ_y just barely exceeding the other cosines. It must satisfy:

The correct-class cosine must beat every other cosine by at least m. This carves an angular decision margin between classes. The descriptors of one place are pulled into a tight cone and a buffer of width set by m is left empty around it — exactly the compactness that retrieval needs.

Worked numerical example. Take s = 30, m = 0.35, and a 3-class problem. Suppose for some descriptor the cosines to the three class directions are cosθ = [0.70 (true class y=0), 0.55, 0.20].

• Plain softmax logits: s·cos = [21.0, 16.5, 6.0]. The true class already wins by 4.5 — loss is small, learning nearly stops.
• CosFace logit for y: s·(0.70 − 0.35) = s·0.35 = 10.5. Other logits unchanged: [10.5, 16.5, 6.0].
• Now class 1 (logit 16.5) outscores the true class (10.5)! The example is "misclassified" under the margin, so the loss is large and the gradient keeps pushing cosθ₀ up and cosθ₁ down until the true cosine clears the margin: 0.70 must rise above 0.55 + 0.35 = 0.90.

So the margin keeps the gradient alive long after a plain softmax would have given up, relentlessly compacting each class. Let's see the geometry.

python
import torch
import torch.nn.functional as F

def cosface_loss(x, W, labels, s=30.0, m=0.35):
    """
    x:      [B, D]   backbone descriptors (pre-norm)
    W:      [D, C]   class direction matrix
    labels: [B]      integer class ids
    """
    x  = F.normalize(x, dim=1)        # put descriptors on unit sphere
    Wn = F.normalize(W, dim=0)        # unit class directions
    cos = x @ Wn                        # [B, C] = cos(theta) to every class

    # subtract margin ONLY from the true-class column
    onehot = F.one_hot(labels, cos.size(1)).float()
    logits = s * (cos - m * onehot)    # [B, C]
    return F.cross_entropy(logits, labels)

Chapter 4: CosPlace Groups — Training at City Scale

GeoClassification plus CosFace gives a beautiful objective, but there's a practical wall: SF-XL has hundreds of thousands of classes. The CosFace classifier head is a matrix W of shape [D, C]. With D = 512 and C in the hundreds of thousands, that single layer is enormous, and worse, every class direction is updated on every batch through the softmax denominator, which is both slow and memory-hungry.

CosPlace's fix is the CosPlace Group. The key observation: adjacent cells in the grid are visually correlated (a photo in cell (32,15) and one in the neighbor cell (33,15) might overlap). If two classes can appear in the same training batch and they're nearly the same scene, the margin loss fights itself. So CosPlace partitions the grid so that classes within one group are guaranteed far apart, and trains on one group at a time.

Concretely, group membership is defined by the cell indices modulo a spacing N:

With N = 2, this creates groups in a checkerboard pattern: group (0,0) contains cells (0,0), (0,2), (2,0), (2,2), … — every class in this group is at least N×M meters from every other, so no two classes in a group can be confused by overlap. Each group has far fewer classes than the full set, so its CosFace head is small. Training cycles through groups round-robin: load group's classifier, train some iterations, save it, move to the next group.

Let's count. Say the city covers 1000×1000 cells (M = 10 m → 10×10 km) with 12 heading bins (α = 30°). That's 1000×1000×12 = 12 million classes total. With N = 2, there are N×N = 4 spatial groups (times heading), so each group holds about 12M / 4 = 3 million classes — and at any moment only one group's head (3M, not 12M directions) is in memory and being updated. The backbone, shared across all groups, accumulates everything it learns from each group, while no single CosFace head ever has to be the full size.

Chapter 5: EigenPlaces — Viewpoint Robustness

CosPlace works beautifully when the query and database images share a viewpoint. But a fundamental VPR failure mode remains: lateral viewpoint shift. A car-mounted camera drives down a street facing forward; later a pedestrian photographs the same building from across the street, facing sideways. Same place, radically different image. CosPlace's heading-bin split actually worsens this — by definition it puts the two views in different classes, so the loss never directly tells the network they depict one place.

EigenPlaces (Berton et al., ICCV 2023) fixes this at the level of how classes are constructed. The idea: instead of one class per (cell, heading), build classes that deliberately span multiple viewpoints of the same physical location, so that during training the margin loss is forced to pull together images of one place taken from genuinely different angles.

The eigen-view: finding the street direction

Where does "eigen" come from? For each cell, EigenPlaces collects the 2D ground positions of all cameras that took images there. Those positions, for a typical street, lie roughly along a line — the street. EigenPlaces runs PCA on the camera positions: the top eigenvector of their covariance is the dominant axis of camera motion (the street direction). That direction defines two canonical viewpoints to look at the place from:

• Frontal view: cameras looking along the principal axis (down the street).
• Lateral view: cameras looking along the orthogonal axis (across the street, at the facades).

Mechanically: take the cameras in a cell, compute the covariance of their (x, y) positions, eigendecompose it. The eigenvector with the larger eigenvalue is the street axis e₁; the smaller is e₂, the across-street axis. EigenPlaces then defines a focal point — a target 3D location a fixed distance out along e₂ — and groups together all images whose camera is positioned-and-oriented to be looking toward that focal point, regardless of which street segment the camera sits on.

Worked example of the PCA step. Suppose four cameras sit at positions (0,0), (2,0.1), (4,−0.1), (6,0). Their mean is (3, 0). The mean-centered points are (−3,0), (−1,0.1), (1,−0.1), (3,0). The covariance:

The large diagonal entry (5.0) is along x and the tiny one (0.005) along y, so the top eigenvector is essentially e₁ ≈ (1, 0) — the street runs east–west, exactly as the points suggest. The across-street axis is e₂ ≈ (0, 1). A lateral class then collects every camera (on any nearby street segment) oriented to look in the ±e₂ direction at a focal point — a building facade. Forced into one class, the frontal-driving and lateral-photographing views of that facade get pulled together by the margin loss.

Chapter 6: MixVPR — Feature-Mixing Aggregation

CosPlace and EigenPlaces are about training. MixVPR (Ali-bey et al., WACV 2023) is about a different component entirely: the aggregation g that turns the CNN feature map into the descriptor. This is the step that NetVLAD, GeM pooling, and average pooling all try to do. MixVPR replaces it with a stack of MLPs and beats them all while being smaller.

The input: a stack of feature maps

Run an image through a backbone (ResNet-50, truncated) and take an intermediate feature map of shape [C, H, W] — for example [C=1024, H=20, W=20]. MixVPR's first move is to view this not as one 3D tensor but as C feature maps, each a flattened H·W vector of length P = H×W (here 400). Reshape to a matrix:

The Feature-Mixer: an MLP across space

The heart of MixVPR is the Feature-Mixer block, lifted from the MLP-Mixer idea (Tolstikhin 2021). A Feature-Mixer is a tiny two-layer MLP applied across the spatial dimension P, shared across all C channels, wrapped in a residual connection:

Read it carefully. LN is layer norm. We transpose so the MLP mixes along P (the H·W positions): W₁ is [P′, P], σ is ReLU/GELU, W₂ is [P, P′]. The MLP takes a length-P vector (one channel's spatial map) and returns a length-P vector — a learned, global recombination of every spatial location. The same MLP weights are applied to all C channels. The residual F + means each block refines rather than replaces.

Why mix across space? A convolution only ever combines a local neighborhood. NetVLAD aggregates by soft-assigning each location to cluster centers. The Feature-Mixer instead lets every spatial position talk to every other through a dense MLP — a global, content-adaptive aggregation. Crucially it operates on the flattened map, so it discards the rigid 2D grid and learns which spatial regions matter for distinguishing places (e.g. building facades over sky/road).

MixVPR stacks L Feature-Mixer blocks (e.g. L = 4). Because each is residual and shares weights across channels, the parameter count stays modest. After the stack, two linear projections compress the [C, P] matrix to the final descriptor:

W_depth projects the channel dimension C → C′ (e.g. 1024 → 1024 or fewer), W_row projects the spatial dimension P → R (e.g. 400 → 4). The result is a [C′, R] matrix, flattened and normalized into the descriptor of dimension D = C′·R (e.g. 1024×4 = 4096, or projected down to 512). No clustering, no parameters-per-cluster, just MLPs.

python
import torch
import torch.nn as nn
import torch.nn.functional as F

class FeatureMixer(nn.Module):
    # a 2-layer MLP that mixes ACROSS the P=H*W spatial positions,
    # shared across all channels, with a residual connection
    def __init__(self, P, hidden):
        super().__init__()
        self.ln = nn.LayerNorm(P)
        self.fc1 = nn.Linear(P, hidden)
        self.fc2 = nn.Linear(hidden, P)

    def forward(self, F_):           # F_: [B, C, P]
        h = self.ln(F_)
        h = self.fc2(F.relu(self.fc1(h)))  # mix along P
        return F_ + h                  # residual

class MixVPR(nn.Module):
    def __init__(self, C, P, L=4, hidden=512, c_out=1024, r_out=4):
        super().__init__()
        self.mixers = nn.Sequential(*[FeatureMixer(P, hidden) for _ in range(L)])
        self.depth = nn.Linear(C, c_out)   # project channels  C -> c_out
        self.row   = nn.Linear(P, r_out)   # project positions P -> r_out

    def forward(self, fmap):          # fmap: [B, C, H, W]
        B, C, H, Wd = fmap.shape
        F_ = fmap.flatten(2)              # [B, C, P]  P = H*W
        F_ = self.mixers(F_)              # L Feature-Mixer blocks
        F_ = self.depth(F_.transpose(1,2)).transpose(1,2)  # [B, c_out, P]
        F_ = self.row(F_)                 # [B, c_out, r_out]
        d = F.normalize(F_.flatten(1), dim=1)  # [B, c_out*r_out]
        return d

Chapter 7: Results & Ablations

What did these three papers actually buy? Let's read the headline numbers and the ablations that justify each design choice. (Exact figures depend on backbone and benchmark; the relationships below are the robust, reported findings.)

CosPlace

CosPlace's two claims are scale and compactness. It introduced SF-XL (tens of millions of images, the whole of San Francisco) and showed that classification-without-mining trains efficiently at that scale where triplet mining becomes impractical. Critically, it reached state-of-the-art recall with a compact 512-D descriptor using a small ResNet-18 backbone that fits in under 4 GB of VRAM — and improved further with ResNet-50. Trained on one city, it generalized strongly to entirely different test sets (Pitts250k, Tokyo, etc.), evidence that GeoClassification learns transferable place features, not city-specific shortcuts.

EigenPlaces

EigenPlaces' claim is viewpoint robustness at no extra inference cost. By rebuilding classes around PCA-derived focal points, it improved recall over CosPlace specifically on benchmarks with strong lateral viewpoint variation, while keeping the same compact descriptor, the same CosFace loss, and the same fast classification training. It became a standard strong baseline for the 2023 generation of VPR systems.

MixVPR

MixVPR's claim is better aggregation, fewer parameters. With a ResNet-50 backbone and the Feature-Mixer stack, it reported recall@1 around 94.6% on Pitts250k-test, 88.0% on MapillarySLS, and strong results on the very hard Nordland (seasonal change) benchmark — outperforming NetVLAD and CosPlace while using less than half the parameters of those aggregations. The win comes from replacing cluster-based or pooling aggregation with global feature mixing.

The ablations that matter

• Margin m (CosPlace): m = 0 (plain softmax) underperforms; recall rises as m grows to a sweet spot (around 0.4) then falls if m is too large (the constraint becomes unsatisfiable and training destabilizes). This is direct evidence that the compactness from the margin, not mere classification, is what helps retrieval.
• Group spacing N (CosPlace): N = 1 (no separation guarantee) is worse; N ≥ 2 ensures non-overlapping classes per group and recovers performance. Confirms the partition is a correctness device.
• Heading vs viewpoint classes (EigenPlaces): swapping raw heading bins for PCA focal-point classes is the entire delta — same loss, same descriptor, higher recall under viewpoint change.
• Number of Feature-Mixer blocks L (MixVPR): recall improves as L increases up to a few blocks (the residual stack adds global mixing capacity), then saturates — you don't need many.
• Mixing vs pooling (MixVPR): replacing the Feature-Mixer with average/GeM pooling drops recall sharply, isolating the global spatial mixing as the source of the gain.

Chapter 8: VPR Studio — Train & Retrieve Live

This is the payoff. Below is a complete, live VPR pipeline on a toy 2D "city." Database images are 2D descriptors on the unit circle, colored by the place they belong to. You control the CosFace margin m and scale s, run gradient steps to see the classes compact under the margin, then issue a query and watch nearest-neighbor retrieval rank the database. Everything you learned is here at once.

Things to try, and what they reveal:

• Train with m = 0 for many steps: classes separate but stay loose; a query near a boundary often retrieves the wrong place. This is the plain-softmax failure CosFace fixes.
• Train with m = 0.5: the classes snap into tight cones with visible empty gaps. Boundary queries now retrieve correctly — the margin bought robustness.
• Crank s up: the green retrieval bars sharpen (softmax becomes confident); too low and even a clear match looks ambiguous. This is why CosFace needs the scale.
• Query repeatedly: R@1 (top bar correct) improves with training and with a sensible margin — you are watching the metric the papers report, live.

Chapter 9: Limitations & Connections

These three papers define the modern classification-and-aggregation lineage of VPR, but they are not the end of the story. Honest limitations:

• Global descriptor ceiling. A single vector per image is fast but discards spatial detail. The hardest VPR cases (severe perceptual aliasing — two different places that look nearly identical) need a second stage: re-rank the top-k retrievals with local-feature geometric verification (SuperGlue, Patch-NetVLAD, or transformer re-rankers). CosPlace/MixVPR are the recall stage, not the final word.
• Appearance shifts. Day→night, summer→winter (Nordland) remain hard. The label-construction tricks help with viewpoint but not illumination/season; that is where foundation-model backbones (DINOv2) and explicit domain augmentation come in.
• Label noise from GPS/heading. The whole GeoClassification scheme trusts the metadata to define classes. Bad GPS or compass corrupts labels.
• Fixed grid granularity. Cell size M is a hyperparameter; the right granularity differs between a dense downtown and a sparse highway.

Where this connects

The CosFace margin and the feature-mixing aggregator both reuse ideas you can study elsewhere on Engineermaxxing:

• The Feature-Mixer is the MLP-Mixer / attention idea applied to spatial positions — the self-attention foundation is in Attention Is All You Need (this Veanors section), and the architectural intuition in the Transformer Gleam and Universal Architecture.
• The cosine-similarity retrieval machine is exactly embeddings + nearest neighbor: see Vector Embeddings, Similarity Metrics, and Vector Databases (the index that makes 10M-descriptor search fast).
• The contrastive / metric-learning ancestor of the margin loss is in Contrastive Learning & CLIP.
• The place-recognition-as-loop-closure use case lives in Classical SLAM and Modern SLAM.

Cheat sheet

"What I cannot create, I do not understand." — You have now created the CosFace loss and the MixVPR mixer by hand. You understand them.