Scale-MAE

Chapter 1: MAE in 90 Seconds

Scale-MAE is a masked autoencoder (He et al., 2022), so we need the MAE recipe crisp before we modify it. The idea: hide most of an image, ask a network to fill in the blanks, and the representations it learns along the way turn out to be excellent for downstream tasks.

Concretely, MAE works on a Vision Transformer. An input image of shape [H, W, 3] is split into non-overlapping patches of size p×p. For a 224×224 image with p = 16, that is (224/16)² = 14² = 196 patches, each a vector of 16·16·3 = 768 numbers. A large fraction of patches — MAE uses 75% — are masked (thrown away). Only the visible 25% (here, 49 patches) are fed to the encoder.

The asymmetry is the trick. The encoder is large and sees only 25% of patches — so it is cheap and forced to reason about global structure from sparse evidence. The decoder is small and lightweight; its only job is reconstruction during pre-training and it is discarded afterward. The loss is mean squared error on the masked patches only:

where M is the set of masked patch indices, x_i is the ground-truth pixels of patch i, and x̂_i is the decoder's prediction. Computing loss only on masked patches stops the model from cheating by copying visible patches through.

A 4×4 worked example of the masking bookkeeping

Take a tiny 4×4 patch grid (16 patches). With a 75% mask ratio we keep 16×0.25 = 4 patches. Suppose the random permutation keeps patches {2, 5, 11, 14}. The encoder receives a length-4 sequence. The decoder then builds a length-16 sequence: it places the 4 encoded tokens back at indices {2,5,11,14} and fills the other 12 slots with a single shared learned mask token. Positional encodings are added to all 16 so each knows where it belongs, and the decoder predicts pixels for all 16, but loss is summed only over the 12 masked indices.

Everything Scale-MAE changes lives in two spots of this pipeline: the positional encoding that gets added to tokens (Chapter 3) and the decoder target (Chapter 4). The encoder, the masking, the asymmetry — all untouched. So the upgrades are bolt-on, which is exactly why they are easy to adopt.

Chapter 2: Ground Sample Distance, Precisely

Before we can encode GSD, we must pin down what it is. Ground sample distance is the distance on the Earth's surface between the centres of two adjacent pixels, expressed in metres per pixel. If a satellite reports GSD = 0.5 m, then a 224-pixel-wide tile covers 224 × 0.5 = 112 metres of ground. A car 4.5 m long spans 4.5 / 0.5 = 9 pixels.

GSD is set by the sensor's geometry: focal length, detector pixel pitch, and altitude. From orbit it is essentially fixed per sensor; for aerial platforms it varies with altitude. The number we care about is purely "metres of world per pixel," and it is shipped as metadata alongside almost every serious geospatial image product.

The key relationship: ground coordinate = pixel index × GSD

Here is the equation Scale-MAE hinges on. A patch at grid index (r, c) — row r, column c — sits at ground position:

where p is the patch size in pixels. Two patches at the same grid index (r, c) but from tiles of different GSD are at different ground coordinates. That is the whole insight: a position encoding built on the grid index is scale-blind; a position encoding built on the ground coordinate is scale-aware. The conversion is a single multiply by GSD.

A worked numerical example

Take patch index c = 3, patch size p = 16. Compare two tiles:

Same grid index (48 px), but 14.4 m versus 480 m of ground — a 33× difference. A scale-blind encoder assigns both the identical positional vector; a GSD-aware encoder gives them vectors as different as their physical positions warrant. The drone tile's 196 patches span 0–108 m of ground; the satellite tile's 196 patches span 0–3600 m. The positional encoding must reflect that the satellite patches are spread across 33× more world.

Chapter 3: GSD Positional Encoding — Derived

Now the core mechanism. Recall the standard 2D sinusoidal positional encoding used by ViT and MAE. For a 1D position u and embedding dimension d, the encoding interleaves sines and cosines at geometrically spaced frequencies:

In a 2D image, ViT builds the encoding by computing a 1D encoding for the row coordinate and a 1D encoding for the column coordinate (each using half of d), then concatenating. Crucially, in standard ViT the position u is just the integer grid index: u = c for column, u = r for row. That is the scale-blind part — the same index always yields the same encoding.

Scale-MAE's modification: feed the ground coordinate, not the index

Scale-MAE replaces the grid index with the ground coordinate from Chapter 2. Instead of u = c, it uses u = c · (GSD / GSD_ref), where GSD_ref is a reference resolution that keeps the numbers in a sensible range. Equivalently, the paper scales the position fed to the sinusoid by the resolution ratio. Writing the scale factor as s = GSD / GSD_ref:

Read that carefully: multiplying the position by s is identical to multiplying every sinusoid's frequency by s. A coarse tile (large GSD → large s) gets higher-frequency positional sinusoids: each patch step covers more ground, so the encoding cycles faster across the grid. A fine tile (small GSD → small s) gets lower-frequency sinusoids: each step covers little ground, so the encoding changes slowly. The frequency of the position code now literally reports the resolution.

Worked: encoding column c=3 at two resolutions

Let d = 4 (so we have frequencies for i = 0 and i = 1) and pick GSD_ref = 1 m/px so s = GSD. The two frequency divisors are 10000^0/4 = 1 and 10000^2/4 = 100. For column c = 3:

Fine tile, GSD = 0.3 → s = 0.3, argument base = s·c = 0.9:

dim 0: sin(0.9 / 1) = sin(0.90) = 0.783
dim 1: cos(0.9 / 1) = cos(0.90) = 0.622
dim 2: sin(0.9 / 100) = sin(0.009) = 0.009
dim 3: cos(0.9 / 100) = cos(0.009) = 1.000

Coarse tile, GSD = 10 → s = 10, argument base = s·c = 30:

dim 0: sin(30 / 1) = sin(30.0) = −0.988
dim 1: cos(30 / 1) = cos(30.0) = 0.154
dim 2: sin(30 / 100) = sin(0.30) = 0.296
dim 3: cos(30 / 100) = cos(0.30) = 0.955

Same grid index c = 3, but the two encoding vectors are completely different — the coarse tile has wrapped many times around the high-frequency sinusoid (sin(30) is far into its cycles) while the fine tile barely moved. The encoder receives an unambiguous signal of which resolution it is looking at, per patch, for free.

Why this preserves MAE's nice property

Sinusoidal encodings have a celebrated trait: PE(u + k) is a fixed linear function of PE(u), which lets attention reason about relative offsets. Multiplying the argument by s does not break this — it just rescales what "one step" means. PE_scale(c + 1) is still a linear (rotation) function of PE_scale(c), with the rotation angle now proportional to s. So the model can still learn "attend k ground-steps away," and the size of a ground-step is set by GSD. Relative-position reasoning survives; it simply becomes resolution-relative.

python
# GSD-aware 1D sinusoidal positional encoding for one axis
import numpy as np

def gsd_pe_1d(n_pos, d, gsd, gsd_ref=1.0):
    """n_pos patch positions, embed dim d (even), resolution gsd."""
    s = gsd / gsd_ref                       # scale factor
    pos = (np.arange(n_pos) * s)[:, None]   # ground coordinate
    i   = np.arange(d // 2)[None, :]
    div = np.power(10000.0, 2 * i / d)        # geometric frequencies
    ang = pos / div                          # [n_pos, d/2]
    pe  = np.zeros((n_pos, d))
    pe[:, 0::2] = np.sin(ang)
    pe[:, 1::2] = np.cos(ang)
    return pe

# Same grid, different resolution -> different encoding
fine   = gsd_pe_1d(32, 64, gsd=0.3)
coarse = gsd_pe_1d(32, 64, gsd=10.0)
print(np.allclose(fine, coarse))   # False -- scale-aware!

Chapter 4: The Laplacian-Pyramid Decoder

The GSD positional encoding fixes the input side. The Laplacian decoder fixes the output side. The motivation: a single-scale pixel reconstruction target lets the model get away with modelling one band of spatial frequencies. We want representations that are good at every scale — coarse layout (where is the runway?) and fine texture (what is the road surface?). So we make the model reconstruct both simultaneously.

What is a Laplacian pyramid?

Start with an image I₀. Blur and downsample it to get I₁ (half resolution), again to get I₂, and so on — that chain is the Gaussian pyramid. The Laplacian pyramid stores the differences: each level L_k = I_k − upsample(I_k+1). L_k captures exactly the detail that level k has but level k+1 lost — a band of spatial frequency. The coarsest Gaussian level plus all the Laplacian bands losslessly reconstruct the original.

Intuitively: the bottom Laplacian level holds fine edges and texture (high frequency); higher levels hold progressively coarser structure (low frequency). By decomposing the reconstruction target this way, we can ask the decoder to produce a low-frequency reconstruction and a high-frequency reconstruction as separate heads, and grade each.

Scale-MAE's decoder structure

Scale-MAE replaces MAE's plain decoder with a multi-scale, progressive decoder. After the lightweight transformer decoder produces a feature map, it is split into two branches that build a small Laplacian pyramid of the reconstruction:

A subtle and important detail: the high-frequency branch targets a resolution higher than the masked input. Scale-MAE's decoder is super-resolving — it asks the encoder to support reconstructing detail finer than what it was given. This pushes the representation to encode information that generalizes to finer GSDs, which is precisely the cross-scale transfer we want. The low-frequency branch, conversely, targets a coarser version, anchoring global structure.

Worked: a 1D Laplacian split, by hand

Take the 8-sample signal I = [4, 6, 8, 6, 4, 2, 4, 6]. Blur it with a 3-tap moving average (reflect at edges) to get the low band G:

G[1] = (4+6+8)/3 = 6.00
G[2] = (6+8+6)/3 = 6.67
G[3] = (8+6+4)/3 = 6.00
G[4] = (6+4+2)/3 = 4.00
... (edges reflected)

The Laplacian residual is L = I − G:

L[2] = 8 − 6.67 = +1.33 (a local peak — high frequency)
L[3] = 6 − 6.00 = 0.00
L[4] = 4 − 4.00 = 0.00

where the signal is smooth, L is near zero; where it has a sharp local feature (the peak at index 2), L spikes. The decoder's high-frequency head is graded against exactly these L values, so it is forced to encode sharp local structure. The low-frequency head is graded against G. Reconstruction is G + L = I exactly.

Chapter 5: The Full Objective

We now have both pieces. Let's assemble the complete Scale-MAE forward pass and loss, with tensor shapes, so you could implement it.

Forward pass, end to end

The loss is a weighted sum of two reconstruction terms

The encoder's gradient comes from grading both pyramid bands. Let GT_lo be the ground-truth low-frequency (downsampled, blurred) image and GT_hi be the ground-truth high-frequency (Laplacian residual, at the super-resolved target size). The total loss is:

As in MAE, the reconstruction loss is taken over the masked regions. The two weights λ_lo and λ_hi balance how much the model invests in coarse structure versus fine texture; the paper finds both bands contribute, and an ablation (next chapter) shows that dropping either hurts downstream transfer. The takeaway: the loss explicitly grades the encoder on producing features useful at two distinct spatial-frequency bands, which is the mechanism behind scale-robust representations.

Worked: combining two band losses

Suppose on a batch the low-band MSE is L_lo = 0.040 and the high-band MSE is L_hi = 0.120 (the high band is typically harder, so larger). With equal weights λ_lo = λ_hi = 1.0:

L = 1.0 × 0.040 + 1.0 × 0.120 = 0.160

If instead we down-weight the noisy high band to λ_hi = 0.5 to stabilize early training:

L = 1.0 × 0.040 + 0.5 × 0.120 = 0.100

The gradient to the encoder is now weighted toward coarse structure early on, then you can anneal λ_hi up. The point of the worked numbers: the two bands enter the loss additively, each with its own knob, and the encoder receives the sum of both gradients.

python
# Scale-MAE total loss = weighted sum of two pyramid-band MSEs
import torch.nn.functional as F

def scale_mae_loss(recon_lo, gt_lo, recon_hi, gt_hi, mask,
                   lam_lo=1.0, lam_hi=1.0):
    # grade each band only on masked regions
    l_lo = (F.mse_loss(recon_lo, gt_lo, reduction='none') * mask).mean()
    l_hi = (F.mse_loss(recon_hi, gt_hi, reduction='none') * mask).mean()
    return lam_lo * l_lo + lam_hi * l_hi

Chapter 6: Results — Does Scale Awareness Pay Off?

Scale-MAE pre-trains a ViT on Functional Map of the World (fMoW) RGB imagery, then is evaluated by freezing the encoder and probing it on downstream geospatial tasks. The headline question: do scale-aware features transfer better, especially across resolutions the model did not see at the probed scale?

The evaluation that matters: cross-scale robustness

The paper's signature experiment evaluates linear-probe / kNN classification accuracy as the evaluation GSD is swept away from the pre-training distribution. A scale-blind model peaks near its training scale and falls off steeply as the test imagery gets coarser or finer. Scale-MAE's accuracy curve is markedly flatter — it holds up across a roughly 3.3× range of ground sample distances, the property the whole method was designed to produce.

Reed et al. report that across a suite of remote-sensing classification datasets, Scale-MAE's frozen representations beat the strongest prior self-supervised geospatial baselines (including SatMAE and a vanilla-MAE control) on average, with the largest gains exactly where you would predict: at evaluation resolutions far from the training resolution. On in-distribution scales the gain is modest; off-distribution it is large. That asymmetry is the evidence that the improvement comes from scale awareness specifically, not from generic better features.

Beyond classification

The scale-aware encoder also transfers to dense prediction. On semantic segmentation of aerial imagery, fine-tuning the Scale-MAE encoder yields competitive-to-improved mean IoU relative to scale-blind pre-training, again with the gains concentrated where train/test resolution differ. The pattern is consistent across tasks: when scale matters, encoding scale helps; when it does not, it is neutral.

Chapter 7: Ablations — Which Piece Does the Work?

A two-contribution paper invites the obvious question: does each piece matter, or is one carrying the result? Scale-MAE's ablations isolate the GSD positional encoding and the Laplacian decoder.

Ablation 1: remove the GSD positional encoding

Swap the GSD-scaled PE back to the standard scale-blind PE, keeping everything else. The cross-scale robustness curve sags toward the baseline shape — the off-distribution accuracy drops. This confirms that the flatness of the accuracy-vs-GSD curve is attributable to the positional encoding carrying resolution information. Without it, the model is back to "this index is always this vector," and it cannot distinguish a coarse tile from a fine one.

Ablation 2: remove the Laplacian / multi-scale decoder

Replace the two-band pyramid decoder with a single-scale pixel reconstruction (vanilla MAE decoder), keeping the GSD PE. Downstream transfer weakens, particularly the model's ability to produce features good at both coarse and fine tasks. The multi-scale target is what forces the encoder to allocate capacity across spatial-frequency bands; remove it and the encoder is free to specialize to one band.

The two are complementary, not redundant

The clean finding is that the best results need both. The GSD PE tells the model what resolution it is at (input side); the Laplacian decoder forces multi-scale feature quality (output side). They attack scale-robustness from two different ends, and the combination beats either alone. This is the ideal ablation outcome: each contribution is independently positive and they stack.

A worked sanity check on "stacking"

Suppose (schematically) the baseline cross-scale score is 0.55, adding GSD PE alone gives 0.64 (a +0.09 lift), adding Laplacian alone gives 0.61 (a +0.06 lift), and the full model reaches 0.70. If the effects were perfectly additive we would predict 0.55 + 0.09 + 0.06 = 0.70 — which matches, indicating the two contributions are near-independent and additive rather than redundant. If the full model had only reached 0.64, we would conclude they overlap; if it reached 0.75, we would conclude they synergize. The arithmetic is how you read an ablation table.

Chapter 8: Scale Explorer — The Whole Mechanism, Live

This is the payoff. The showcase lets you drive Scale-MAE's central mechanism end-to-end: pick a GSD, watch the ground coordinates of the patches stretch, watch the GSD-scaled positional encoding's frequency change, and watch a toy reconstruction split into its low- and high-frequency Laplacian bands. Everything you derived in Chapters 3–5 is wired together here.

Use the controls to sweep the resolution and the band weighting. Toggle between the scale-blind encoding (standard ViT/MAE) and the GSD-aware encoding to see, side by side, why the same grid index produces different position codes once GSD enters — and how the reconstruction target decomposes into bands the decoder grades separately.

Notice three things as you experiment. First, when GSD-aware is off, sliding GSD does nothing to the positional-encoding panel — that is the scale-blind failure mode the paper fixes. Turn it on and the encoding's stripe frequency now tracks GSD. Second, the ground ruler always stretches with GSD regardless of the toggle, because physics does not care what the encoder knows — only the encoder's awareness of it changes. Third, the band-emphasis slider redistributes energy between the coarse and fine reconstruction targets without changing their sum, exactly as the Laplacian decomposition guarantees.

Chapter 9: Limitations & Connections

Scale-MAE is elegant and effective, but it is not the end of the story. Knowing where it stops is part of understanding it.

Limitations

• RGB and single-time-step. The flagship Scale-MAE pre-trains on RGB fMoW imagery. Many remote-sensing payloads are multispectral, hyperspectral, SAR, or temporal stacks. Extending the GSD encoding cleanly across spectral bands of different native GSD (common on a single satellite) is non-trivial and largely left to follow-up work.
• GSD must be known and accurate. The whole mechanism assumes a reliable GSD value per tile. Mislabeled or missing metadata feeds the encoder the wrong scale signal. For airborne data with varying altitude, GSD can drift within a single collection.
• Absolute, not viewing-geometry-aware. GSD captures resolution but not off-nadir angle, terrain relief, or projection distortion. Two tiles at the same GSD but different look angles are still treated identically.
• Decoder design is hand-tuned. The number of pyramid levels, the super-resolution factor, and the band weights are design choices, not learned. Different tasks may want different settings.
• Still O(T²) attention. Scale-MAE inherits the Vision Transformer's quadratic attention cost; very large tiles remain expensive, independent of the scale-awareness contribution.

Connections — where this sits in the landscape

Scale-MAE is a focused modification of three ideas you can study deeper on this site:

Related directions worth chasing: rotary positional encoding (RoPE), which injects relative position into the attention dot-product and is a cousin of the "scale the position" idea; SatMAE, the temporal/spectral MAE baseline Scale-MAE compares against; and the broader multi-scale feature literature (feature pyramid networks, Laplacian pyramids in classical vision) that the decoder draws on. On this site, the Transformer veanor and the ViT gleam are the natural prerequisites; the diffusion lessons share the Laplacian/multi-scale theme from a different angle.

Cheat sheet