SatMAE: Masked Autoencoders for Satellite Imagery

Chapter 1: MAE in 90 Seconds (the part SatMAE keeps)

SatMAE is a modification of MAE, so we must first nail the base recipe. The Masked Autoencoder is an asymmetric encoder-decoder that learns by filling in the blanks.

Start with an image of resolution H×W. A Vision Transformer chops it into a grid of non-overlapping patches — say 16×16 pixels each — giving N = (H/16)(W/16) patch tokens. For a 224×224 image that is N = 14×14 = 196 patches. Each patch is flattened and linearly projected to a token of dimension D (e.g. D = 1024 for ViT-Large).

Now the masking. Choose a high masking ratio ρ (MAE uses ρ = 0.75). Sample a random subset of the N tokens to keep visible; the rest are masked. The numbers matter:

The asymmetry is the key engineering decision: the encoder sees only the visible tokens. With ρ = 0.75, the encoder processes only N_vis = 0.25N = 49 tokens instead of 196. Attention is O(N²), so processing a quarter of the tokens makes the encoder roughly 16× cheaper — this is why MAE can afford a huge encoder and pre-train fast.

After the encoder, a shared learned [MASK] token is inserted at every masked position (so the decoder sees a full-length sequence of N tokens again), positional embeddings are added back, and a small, shallow decoder reconstructs the pixels of the masked patches. Then — crucially — the decoder is discarded. Only the encoder survives into fine-tuning.

Worked example. Take a 96×96 satellite chip with patch size p = 8. Then the grid is (96/8)×(96/8) = 12×12 = 144 patches. With ρ = 0.75: N_vis = 0.25×144 = 36 visible tokens and N_mask = 108 masked tokens. The encoder runs self-attention over 36 tokens (36² = 1296 attention entries) instead of 144 (144² = 20,736) — a 16× reduction. The decoder, by contrast, must process all 144 positions, which is why it is kept deliberately small.

SatMAE inherits this entire skeleton: asymmetric encoder-decoder, high mask ratio, predict-the-masked-pixels objective, discard-the-decoder. What it changes is how the tokens are formed (Chapters 3-4) and how the mask is chosen (Chapter 5). Everything else is MAE.

Chapter 2: The Reconstruction Loss, Derived

The entire learning signal in SatMAE (and MAE) comes from one loss. Get this exactly right and the rest of the paper is plumbing. The objective is a mean squared error between predicted and true pixels, computed on the masked patches only. Let us build it up term by term.

Let the decoder output, for each patch i, a vector of predicted pixel values x̂_i ∈ ℝ^P, where P = p·p·C is the number of pixels-times-channels in a patch (for a 16×16 RGB patch, P = 16·16·3 = 768). Let x_i be the true patch pixels. The per-element squared error for patch i is:

This is just the average squared pixel error for one patch. Now the two design choices that make it the MAE loss rather than a generic autoencoder loss.

Choice 1: Loss on masked patches only

Let M be the set of masked patch indices (|M| = N_mask). The total loss averages ℓ_i over the masked set, and over no others:

Why only masked patches? The visible patches were handed to the encoder unchanged — reconstructing them is trivial copying and provides no learning pressure. Including them in the loss would let the model lower its loss by perfecting the easy copy task while neglecting the hard prediction task. MAE found that computing loss on visible patches hurts the learned representation. So the loss looks only where the model had to actually predict.

Choice 2: Normalize the target pixels per patch

MAE (and SatMAE) found that reconstructing normalized pixels gives a better representation. For each patch, compute the mean μ_i and standard deviation σ_i of its true pixels, and ask the decoder to predict the standardized target:

Per-patch normalization makes the target locally contrastive: it strips out each patch's overall brightness and contrast, so the model must predict relative structure (texture, edges) rather than just the average tone. For satellite bands this matters even more — different bands and different scenes have wildly different absolute reflectance ranges, and per-patch standardization equalizes them before the loss.

Worked numerical example

Take a toy with 4 patches, P = 2 pixels each. Suppose patches 1 and 3 are masked (M = {1, 3}), patches 0 and 2 are visible. Predictions and targets (already normalized for simplicity):

The masked-only loss is L = (0.50 + 0.125) / 2 = 0.3125. Note that patch 0's error of 0.04 and patch 2's error of 0.01 are not in the sum at all — the gradient flows only from patches 1 and 3. If we had (wrongly) averaged all four, we would get (0.04+0.50+0.01+0.125)/4 = 0.169 — a smaller, but meaningless, number diluted by the visible patches.

python
# The MAE / SatMAE reconstruction loss
import torch

def mae_loss(pred, target, mask, norm_pix=True, eps=1e-6):
    """
    pred   : [B, N, P]  decoder output per patch (P = p*p*C)
    target : [B, N, P]  true patch pixels
    mask   : [B, N]     1 = masked (predict it), 0 = visible (ignore)
    """
    if norm_pix:                                  # Choice 2: per-patch standardize
        mu  = target.mean(dim=-1, keepdim=True)
        var = target.var(dim=-1, keepdim=True)
        target = (target - mu) / (var + eps).sqrt()

    se   = (pred - target) ** 2              # [B, N, P]
    per_patch = se.mean(dim=-1)            # [B, N]  -> l_i

    # Choice 1: average over MASKED patches only
    loss = (per_patch * mask).sum() / mask.sum()
    return loss

Chapter 3: Temporal SatMAE

Now the first satellite-specific contribution. A field is not a single image; it is a time series of images of the same place. SatMAE's temporal variant feeds the model a stack of T timestamps of the same scene and must answer two questions: how do we tell the model when each patch was taken, and how do we mask across time?

Telling the model "when": the temporal embedding

Plain ViT adds a positional embedding to each token so the model knows where a patch is. With multiple dates, the model also needs to know when. SatMAE adds a temporal embedding on top of the spatial position. Crucially, the timestamp is not just an index 0,1,2 — it is the actual acquisition date, because the gap between images is irregular (one image in March, the next in June, the next two weeks later).

SatMAE encodes each component of the date — year, month, and hour/day-of-year — with its own sinusoidal positional encoding (the same sin/cos-at-many-frequencies scheme as the original Transformer), then concatenates them into the temporal embedding. So a token's full embedding is:

where s indexes spatial position and t indexes the date. The same spatial patch at two different dates gets the same pos_embed(s) but a different time_embed — so the model can line up "this is the same field, three months later." Encoding the real date (not the index) means the model learns calendar structure: it can tell that the gap from March to June is larger than March to April, and that June crops differ from March bare soil.

Masking across time: independent, not shared

Here is the second temporal choice, and it mirrors the spectral choice we will see in Chapter 5. Given T frames of the same scene, do we mask the same spatial patches in every frame (shared mask) or different patches in each frame (independent mask)?

SatMAE masks independently across time. If patch (s=12) is masked in the March frame, it might be visible in the June frame. Why? Because a shared mask creates a shortcut: if patch 12 is masked in every frame, the model genuinely has to predict it from spatial neighbours. But that is the easy spatial-interpolation task again. With independent masking, patch 12 is hidden in March but visible in June — so the model can (and is encouraged to) use the temporal neighbour: "what did this field look like at the same location a few months ago?" Independent masking is what forces the encoder to learn temporal correspondence rather than purely spatial inpainting.

Worked example. Suppose a scene has T = 3 frames, each a 12×12 = 144-patch grid, and ρ = 0.75. Under independent masking we draw a fresh 75%-mask per frame, so per frame 36 patches are visible. Across the stack of 3×144 = 432 tokens, 3×36 = 108 are visible to the encoder and 324 are masked. A given ground location (one of the 144 spatial cells) is, on average, visible in 3×0.25 = 0.75 of its 3 frames — so most locations are visible in at least one frame, giving the model a temporal anchor to predict the others. Under a shared mask, a masked location is masked in all 3 frames, destroying that anchor.

Chapter 4: Spectral Band Groups

This is the headline mechanism for multi-spectral SatMAE, and the one most worth your attention. The question: a Sentinel-2 pixel has 13 bands — how should the model ingest them?

The naive baseline: one fat patch

The obvious move is to treat the 13 bands like 13 colour channels: a 16×16 patch becomes a vector of 16·16·13 = 3328 numbers, projected by a single learned matrix to a token. This is the "flat 13-channel MAE." It is simple, and it is what you'd do first. But it has a structural flaw: every band shares the same projection and the same positional embedding. The model is told nothing about which dimension is the deep-blue band vs the shortwave-infrared band. The wildly different physics of those bands are forced through one undifferentiated embedding, and the model must disentangle them with no help.

SatMAE's answer: group the bands, embed each group with its own spectral position

SatMAE partitions the 13 bands into spectral groups of bands that share similar physical character and ground resolution. For Sentinel-2 the paper uses three groups, roughly:

Each group is patchified and projected separately, and — the crucial part — each group gets its own learned spectral positional encoding, a vector e_g added to every token in group g. So a token's embedding now carries three position signals:

The spectral embedding spec_embed(g) plays the same role for "which band group am I?" that the positional embedding plays for "where in the image am I?" It lets attention treat the deep-blue token and the SWIR token as distinct kinds of thing, even at the same spatial location. The model can now learn band-specific structure: that the red-edge group is the one that lights up over healthy vegetation, that the SWIR group separates bare soil from water.

Why grouping beats both extremes

There are two extremes and SatMAE sits between them deliberately:

• One group of all 13 bands (the flat baseline): no spectral position info; the model can't distinguish bands. Underfits the spectral signal.
• 13 groups of one band each (fully independent): maximal spectral resolution, but 13× more tokens (13× the sequence length, O(13²) more attention) and it ignores that B2/B3/B4 are tightly correlated and share resolution — wasteful and slow.
• 3 physically-motivated groups (SatMAE): bands that share resolution and physics ride together (so correlated bands share a projection, which is efficient and regularizing), but distinct physics get distinct spectral codes. The paper's ablations show grouping helps over the flat baseline.

Worked example of the token count. Take a 96×96 chip, patch p = 8, so 12×12 = 144 spatial positions. Flat 13-channel: 144 tokens (each of input dim 8·8·13 = 832). Grouped into 3 groups: 3×144 = 432 tokens (each of input dim 8·8·|bands in group|, e.g. 8·8·4 = 256 for g0). Grouping multiplies the sequence length by the number of groups — that is the cost — but in exchange each band group gets a clean, dedicated representation and its own spectral position, which is what buys the accuracy.

Chapter 5: Independent Group Masking

We have grouped the bands and given each group a spectral code. Now the masking question returns, in spectral form: given the same spatial location across G band groups, do we mask the same patches in every group, or mask independently per group? This is the spectral twin of Chapter 3's temporal masking choice — and it is the core mechanism you will implement in Code Lab 1.

The shortcut a shared mask hands the model

Bands are correlated. The red band and the green band at the same pixel move together; the red-edge bands track vegetation jointly. If you mask the same spatial patch in every group, then to reconstruct group g's masked patch the model truly has to predict it from spatial neighbours. But there is a sneakier failure: with a shared mask, the visible band groups at a location are also all masked at the masked locations, so the model never sees the easy cross-band correlation. SatMAE instead masks independently within each group — a fresh 75% mask drawn per group.

With independent group masking, a spatial location might be masked in group g0 (visible bands) but visible in group g1 (red-edge bands). Now the model can — and must learn to — exploit the strong cross-band correlation: "I can see the red-edge reflectance here; predict the masked visible-band reflectance from it." This is the spectral analogue of borrowing across time. It forces the encoder to learn the inter-band relationships that carry the physical signal (e.g. NDVI-like contrasts), instead of treating each band group as an isolated spatial inpainting problem.

How the mask is actually drawn (the algorithm)

For each band group independently, take the N spatial patch indices, shuffle them, keep the first N_vis = (1−ρ)N as visible and mark the rest masked. The shuffle is the whole trick — it is how you draw a uniform random subset without replacement, and doing it per group with a fresh shuffle is what makes the masks independent. This is exactly the argsort(random) idiom MAE uses, applied once per group.

python
# Independent per-group random masking (the SatMAE spectral mechanism)
import numpy as np

def group_random_mask(N, G, mask_ratio, rng):
    """Return mask[G, N]: 1 = masked, 0 = visible. A FRESH random
       subset per group -> the masks are independent across groups."""
    n_vis = int(round((1 - mask_ratio) * N))
    mask = np.ones((G, N))
    for g in range(G):
        order = rng.permutation(N)        # fresh shuffle per group
        mask[g, order[:n_vis]] = 0      # first n_vis kept visible
    return mask

Worked example. N = 8 spatial patches, G = 3 groups, ρ = 0.5 so N_vis = 4 per group. Group 0 might keep visible {1,3,4,6}, group 1 {0,2,4,7}, group 2 {2,3,5,6}. Look at spatial location 4: visible in g0 and g1, masked in g2. So to reconstruct location 4's group-2 bands, the model has its group-0 and group-1 bands as evidence — cross-band inference is available because the masks differ. Under a shared mask, location 4 would be all-visible or all-masked, and that evidence channel would never exist.

Chapter 6: End-to-End Data Flow (tensor shapes)

Let us trace a single multi-spectral SatMAE forward pass with explicit shapes, so you could implement it. We use a concrete config: input chip 96×96, patch p = 8, so N = 144 spatial patches; G = 3 band groups; embedding dim D = 768 (ViT-Base scale); mask ratio ρ = 0.75.

Read the shapes carefully — they encode the whole design:

• [B, 432, D] after grouping: the 3× expansion of the sequence (vs 144 for the flat baseline) is the cost of band groups. Each of those 432 tokens carries pos_embed(s) telling it where and spec_embed(g) telling it which band group.
• [B, 108, D] into the encoder: 75% of the 432 are masked away, so the deep encoder runs on 108 tokens. This is the asymmetry that keeps SatMAE affordable even with 3× the tokens.
• [B, 432, P] out of the decoder: P = p·p·(bands in that token's group). The decoder predicts pixels for all 432 positions, but the loss reads only the 324 masked ones.

Worked example. With G = 3, T = 1, ρ = 0.75 and N = 144 per (group,time): total tokens = 3·1·144 = 432; visible = 0.25·432 = 108. Bump to T = 3 dates: tokens = 3·3·144 = 1296; visible = 324. The encoder attention cost scales like (visible)², so going from 108 to 324 visible tokens is ~9× more encoder compute — which is why SatMAE keeps ρ high (to claw back the multiplicative cost of groups and time) and keeps the decoder shallow.

Chapter 7: Experiments & Ablations

A method paper lives or dies on whether the design choices actually help. SatMAE pre-trains a ViT-Large encoder with the MAE objective on a large corpus of unlabelled satellite imagery (the fMoW — Functional Map of the World — dataset, in both RGB and full Sentinel-2 multi-spectral form), then evaluates two regimes: fine-tuning on the same domain and transfer learning to other remote-sensing tasks.

Headline results

Against the strongest prior self-supervised baselines for satellite imagery, SatMAE reports:

The transfer numbers are the more important story. A +14% jump on land cover from better pre-training alone — same labelled data, same fine-tuning budget — is the kind of result that says the representation, not the classifier head, is doing the work. This is the promise of self-supervision realized: the unlabelled ocean paid off.

Ablations: which choice mattered?

The paper isolates each design decision. The qualitative verdict (the directions, which is what you should remember):

Reading the ablation correctly. The choices are not redundant — each addresses a different failure of plain MAE. Band groups fix the spectral averaging failure; independent masking fixes the cheat-via-correlated-bands failure; the real-date embedding fixes the lost phenology failure. Because they target different failures, they stack: the full SatMAE is meaningfully above any single addition. This is the hallmark of a well-decomposed paper — orthogonal contributions that compose.

Chapter 8: The SatMAE Explorer

This is the payoff. Everything we have built — band groups, independent per-group masking, the encoder seeing only visible tokens, and the masked-only reconstruction loss — is wired into one interactive pre-training step you can drive. There is no quiz; the simulation is the test. If you can predict what each control does before you move it, you understand SatMAE.

The grid shows a small scene's spatial patches stacked across the G band groups you select. Drive the controls and watch: which tokens reach the encoder, which become [MASK], and where the reconstruction loss is actually measured. The "cross-band evidence" overlay lights up the visible band groups that a hovered masked patch can borrow from — the mechanism from Chapter 5 made literal.

Things to try, and what they reveal:

• Switch to "shared" masking and click a masked cell: every group's same-location cell goes dark — no cross-band evidence exists. Switch back to "independent" and the same cell often has visible neighbours in other groups (lit). This is the entire argument of Chapter 5 in one click.
• Raise ρ toward 0.85: the visible-token count (and the encoder-cost bar) plummets — pre-training gets cheaper, the task gets harder. This is why MAE-family methods use such high ratios.
• Drop G to 1: the spectral structure collapses to the flat baseline — one row, no band identity. The cross-band evidence channel disappears entirely.

Chapter 9: Limits & Connections

SatMAE is a clean, influential result, but no paper is the last word. Knowing its limits is how you know what came next.

Limitations

• Fixed, hand-designed band groups. The 3-group partition for Sentinel-2 is chosen by physics/resolution, not learned. A different sensor (Landsat, hyperspectral with hundreds of bands) needs a new partition, and the optimal grouping is not obvious.
• No multi-scale reasoning. SatMAE patchifies at one scale. Satellite scenes contain objects from single buildings to entire watersheds; later work (SatMAE++, Scale-MAE) adds explicit multi-scale pre-training and ground-sample-distance awareness.
• Token-count blow-up. Groups × timestamps multiply the sequence length; the asymmetric encoder helps but very long temporal stacks remain expensive. The O(T²) attention cost is the familiar transformer ceiling.
• Reconstruction objective. Predicting pixels is a proxy for "understand the scene." Contrastive and pixel-free objectives (e.g. masked feature prediction) are alternative recipes that SatMAE does not explore.

Where SatMAE sits in the lineage

Related lessons on this site

• Attention Is All You Need — the self-attention and sinusoidal positional encoding that SatMAE's encoder and temporal embedding are built on.
• Vision Transformer (Gleam) — how an image becomes patch tokens; the exact patchify step SatMAE inherits.
• Contrastive / CLIP (Gleam) — the other major self-supervised recipe; contrast SatMAE's reconstruction objective against contrastive learning.
• Diffusion (Gleam) — another "predict what was corrupted" objective, useful for seeing reconstruction as a learning signal.
• VAE / VQ-VAE (Gleam) — classic encoder-decoder representation learning, the lineage MAE descends from.

The cheat sheet

"What I cannot create, I do not understand." — and you can now create both of SatMAE's signature mechanisms in NumPy, from the two Code Labs above.