One self-supervised model, pretrained on a continent of satellite imagery by predicting the pixels it cannot see — then fine-tuned to map floods, fire scars, and crops from a handful of labels.
A flood is rising through a river delta. You have a satellite that just imaged the region — six spectral bands, 30-meter pixels, a few hundred kilometers on a side. You want a map: for every pixel, is this water or land? If you had that map in an hour, you could route boats and warn villages. The pixels are sitting right there in the downlink. The problem is not the image. The problem is the map.
To train a flood-segmentation model the classical way, someone has to hand-label water versus land for thousands of these scenes. Labeling a single Sentinel-2 tile to pixel accuracy can take an expert hours. For a new disaster — a new delta, a new sensor, a new season — you often have a few dozen labeled scenes at best. A standard convolutional segmentation network trained from scratch on a few dozen scenes overfits and generalizes poorly. The bottleneck in Earth observation is almost never the imagery; petabytes pour down every day. It is labels.
Meanwhile, in natural-language processing and ordinary computer vision, this exact problem had already been solved by foundation models: pretrain one large model on an ocean of unlabeled data with a self-supervised objective, then fine-tune it to each downstream task with very few labels. BERT learned language by filling in masked words. The Masked Autoencoder (MAE, He et al. 2022) learned vision by reconstructing masked image patches. Prithvi asks the obvious, valuable question: can we do the same for satellite imagery?
Two learners on the same flood task. "From scratch" sees only the labeled scenes (orange). "Pretrained then fine-tuned" first absorbed millions of unlabeled scenes, so it needs far fewer labels to reach the same accuracy. Drag the slider to change how many labeled scenes you can afford.
Prithvi (named for Prithvi, Sanskrit for "Earth") is a geospatial foundation model jointly built by NASA and IBM Research and released openly in 2023. It is a Vision Transformer trained as a Masked Autoencoder on multi-temporal, multispectral satellite imagery from the Harmonized Landsat–Sentinel-2 (HLS) product. After pretraining on a large unlabeled archive covering the contiguous United States, the same encoder is fine-tuned — with comparatively tiny labeled datasets — to flood mapping, wildfire-burn-scar mapping, and multi-temporal crop classification.
| Approach | What it learns from | Labels needed downstream | Generalization |
|---|---|---|---|
| From scratch (per task) | Only the task's labeled scenes | Thousands per task | Brittle; overfits small sets |
| ImageNet transfer | RGB natural photos | Hundreds-thousands | Domain gap: 3 bands, not 6+; no time |
| Prithvi (geo foundation) | Petabyte-scale unlabeled HLS cubes | Tens to low hundreds | Strong: pretrained in-domain |
The rest of this lesson builds Prithvi from the ground up: the HLS data cube it eats, the spatiotemporal tubelet tokenization that turns that cube into tokens, the masking that creates the self-supervised puzzle, the reconstruction loss that supervises it, the encoder/decoder split that makes pretraining cheap, and finally how a frozen-or-fine-tuned encoder becomes a flood map. We end with the paper's reported results and its honest limitations.
Before any architecture, you have to know the shape of the thing the model eats. Prithvi is pretrained on HLS — Harmonized Landsat Sentinel-2 — a NASA product that fuses observations from the Landsat-8/9 and Sentinel-2 satellites into one consistent, atmospherically corrected, 30-meter surface-reflectance archive. "Harmonized" is the key word: the two sensor families are radiometrically and geometrically aligned so a pixel from Landsat and a pixel from Sentinel-2 mean the same physical reflectance. Together they revisit the same spot roughly every two to three days, which is what makes dense time series possible.
Prithvi keeps six surface-reflectance bands: Blue, Green, Red, Narrow Near-Infrared (NIR), Short-Wave Infrared 1 (SWIR-1), and Short-Wave Infrared 2 (SWIR-2). These six are common to both sensors and carry most of the land-cover signal. The non-visible bands matter enormously: NIR separates healthy vegetation from bare soil and water; SWIR is sensitive to moisture and burn, which is precisely why fire scars and floods pop out in those channels.
A single Prithvi training sample is therefore not an image — it is a 4-D data cube with axes (time, band, height, width). Concretely the model is pretrained on chips of:
Read that as: T = 3 timestamps (three dates of the same place), C = 6 spectral bands, and a 224×224 spatial grid (a 224×224 chip at 30 m/pixel covers about 6.7 km on a side). So one sample is 3 × 6 × 224 × 224 ≈ 902,000 reflectance numbers. The third axis — time — is the whole reason Prithvi is more than "MAE with six channels." Information about whether a pixel is flooding lives in how it darkens across the three dates.
The cube has three dates (rows) and six bands (columns). Each tile is one 224×224 image. Click a tile to highlight which (date, band) slice it is. Notice the SWIR columns where water and burn separate most strongly.
Let us tally the numbers by hand, because the shapes drive everything that follows. One date is C·H·W = 6 · 224 · 224 = 301,056 values. Three dates is 3 · 301,056 = 903,168 values per sample. The pretraining archive is on the order of tens of thousands of such chips drawn from across the contiguous US and across seasons, so the model sees billions of reflectance numbers — all unlabeled. That abundance is the fuel; the masking puzzle (Chapter 3) is the engine that converts it into learning.
One more practical point: reflectance values per band have very different ranges, so each band is standardized (subtract the band mean, divide by the band standard deviation) before tokenization. This is the geospatial analogue of per-channel normalization in ordinary vision — without it the high-reflectance SWIR bands would dominate the reconstruction loss and the model would ignore the dim blue band.
A Transformer does not eat pixels. It eats a sequence of tokens — fixed-length vectors. The Vision Transformer (ViT) turned a 2-D image into tokens by cutting it into non-overlapping patches (say 16×16 pixels) and linearly projecting each patch into a vector. Prithvi extends this idea into the time axis. Instead of a 2-D patch, it cuts the cube into 3-D tubelets — little bricks that span a patch of space and a span of time.
Formally, the cube x ∈ ℝT×C×H×W is partitioned into non-overlapping tubelets of size (tp, p, p) along (time, height, width), keeping all C bands inside each tubelet. With Prithvi's pretraining settings the tubelet is tp = 1 in time and p = 16 in space, so one tubelet is a 1×16×16 spatial patch carrying all 6 bands at one date. (The mechanism allows tp > 1 — a single token literally spanning multiple dates — and the showcase lets you slide it.)
Each tubelet's raw pixels are flattened and passed through a single learned linear layer (equivalently a strided 3-D convolution) that projects them to the model width D. This projection is the patch embedding; it is the only place raw reflectance enters the network.
Plug in Prithvi's numbers. With T = 3, tp = 1: the time factor is 3/1 = 3. With H = W = 224, p = 16: each spatial factor is 224/16 = 14. So:
And each token's raw dimension before projection is dtok = tp·p·p·C = 1·16·16·6 = 1,536 numbers. The patch-embedding linear layer maps that 1,536-vector to the encoder width D (Prithvi-100M uses D = 768, matching ViT-Base). So the cube of 903,168 numbers becomes a sequence of 588 tokens × 768 = 451,584 numbers — a different packaging of similar information, now in the shape a Transformer loves.
A simplified cube (here 12×12 spatial per date, 3 dates) being sliced into tubelets. Drag the patch-size slider: bigger patches mean fewer, fatter tokens; smaller patches mean more, finer tokens. The token count N updates live.
Why tubelets and not "one token per pixel"? Two reasons. First, cost: self-attention is O(N2) in the number of tokens, and 224×224×3 = 150,528 pixel-tokens would be hopeless, whereas 588 tubelet-tokens is comfortable. Second, locality: a 16×16 patch is a meaningful unit of land — a field, a roof, a stretch of river — so projecting it into one vector gives the attention layers a sensible vocabulary to reason over.
Flattening into a sequence destroys the cube's geometry — token 200 and token 201 carry no inherent sense of where or when they came from. Prithvi restores this with additive positional encodings: a spatial position embedding (which row/column of the 14×14 grid) and a temporal position embedding (which of the three dates). These are added to the patch embeddings before the encoder, exactly as the original Transformer added sinusoidal positions, so attention can express "this token sits next to that one" and "this token is one date earlier."
We now have 588 tubelet tokens and no labels. How do we extract learning from unlabeled data? The trick, borrowed from MAE, is to hide most of the tokens and ask the model to reconstruct them. The puzzle creates its own supervision: the answer key is the original pixels we deleted. Solving "predict the missing patch from the visible patches" forces the model to learn what land looks like — texture, spectral relationships, how a place changes over time.
Concretely, Prithvi samples a masking ratio r (the fraction of tokens to hide) and randomly selects r·N tokens to drop. The remaining (1−r)·N visible tokens go into the encoder; the masked tokens are held out as reconstruction targets. Following MAE, Prithvi uses a high masking ratio around 0.75 — three out of every four tubelets are hidden.
With N = 588 and r = 0.75: Nmasked = 0.75 · 588 = 441 tokens hidden, and Nvisible = 588 − 441 = 147 tokens kept. The encoder — the expensive part — only ever processes those 147 tokens. That is the second reason for the high ratio (the first is difficulty): with 75% of tokens dropped, the encoder runs on a quarter of the sequence, so pretraining is roughly 3–4× cheaper per step than processing the full cube.
A 14×14 token grid (one date). Masked tokens go dark; visible tokens stay lit. Drag the ratio: at 0.75 only a sparse scatter of tokens remains — yet the model must reconstruct everything. "Reshuffle" re-samples which tokens are hidden.
Because tokens carry a time index, the random mask hides tokens from all three dates indiscriminately. So the model is sometimes asked: "Given this patch on dates 1 and 3, fill in date 2." That is a temporal-interpolation puzzle — and learning to solve it is exactly what teaches the model the dynamics of land cover (a river widening, a field greening). A space-only mask could never pose that question. This is the payoff of the tubelet design from Chapter 2: masking inherits the joint spatiotemporal structure for free.
The masking made the puzzle; now we need the grade. Prithvi is trained with the MAE objective: mean squared error between the reconstructed pixels and the original pixels, computed only on the masked tokens. Let us derive it carefully, because the two qualifiers — "pixels" and "only on masked tokens" — are both deliberate.
Let M be the set of masked token indices. For a masked token i, let yi ∈ ℝdtok be the original (normalized) tubelet pixels — the ground truth — and let ◯i be the decoder's prediction for that token. The per-token squared error is the squared Euclidean distance, and the loss is its mean over masked tokens:
Read it inside-out. The inner sum over j is the squared error across the dtok pixel values inside one tubelet (all bands, all pixels of the patch), averaged so each token contributes a per-pixel MSE. The outer sum averages over the |M| masked tokens. Visible tokens contribute nothing to the loss.
If we also scored the visible tokens, the model could earn a low loss by simply copying its inputs straight through (an identity map on the visible part), which teaches nothing. By grading only the patches it could not see, we force every gradient to come from genuine inference. This is identical to BERT scoring only the masked words, and to MAE scoring only the masked patches — the masked-only loss is what makes the self-supervision real.
Reflectance is continuous. There is no discrete "vocabulary" of pixel values to softmax over, so regression (MSE) is the natural fit: predict the actual reflectance numbers and penalize the squared miss. (MAE-for-classification variants exist, but Prithvi's surface-reflectance reconstruction is a regression.)
Suppose, after normalization, one masked token's true pixels start as y = [0.20, −0.50, 0.30, 0.10] and the decoder predicts ◯ = [0.10, −0.20, 0.30, 0.40]. The residuals are [0.10, −0.30, 0.00, 0.30]. Squared: [0.0100, 0.0900, 0.0000, 0.0900], summing to 0.1900. Divide by dtok = 4 to get a per-token MSE of 0.0475. If this were the only masked token, L = 0.0475. Average over all 441 masked tokens for the real batch loss. Gradient descent nudges the weights to shrink exactly these residuals.
Top bars: a masked token's true pixel values (teal) and the model's prediction (warm). Bottom: the squared residual per pixel, and the per-token MSE. Drag "training progress" to move the prediction toward the truth and watch the loss fall.
We have tokens, a mask, and a loss. Now the architecture that turns them into a trained encoder. The MAE design is asymmetric on purpose: a large encoder that sees only the visible tokens, and a small, lightweight decoder that sees everything and reconstructs the masked pixels. After pretraining the decoder is thrown away; the encoder is the prize.
The encoder is a standard Vision Transformer. Prithvi-100M uses D = 768 width, 12 transformer blocks, and 12 attention heads — the ViT-Base configuration, about 100 million parameters. Each block is the familiar pair: multi-head self-attention (so every visible token can attend to every other visible token, across space and time) followed by an MLP, each wrapped in a residual connection and layer norm. With only 147 tokens entering, the O(N2) attention is cheap.
For reconstruction the decoder needs all N = 588 positions back. So we take the 147 encoded visible tokens and re-insert 441 copies of a single shared, learnable [MASK] token at the masked positions, restore the positional encodings, and run a shallow transformer (far fewer/narrower blocks than the encoder). A final linear head maps each token to dtok = 1,536 numbers — the reconstructed tubelet pixels.
Tokens flow left to right. Only the visible (teal) tokens enter the tall encoder; the masked positions (gray) are re-inserted as shared [MASK] tokens before the short decoder. Click "Run" to animate one forward pass and see the token counts at each stage.
Tracing one sample, shapes in brackets: cube [3,6,224,224] → patch-embed to tokens [588,768] → add position/time embeddings [588,768] → keep visible [147,768] → encoder (12 blocks) [147,768] → re-insert mask tokens [588,768] → decoder (shallow) [588, Ddec] → pixel head [588,1536] → gather masked rows [441,1536] → scalar loss. Every arrow is a concrete tensor op; nothing is hand-waved.
Pretraining produced an encoder that turns an HLS cube into rich spatiotemporal features. Now we cash that in. The pattern is the same for all three downstream tasks: keep the pretrained encoder, throw away the MAE decoder, and attach a small task-specific head, then train on a (small) labeled dataset.
For the two segmentation tasks — flood mapping and burn-scar mapping — the head is a lightweight semantic-segmentation decoder (in the paper, a UNet-style / convolutional upsampling head) that takes the encoder's token features and produces a full-resolution per-pixel class map: water vs. not-water, burned vs. not-burned. For crop classification the head emits a per-pixel multi-class label across crop types. In each case the head is small and the heavy lifting is done by the pretrained encoder.
Two regimes. Frozen: the encoder weights are fixed and only the head trains — cheapest, and a clean test of how good the pretrained features are on their own. Fine-tuned: the encoder weights are also updated (typically with a small learning rate) so they adapt to the task — usually a bit more accurate, at more compute. The paper's headline message is that even the cheap regimes work well because the pretrained features already encode the relevant geospatial structure.
Left: an HLS scene over a delta. The pretrained encoder yields token features; the segmentation head colors each pixel water (blue) or land. Drag "labels available" — with Prithvi the mask stays sharp even at very few labels; the dashed "scratch" line collapses.
| Task | Signal exploited | Head | Output |
|---|---|---|---|
| Flood mapping | Water darkens in NIR/SWIR; change across dates | Segmentation | Per-pixel water / no-water |
| Burn-scar mapping | Burn darkens NIR, brightens SWIR; before/after | Segmentation | Per-pixel burned / unburned |
| Crop classification | NIR phenology over the season (multi-date) | Multi-class segmentation | Per-pixel crop type |
A foundation model lives or dies by transfer. The claim Prithvi must back up is: pretraining on unlabeled HLS, then fine-tuning on small labeled sets, beats training the same architecture from scratch — and does so with far fewer labels. The paper reports exactly this across the three tasks.
Segmentation tasks are scored with Intersection-over-Union (IoU) and the F1 score. IoU for a class is the area of overlap between predicted and true regions divided by the area of their union — 1.0 is perfect, 0 is no overlap. F1 is the harmonic mean of precision and recall. Both reward getting the shape of the water/burn region right, not just the per-pixel accuracy (which can be inflated by the easy background).
Rather than invent precise figures, here is the robust, reported pattern (read the directions, not fabricated decimals):
| Task | Metric | Reported finding |
|---|---|---|
| Flood mapping | IoU / F1 | Fine-tuned Prithvi outperforms a from-scratch baseline of the same size; strong water/land delineation |
| Burn-scar mapping | IoU / F1 | Prithvi matches or exceeds strong baselines, capturing scar boundaries from before/after dates |
| Crop classification | Accuracy / F1 | Competitive multi-class crop maps from multi-temporal input, leveraging seasonal phenology |
| Label efficiency | metric vs. #labels | The headline: Prithvi reaches baseline-level accuracy with a small fraction of the labels needed from scratch |
Accuracy vs. number of labeled scenes for from-scratch (orange) vs. pretrained-then-fine-tuned (teal). Drag your label budget; read off how many labels each needs to clear the dashed target line. The pretrained model clears it far to the left.
Prithvi was released openly (model weights and code on the Hugging Face Hub and GitHub), explicitly so the geoscience community could fine-tune it on their own tasks without retraining a foundation model. This open release is part of the contribution: a from-scratch geospatial foundation model is out of reach for most labs, but fine-tuning an existing 100M-parameter encoder on a few labeled scenes is not. The model has since seeded a family of larger geospatial models.
A foundation-model paper has to answer "which design choices earned their keep?" Prithvi's ablations probe the pieces we built: the temporal axis, the masking ratio, and pretraining itself. Here is what they reveal, stated as directions rather than invented numbers.
Compare a single-date variant (T = 1, plain spatial MAE) against the multi-temporal model (T = 3 with tubelets). For tasks whose signal is temporal — floods (a place changes from dry to wet) and crops (seasonal phenology) — the multi-temporal model wins, confirming that the tubelet's time fold is doing real work, not just adding cost. This is the empirical justification for the paper's central mechanism.
Sweep r. Too low (e.g. 0.25) and the pretext is too easy — the encoder learns little because copying neighbors suffices, so downstream transfer is weaker. Too high (e.g. 0.95) and even the structure needed to reconstruct is gone. The sweet spot sits around 0.75, echoing MAE on natural images. The interactive below lets you feel why a middle-high ratio is best.
A schematic transfer-quality curve over masking ratio. Too low = trivial pretext (low learning). Too high = not enough context to reconstruct. The peak sits near 0.75. Drag to find the peak; the bars show "context kept" vs. "challenge".
Initialize the same encoder randomly and fine-tune directly (no MAE pretraining). Downstream accuracy drops, and the drop is largest in the low-label regime — exactly where the foundation-model premise is supposed to pay off. This is the cleanest demonstration that the self-supervised pretraining, not just the architecture, is responsible for the gains.
| Ablation | Variant | Direction of effect |
|---|---|---|
| Temporal modeling | T=1 vs. T=3 tubelets | Multi-temporal helps on flood & crop (temporal tasks) |
| Masking ratio | 0.25 / 0.75 / 0.95 | Best near 0.75; too-low and too-high both hurt transfer |
| Pretraining | random init vs. MAE-pretrained | Pretraining helps most, especially with few labels |
| Bands | RGB vs. 6-band | NIR/SWIR add signal for water, burn, vegetation |
This is the payoff. Everything you have built — tubelet tokenization, random masking, encoder/decoder, and the masked reconstruction loss — running on a tiny synthetic HLS cube you can poke. There is no quiz; the simulation is the test. If the controls make intuitive sense to you, you understand Prithvi.
The left panel shows the original cube (three dates, one synthetic band, with a "river" that widens across time — the flood signal). The middle panel shows it after masking: hidden tubelets are blanked. The right panel shows a toy reconstruction — a simple inpainting that fills masked tubelets from their visible neighbors and from the same patch on other dates, exactly the kind of spatiotemporal inference real Prithvi learns. The live readout reports tokens, visible/masked counts, and the masked-only reconstruction MSE.
Controls: patch size sets the tubelet (token count N); mask ratio sets how much is hidden; tubelet time-span tp folds 1 or all dates into each token; Reshuffle re-samples the mask. Watch the reconstruction sharpen as you lower the ratio, and watch the masked-only MSE move. Try a high ratio with a small patch — the puzzle gets hard and the MSE climbs.
What to notice as you play:
Prithvi is a milestone, not an endpoint. Naming its limits honestly is part of understanding it — and each limit is a door to a related idea you can study next.
Prithvi is a careful synthesis of three ideas, each of which has its own deep lesson here:
| Concept | Prithvi specifics |
|---|---|
| Data | HLS: harmonized Landsat + Sentinel-2, 6 surface-reflectance bands (B,G,R,NIR,SWIR-1,SWIR-2), 30 m |
| Sample shape | T×C×H×W = 3×6×224×224 (a 4-D space-time cube) |
| Tokenization | Tubelets tp=1, p=16 → N = 3·14·14 = 588 tokens, dtok = 1,536 → project to D=768 |
| Pretext task | Masked Autoencoder: hide ~75% of tubelets, reconstruct their pixels |
| Loss | MSE on reconstructed pixels, masked tokens only |
| Architecture | ViT-Base encoder (D=768, 12 blocks, 12 heads, ~100M params) on visible tokens; shallow decoder; decoder discarded after pretraining |
| Positions | Additive spatial + temporal position embeddings |
| Downstream | Flood mapping, burn-scar mapping, crop classification — small head, few labels |
| Metric | IoU / F1 (segmentation); the headline is label-efficiency vs. from-scratch |
| Release | Open weights + code (NASA + IBM), seeding a family of geospatial foundation models |