SpectralGPT: A Spectral Remote-Sensing Foundation Model

Chapter 1: The Spectral Cube as a Data Object

Before any architecture, we need to be precise about the thing being modelled. A Sentinel-2 tile, after SpectralGPT's preprocessing, is a tensor of shape [H, W, D] = [96, 96, 12] in the first training stage and [128, 128, 12] in the second. H and W are spatial pixels; D = 12 is the number of spectral bands kept (Sentinel-2 has 13 bands; the cirrus band B10 is dropped because it carries no surface information).

Read the indices carefully. The pixel at spatial location (r, c) is not a scalar and not an RGB triple — it is a 12-dimensional spectral vector x_r,c = [x_r,c,1, x_r,c,2, ..., x_r,c,12]. Walk down that vector and you are walking up the electromagnetic spectrum, from ~490 nm (blue) to ~2190 nm (short-wave infrared). That ordered walk is the spectral signature.

The two structures we must preserve

A good representation of this cube must respect two kinds of locality at once:

The paper's central design claim is that you cannot get the third box (coupling) by handling the first two separately. If you patch spatially and then run an RNN over bands, or vice versa, you bottleneck the interaction. The fix is to make the atomic unit of computation already 3D: a token that is simultaneously a small spatial patch and a small spectral slice. Then a single stack of self-attention layers can model spatial, spectral, and coupled structure with one mechanism.

A small worked example: the cube's size

Let us count, because the numbers drive every later design decision. One stage-1 cube is 96 × 96 × 12 = 110,592 scalar reflectance values. A million such cubes is over 10¹¹ values — far too much to label, which is exactly why self-supervision (no labels needed) is the only viable route. The pretraining set is fMoW-S2 (712,874 images) followed by BigEarthNet-S2 (354,196 images), about 1.07 million Sentinel-2 cubes total, none of them requiring a human annotation for the pretraining objective.

Chapter 2: 3D Tokenization — Cutting the Cube

The first real machinery. A 2D ViT cuts an image into non-overlapping spatial patches; SpectralGPT cuts the cube into non-overlapping 3D tensor tokens of size p × p × k pixels — a small square of space (p = 8) carrying a thin stack of bands (k = 3). Every token is itself a little cube.

Counting the tokens (the load-bearing arithmetic)

Take the stage-1 cube X ∈ ℝ^96×96×12. Slice it with a stride of 8 in each spatial direction and 3 along bands. The number of tokens along each axis is:

So a single 96×96×12 cube becomes a sequence of 576 3D tokens. For the stage-2 cube (128×128×12) the same token size gives 16 × 16 × 4 = 1024 tokens. The spectral axis is split into n_D = 4 spectral groups: bands {1,2,3}, {4,5,6}, {7,8,9}, {10,11,12}. This is the crucial difference from a 2D ViT — there, the band axis would be n_D = 1 (all bands crammed into one channel-stack). SpectralGPT keeps four distinct, ordered spectral positions, so attention can route information along the wavelength axis.

From a token cube to an embedding vector

Each 8×8×3 token contains 8·8·3 = 192 raw reflectance values. A shared linear projection (implemented as a 3D convolution with kernel = stride = token size, the 3D analogue of a ViT patch-embed) maps that flattened 192-vector to a model embedding of dimension d_model (768 for ViT-Base). Concretely, with the shared projection matrix E_s:

The tensor-shape pipeline for one stage-1 cube, end to end through the embedder:

Two positional embeddings, not one

A subtle but essential detail: SpectralGPT adds two positional embeddings to each token. One encodes the spatial location (which of the 12×12 grid cells), the other encodes the spectral position (which of the 4 band-groups). Because attention is permutation-invariant, without the spectral positional embedding the model could not tell band-group {1,2,3} from band-group {10,11,12} — it would lose the very ordering that makes the data spectral. The spatial embedding does the same job a 2D ViT's does; the spectral embedding is what 2D models lack.

Worked example: which raw pixels are in token (1,1,2)?

Token grid index (r=1, c=1, g=2), zero-based, with p=8, k=3, picks the raw region rows 8..15, columns 8..15, bands 6..8 (the second-to-last spectral group). That is one 8×8×3 sub-cube = 192 values. Mask this token and the model must reconstruct those 192 numbers using only the visible neighbours — both spatial (the surrounding tokens at the same band-group) and spectral (the same spatial patch at other band-groups). The spectral neighbours are the ones an RGB MAE never had.

Chapter 3: 3D Masking — Hide 90%, Reconstruct the Rest

Now the self-supervised heart. SpectralGPT follows the masked-autoencoder (MAE) recipe: corrupt the input by hiding most of it, then train the network to fill in what was hidden. The twist is what gets hidden. In image MAE you mask spatial patches; in SpectralGPT you mask 3D tokens, each one a chunk of space and bands. Hiding a 3D token hides an 8×8 spatial region at three specific wavelengths simultaneously.

Formally, a random binary mask M is drawn over the token grid:

where ρ is the masking ratio. The tokens with M = 1 are visible (x_vis); the tokens with M = 0 are masked (x_mask) and never shown to the encoder. SpectralGPT sets ρ = 0.90: ninety percent of tokens are dropped before the encoder ever runs.

The arithmetic of 90%

Out of N = 576 tokens, the encoder sees only (1 − 0.90) · 576 = 57.6 ≈ 58 visible tokens, and must enable the decoder to reconstruct the remaining ~518. That is a brutal ratio — far harsher than the 75% used by image MAE, where 25% of patches (about 49 of 196) are kept. Why can spectral data tolerate, and even benefit from, hiding so much more?

Two ways to mask — and why the choice matters

Because the token grid is 3D, "random masking" is genuinely 3D: a masked token removes a specific spatial patch at a specific band-group. This is strictly more informative than 2D masking, where you would remove an entire spatial column through all bands at once. Compare:

This is the mechanism that injects the spectral signal into the learning objective. When you hide the {7,8,9} slice of a spatial patch but leave its {1,2,3} and {10,11,12} slices visible, the only way to reconstruct the missing infrared is to learn how a pixel's visible/SWIR response constrains its NIR response — i.e., to learn the shape of the spectral curve. A 2D mask, hiding all bands of a patch at once, never poses this spectral-completion question.

Chapter 4: The Encoder & Decoder

SpectralGPT inherits MAE's asymmetric encoder-decoder design, and the asymmetry is the whole efficiency trick. A heavy encoder processes only the visible tokens; a light decoder sees everything and does the reconstruction. After pretraining the decoder is thrown away — only the encoder transfers to downstream tasks.

The encoder: heavy, but only on 10% of tokens

The encoder is a vanilla ViT (Base = 12 layers / 768-dim / ~86M params; Large = 24 layers; Huge = 32 layers). It receives only the ~58 visible tokens, each already embedded to 768 dims with its spatial and spectral positional embeddings. Because self-attention is O(N²) in token count, running the encoder on 58 tokens instead of 576 cuts attention cost by roughly (58/576)² ≈ 1% of the full-sequence cost. This is why MAE-style pretraining is cheap enough to run at the million-image scale: the expensive part never touches the masked tokens.

The decoder: light, but sees the whole grid

Now reconstruction. The decoder is deliberately narrower and shallower than the encoder — the SpectralGPT default is 4 transformer blocks at a smaller width. The decoder input is the full grid of 576 positions, assembled as:

The single learned mask token is the elegant part: the model does not need 518 different "I don't know" vectors. It uses one shared placeholder and relies entirely on the positional embeddings (spatial + spectral) to tell each masked slot where in space and wavelength it sits. The decoder's job is then "given where you are and what your visible neighbours encoded, predict your 192 numbers." Because every masked slot can attend to every visible latent across the whole grid, the reconstruction integrates long-range spatial and spectral evidence in one shot.

Why asymmetric? A design-decision ledger

Chapter 5: The Multi-Target Reconstruction Loss

This is SpectralGPT's signature contribution and where the spectral physics enters the objective. A vanilla MAE has one loss: predict the masked pixels. SpectralGPT uses two reconstruction targets summed together — a token-level term and a spectral-wise term — so the model is rewarded not just for getting pixels right, but for getting the spectral sequence right.

Both terms are mean-squared errors, but computed over different groupings of the reconstruction. Let x be the (normalized) ground-truth values and x̂ the decoder's predictions. We derive each term.

Term 1: token-level reconstruction

The standard MAE term — average squared error over the masked tokens. With m masked tokens:

This says: for each hidden 8×8×3 token, the predicted 192 numbers should match the true 192 numbers. It is the spatial-spectral "fill in the blank" loss. Computed only on masked tokens (the visible ones are trivially known), exactly as in image MAE.

Term 2: spectral-wise reconstruction

Here is the new idea. Fix a spatial location (r, c). Walking down the spectral axis at that location gives a sequence of n_D = 4 spectral-group values: [x_r,c,1, x_r,c,2, x_r,c,3, x_r,c,4] (each entry a band-group). The spectral-wise loss asks that this whole spectral profile at each spatial location be reconstructed coherently:

where s_j is the spectral sequence at the j-th spatial location and n is the number of such spatial locations. The difference from ℒ_token is the grouping: ℒ_token averages error per spatial-spectral chunk; ℒ_spectral averages error per spectral profile, tying together all band-groups at a fixed (r, c). This explicitly penalises a reconstruction that gets each band roughly right but produces a spectrally inconsistent curve (e.g., a kink that no real material would have).

The reconstruction target: normalize first

One more design decision with a measured payoff. Before computing the loss, the targets are normalized (scaled into [0,1]) rather than used raw. The paper tests three target choices — raw values, [0,1] normalization, and per-patch standardization (mean 0, std 1) — and finds normalization gives the best transfer, beating raw targets by about 0.66 mAP points downstream. The reason mirrors the √d_k story in attention: well-scaled targets keep the MSE gradient well-conditioned across bands whose raw reflectance magnitudes differ a lot, so no single bright band dominates the loss.

A tiny numerical worked example

Suppose at one spatial location the true (normalized) spectral profile is s = [0.20, 0.30, 0.80, 0.75] (low visible, high NIR — a vegetation-like curve) and the decoder predicts ŝ = [0.20, 0.30, 0.50, 0.75]. The token term might be small (three of four entries are exact). But the spectral term sees that the third entry — the NIR peak — is badly off: (0.80 − 0.50)² = 0.09, dominating the profile error and signalling that the reconstructed curve has lost its red-edge. Squared error on the profile makes that single physically-critical mistake expensive, which is precisely the point.

python
# SpectralGPT multi-target reconstruction loss (schematic)
import torch
import torch.nn.functional as F

def spectralgpt_loss(x, x_hat, mask, lam=0.5):
    """
    x, x_hat : [B, n_tokens, 192]  normalized true / predicted tokens
    mask     : [B, n_tokens]       1 = masked (compute loss), 0 = visible
    """
    # --- Term 1: token-level MSE on masked tokens only ---
    per_tok = ((x - x_hat) ** 2).mean(dim=-1)      # [B, n_tokens]
    L_token = (per_tok * mask).sum() / mask.sum()

    # --- Term 2: spectral-wise MSE over the band-group profile ---
    # reshape so the spectral (band-group) axis is grouped per spatial loc
    xs   = x.reshape(x.size(0), -1, N_D, 192)     # [B, n_spatial, n_D, 192]
    xh   = x_hat.reshape(x.size(0), -1, N_D, 192)
    L_spec = ((xs - xh) ** 2).mean()              # error along the spectral profile

    return L_token + lam * L_spec

Chapter 6: Progressive Training Across Datasets

A foundation model is only as good as the data it sees, and remote-sensing data comes in many sizes, resolutions, dates, and regions. SpectralGPT's answer is progressive training: pretrain in two stages, starting on a large but lower-quality corpus and continuing on a smaller, higher-quality, more varied one. The model "warms up" on quantity and "refines" on quality and diversity.

Why start small then go large?

Two reasons, both about easing the optimisation. First, the cube size grows from 96 to 128 between stages. Because the token size (8×8×3) is fixed, the same architecture handles both — a 96-cube is 576 tokens, a 128-cube is 1024 tokens, and the ViT simply processes a longer sequence. Starting on the shorter sequence lets the model lock in local spatial-spectral structure before being asked to integrate over a larger field of view. Second, fMoW-S2 is large but comparatively lower-quality; BigEarthNet-S2 adds temporal (time-series) and geographic diversity. Learning easy-and-plentiful first, then hard-and-varied, is a curriculum that the paper reports improves final transfer.

The model family

SpectralGPT ships three sizes, scaling the ViT backbone exactly as image ViTs do. The ">600M params" headline refers to the Huge variant; the family spans roughly:

The variant trained with the full two-stage progressive schedule is denoted SpectralGPT++ and is the one that posts the headline numbers. The "++" is the progressive-training boost on top of the base pretrained model.

Chapter 7: Results & Ablations

A foundation model lives or dies on transfer. SpectralGPT is pretrained once with no labels, then the encoder is fine-tuned on a battery of downstream tasks. The headline: it sets the state of the art across scene classification, multi-label tagging, semantic segmentation, and change detection on multispectral benchmarks.

Headline numbers

Read the EuroSAT line carefully: the gain over SatMAE is small in absolute terms (~0.1 points) because EuroSAT is nearly saturated — almost everything scores above 99%. The more telling result is BigEarthNet, a harder multi-label task with 19+ co-occurring land-cover classes, where the margin is larger, and especially the few-shot result: SpectralGPT fine-tuned on only 10% of the labels can match or beat prior models that used all the labels. That data efficiency is the real payoff of strong self-supervised pretraining — the encoder already "knows" the spectral world, so it needs few labels to specialise.

The ablations — what actually mattered

The paper isolates each design choice. The findings, in order of impact:

What the wins tell us

Every ablation points the same direction: the spectral axis was the missing ingredient. The two largest levers — the 90% masking ratio and the small 3D token — both exist only because the model treats wavelength as a real, redundant, ordered dimension. The multi-target loss and normalization are refinements on top. SpectralGPT's lesson for the field is that you do not get a great multispectral model by porting an RGB recipe; you get it by redesigning tokenization, masking, and loss around the physics of the spectrum.

python
# Fine-tuning SpectralGPT for a downstream task (schematic)
import torch.nn as nn

# 1. Load the pretrained ENCODER only (decoder is discarded)
encoder = load_spectralgpt_encoder("spectralgpt-base.pt")

# 2. Attach a small task head (e.g. linear classifier for EuroSAT)
class SceneClassifier(nn.Module):
    def __init__(self, encoder, n_classes=10):
        super().__init__()
        self.encoder = encoder           # pretrained ViT, sees ALL tokens now
        self.head = nn.Linear(768, n_classes)

    def forward(self, cube):       # cube: [B, H, W, 12]
        tokens = patchify_3d(cube)   # [B, N, 192], NO masking at fine-tune
        feats  = self.encoder(tokens) # [B, N, 768]
        return self.head(feats.mean(1))  # global-pool -> logits

# 3. Fine-tune with few labels — pretraining did the heavy lifting

Chapter 8: The Cube Explorer

This is the showcase — SpectralGPT's full mechanism in one interactive loop. You drive a small spectral cube through the entire self-supervised pipeline: choose a material so the cube has a real spectral signature, set the masking ratio, re-roll the random 3D mask, and watch the encoder-decoder reconstruct the hidden tokens. Then read off both loss terms. Everything you learned in Chapters 1–7 is on screen at once.

Three things to notice as you play. First, raising ρ thins the visible tokens and makes the reconstruction visibly worse where holes cluster — the harder the task, the more the model must rely on global structure. Second, the squared-error map lights up most on the infrared slices for vegetation, because the NIR peak is the most informative and therefore the most expensive to miss — exactly what the spectral loss is designed to penalise. Third, water (near-zero infrared, almost flat curve) is easy to reconstruct, while vegetation (a sharp red-edge) is hard; the loss reflects the information content of the material's spectrum.

Chapter 9: Limits & Connections

SpectralGPT is a milestone, not an endpoint. Holding it up against its own design reveals the open problems, and placing it in the lineage of self-supervised vision shows where its ideas came from and where they go next.

Limitations and open questions

Where the ideas came from, and where they connect

Cheat sheet

Continue on Engineermaxxing

Deepen the building blocks this paper assumes:

• Veanor: SatMAE — the prior remote-sensing MAE and SpectralGPT's main baseline. Compare its band-grouping to 3D tokenization.
• Veanor: Attention Is All You Need — the self-attention mechanism inside every ViT block here.
• Veanor: BERT — the original masked-prediction objective in language; SpectralGPT is its spatial-spectral cousin.
• Gleam: Vision Transformer — patchify-and-attend from zero, the 2D foundation SpectralGPT lifts to 3D.
• Gleam: Diffusion — another self-supervised generative objective; contrast denoising with masked reconstruction.
• Gleam: Embeddings — what the pretrained encoder ultimately produces and why those features transfer.