Earth Observation Embedding Fields

Chapter 1: What Actually Arrives

Chapter 0 wrote E as a clean function of place and time. This chapter is about the input side of that arrow, which is anything but clean, and about why the messiness dictates almost every architectural decision that follows.

The naive mental model of satellite data is a video: a stack of images of the same place, evenly spaced in time, all the same size. Every part of that sentence is false.

The instruments, and what each one is actually measuring

AlphaEarth trains against nine gridded sources plus one unstructured text source. They are not variations on a theme; they measure physically different things at wildly different scales.

Sit with the resolution column for a second. The spread from 10 m to 300 km is a factor of 30,000. A single GRACE cell covers 90,000 square kilometres and about 900 million Sentinel‑2 cells. These are not layers of the same picture. They are different physics sampled on different lattices.

What a band actually is, and why ratios beat magnitudes

Before the irregularity, the physics — because one small derivation here explains a design choice that echoes through the entire lesson.

A multi-spectral sensor measures, per pixel, the fraction of incoming sunlight reflected back in each of several wavelength intervals. Those fractions are called reflectances, and they are between 0 and 1. Different materials have characteristic curves. Healthy leaves absorb red light hard, because chlorophyll uses it for photosynthesis, and scatter near-infrared hard, because the internal cell structure is transparent at that wavelength and bounces it around. So a green leaf might reflect 4% of red light and 45% of near-infrared.

That contrast is the single most useful fact in optical remote sensing, and the classic way to extract it is a normalised difference:

Work three surfaces:

Now the part that matters. Suppose a thin haze, or a lower sun angle, scales both bands by 0.7. Recompute the vegetation case:

Identical, to every digit. The ratio form cancels any multiplicative illumination factor exactly, because the same k appears in numerator and denominator. That is why vegetation indices dominated the field for forty years: they are the cheapest possible way to strip out the largest nuisance variable in the data.

One caution about what the released embeddings are computed from. AlphaEarth's optical inputs are Sentinel‑2 L1C and Landsat TOA — both top-of-atmosphere products, not atmospherically corrected surface reflectance. The model sees the atmosphere. It is left to learn atmospheric effects as a nuisance to be suppressed by the consistency objective, rather than being handed a corrected input. That is a defensible choice — atmospheric correction is itself a model, with its own failures, and one fewer preprocessing dependency is one fewer thing that can silently change between years — but it means the raw signal is noisier than a corrected product would be.

The revisit problem, derived

Here is the arithmetic that quietly ruins optical remote sensing. Sentinel‑2 has two satellites in the same orbit, giving a nominal revisit of about five days at the equator. So the number of attempts per year is:

But an optical sensor sees sunlight bounced off the ground, and a cloud is opaque. If the probability that a given pixel is cloud-covered at a given overpass is p, then the expected number of usable observations is:

Work three cases by hand. A dry site in central Australia, p ≈ 0.15: 73 × 0.85 = 62 usable scenes. A temperate site in France, p ≈ 0.50: 73 × 0.50 = 36.5. Borneo in the monsoon, p ≈ 0.85: 73 × 0.15 = 11 usable scenes for the entire year.

Eleven. And they will not be evenly spread — cloudiness is seasonal, so those eleven scenes cluster in the dry months and leave a six-month hole exactly when the crop you care about is growing. The same model, at the same nominal resolution, is given six times more evidence about France than about Borneo. Any architecture that assumes a fixed-length, evenly-spaced input sequence has already failed on the hardest half of the planet.

Radar is not a cloud-free camera

The obvious fix is radar. Synthetic aperture radar (SAR) emits its own microwave pulse and measures what comes back, so it works at night and straight through cloud. Sentinel‑1 gives you a near-guaranteed observation every 6 to 12 days regardless of weather.

But you have not filled the gap with the same quantity. Optical reflectance tells you about pigment and chemistry — chlorophyll absorbs red, cell structure scatters near-infrared, water absorbs shortwave infrared. Radar backscatter tells you about geometry and dielectric constant — how rough the surface is at the scale of the wavelength, and how wet it is. A flooded rice paddy is dark in radar because a smooth water surface reflects the pulse away from the sensor; a forest is bright because branches scatter in all directions.

And radar has its own geometry problem. The same hillside imaged from an ascending pass (satellite heading north) and a descending pass (heading south) produces genuinely different backscatter, because the local incidence angle changed. Terrain that slopes toward the sensor gets compressed and brightened; terrain sloping away gets stretched and darkened. So the radar record is not just irregular in time, it is conditional on acquisition geometry — which is why AlphaEarth's decoders take "orbital geometry and metadata that is only relevant to the act of measurement, not the measurement itself" as an explicit conditioning input. We will come back to that phrase in Chapter 2, because it is one of the sharpest ideas in the paper.

GEDI is not a raster at all

One source deserves special attention because it breaks the mental model hardest. GEDI is a LiDAR on the International Space Station. It fires laser pulses and times the return, giving you a vertical profile of whatever the beam hit — the canopy top, the layers of branches, the ground. It is the only source here that directly measures structure rather than inferring it.

But GEDI does not image. It samples. Each shot illuminates a footprint about 25 m across, and the shots fall in sparse tracks along the ISS ground path, which is confined to roughly ±51.6° latitude. Over most of the planet, most cells have never been hit by a GEDI beam and never will be.

You cannot stack that with an image. What you can do — and this is the move — is use it as a reconstruction target at the locations where it exists, and let the model learn to predict it everywhere else. The paper notes the consequence with visible pleasure: the decoders "have the effect of generating spatially continuous predictions for an arbitrary timestamp (e.g., dense, superresolved LiDAR profiles from GEDI)." Chapter 7 will insist that you read those dense profiles as predictions, not measurements.

The decision that makes the whole thing tractable

Given ten sources, the obvious design is to feed all ten in. AlphaEarth deliberately does not. The paper is explicit:

So the source list splits in two. Four sources are inputs (encoded). All ten are targets (decoded). The embedding must be rich enough to reconstruct GEDI canopy profiles, ERA5 climate, GRACE gravity and NLCD labels — from optical and radar alone.

Think about why "ill-posed" is the right word. If you feed monthly climate into the encoder and then ask the decoder to reconstruct radar backscatter, there is no function from climate to backscatter — the mapping genuinely does not exist at that resolution. The model would be forced to either ignore the climate channel or hallucinate. Worse, if you feed a source in and ask for it back, the easiest solution is a copy, and copies teach nothing. Restricting the inputs is what turns reconstruction into a real bottleneck.

What "normalised using global statistics" hides

The paper says inputs were "normalized based on global image statistics and no further value modification or augmentation was performed." Those fourteen words carry two decisions.

Global, not local. A per-tile normalisation — subtract this tile's mean, divide by this tile's standard deviation — is the standard vision default, and it would be a disaster here. It would delete exactly the information you want: that this tile is darker than average, or has less variance than average. Absolute reflectance is a physical measurement of the ground, not a nuisance offset. Normalising it away would make a desert and a rainforest look statistically identical after standardisation.

No augmentation. Vision pipelines flip, crop, rotate and colour-jitter as a matter of course. Here, colour jitter would corrupt a physical measurement, and vertical flipping would put the sun in the north. The only "augmentation" in the whole system is the deliberate input dropping of the consistency objective in Chapter 2 — and that is not augmentation for invariance, it is a simulation of the actual failure mode the data exhibits.

What counts as "the same pixel"?

One more source of mess, and it is the one nobody warns you about. When you compute a reconstruction loss between a prediction and a target, you are asserting that the two grids line up. They do not.

Sentinel‑2's geolocation accuracy is on the order of a pixel or two. Landsat's grid is 30 m and does not nest inside the 10 m grid at a fixed offset. GRACE's is 300 km. Naively differencing a 10 m prediction against a 30 m target teaches the model to blur, because blurring is the optimal response to registration noise.

AlphaEarth uses two different fixes, chosen per source:

Shift-invariant loss computes the error under every planar shift up to the stated distance and keeps the minimum. At 20 m on a 10 m grid that means searching shifts of up to two pixels in each direction — a 5×5 grid of candidate offsets, keep the best. The model is then not penalised for being right in the wrong place by one pixel.

Re-gridding loss area-averages both prediction and target down to the target's real resolution before comparing. Predict at 10 m, average 3×3 blocks to 30 m, compare against Landsat there. The model keeps its fine detail; it just is not graded on detail the target never had.

And notice the loss weights: NLCD and GRACE get 0.5, everything else 1.0. NLCD is a human-assigned label product covering only the United States. Giving it full weight would tilt the global representation toward one country's legend. Chapter 7 tells the story of what happened when even 0.5 turned out to be too much.

Chapter 2: AlphaEarth Foundations

We now have the problem in full: irregular, multi-scale, physically heterogeneous observations, and a requirement to emit one 64-byte unit vector per 10 m cell per time window, everywhere on Earth. This chapter builds the model that does it, term by term.

The objective, in four pieces

Everything AlphaEarth learns comes from one loss. Here it is, and then we will take it apart:

Four terms, four weights: a = 1.0, b = 0.05, c = 0.02, d = 0.001. Look at those magnitudes before anything else. The reconstruction term carries a thousand times the weight of the text term. This is not a contrastive language model with a reconstruction extra; it is a reconstruction model with a whisper of language. Keep that ratio in mind when you read Chapter 3, where SatCLIP is the exact opposite.

Term by term:

(a) Reconstruction: an implicit decoder, not a decoder stack

The reconstruction term is the engine, and its shape is unusual enough to be worth slowing down for.

A normal autoencoder has a decoder that mirrors the encoder: upsampling blocks that turn a small latent back into a big image. AlphaEarth does not do that. Instead, for each of the ten decoded sources there is a tiny MLP — two hidden layers, width 512 — and it is applied independently at every pixel of the embedding grid.

What goes into that MLP at each pixel is the interesting part. Three things are concatenated:

That is the design that makes the embedding clean. Everything that belongs to the act of measurement — which satellite, which orbit direction, what incidence angle, what moment — is supplied to the decoder as side information. The embedding is never asked to carry it, and is punished for wasting capacity on it, because that capacity is needed to reconstruct all ten sources.

Because the decoder is a function of a continuous timecode rather than a fixed frame index, it can be evaluated at any instant, including ones with no observation. That is where the paper's continuous-time claim comes from, and it is worth stating carefully.

Support period vs. valid period: the continuous-time idea

Two time intervals matter, and conflating them is the most common misreading of this paper.

The support period is the span covered by the input timestamps — the range of t_j over the frames you fed in. The valid period [t_s, t_e) is the span you are asking about. They are decoupled. The paper spells out three regimes:

The paper claims AEF is, to their knowledge, "the first EO featurization approach to support continuous time." That claim rests on a specific mechanism, not on wishful thinking: the temporal summary is produced by time-axial attention pooling driven by a learned query derived from [t_s, t_e), after converting the interval to sinusoidal timecodes.

Unpack that. Ordinary attention pooling learns one fixed query vector and asks "which frames matter?" Here the query is a function of the window you asked about. Ask for January and the query attends to the January frames; ask for the growing season and it attends to those. The same 103 input frames yield different embeddings for different questions, with no fine-tuning and no re-encoding of the imagery.

That is why the paper can say you "can apply AEF to time dependent problems requiring a precise date range without fine-tuning." A crop-type question about a specific planting window and a fire-scar question about a specific week are the same forward pass with a different query.

The encoder: Space, Time, Precision

Now the network that produces the embedding. AlphaEarth calls it STP, for Space‑Time‑Precision, and it is built to resolve a genuine tension: you need long-range context (is this pixel in a floodplain or on a ridge?) and 10 m precision (is this pixel palm or forest?), and attention over a full-resolution video is unaffordable.

The resolution is three parallel pathways running at different scales, in every block. For a square input of L pixels a side:

Do the cost arithmetic and the design justifies itself. Self-attention is quadratic in token count. At L/2 with L = 128 you have 64×64 = 4,096 tokens, so 4,096² ≈ 16.8 million pairwise interactions per frame. At L/16 you have 8×8 = 64 tokens and 4,096 interactions — four thousand times cheaper. Meanwhile a 3×3 convolution is linear in pixels, so it is affordable at the fine scale where attention is not. Each mechanism is deployed exactly where its cost curve allows.

The three paths are not independent. Blocks terminate with learned Laplacian pyramid rescaling, so each operator can pass its state to each operator in the next block. A Laplacian pyramid decomposes a signal into a coarse version plus the band-limited detail you would need to reconstruct the fine version — which makes it the natural currency for exchanging information between paths that live at different resolutions. Fifteen STP blocks in total, and then a final learned spatial resample up to the precision path's resolution.

Note the parameter allocation: 1024 dimensions on the coarsest path, 128 on the finest. Width is spent where tokens are few. The total is about 480M parameters in the shipped model — they also trained a ~1B variant and chose the smaller one "for improved inference efficiency", which is exactly the choice you make when you intend to run 162 million forward passes.

The bottleneck: a direction, not a point

After STP and the time-attention pooling, the summary is upsampled to L and hits the bottleneck. This is where 64 comes from, and its construction is the second-cleverest thing in the paper.

A standard variational bottleneck collapses everything into one latent vector per input. That destroys spatial precision — you would get one embedding per tile, and then you would need to fine-tune a segmentation head to get anything dense back. AlphaEarth instead estimates a von Mises–Fisher distribution at every cell of an L×L grid.

The von Mises–Fisher (vMF) distribution is the sphere's answer to the Gaussian. A Gaussian on Rⁿ has a mean and a covariance; a vMF on the unit sphere Sⁿ⁻¹ has a mean direction μ and a concentration κ. Large κ means the distribution hugs μ tightly; κ = 0 is uniform over the whole sphere.

AlphaEarth's embeddings are the mean directions of vMF distributions on S⁶³, the unit sphere in 64 dimensions, with a fixed κ = 8×10³. Decoding samples from that distribution rather than using μ directly — so the decoder is always fed a slightly perturbed version of the embedding.

That perturbation is a noise channel, and noise channels are how you control smoothness. The paper says it plainly: lower κ means more noise, which regularises the latent manifold, "and this last property is desirable when using embeddings for nearest neighbor retrieval as distances measured along a smooth (lower dimensional) manifold are more meaningful." Higher κ means a cleaner channel and more capacity. Their sweep over embedding dimension and κ found that datasets with big legends — fine crop types, 40-class land use, 39 tree genera — prefer higher dimension and higher κ, while small-sample regimes prefer noisier bottlenecks. They settled on D = 64, κ = 8e3.

(b) Batch uniformity: spreading out on purpose

A representation trained only to reconstruct can cheat by collapsing. If half the planet maps to nearly the same vector and the decoder is expressive enough to still reconstruct on average, reconstruction loss is happy and your embedding is useless for retrieval.

AlphaEarth's counter-pressure is elegant. Take the batch of embeddings, rotate it by one position along the batch axis to get u′, and now u_i and u′_i are a pair of embeddings from two unrelated places. Minimise the sum of |u_i · u′_i|.

The logic runs through a fact about high dimensions: random unit vectors are nearly orthogonal. Concretely, for two uniformly random unit vectors in R^d, the cosine has mean 0 and standard deviation 1/√d. For d = 64:

So if the embeddings really were uniform over S⁶³, the average |u · u′| would sit at about 0.10, not 0. The objective cannot reach zero and is not meant to — it is a repulsion pressure whose equilibrium is uniformity. The paper is careful to note the loophole: "alone, there are perfectly valid non-uniform distributions for which this tends to zero e.g. clusters of points on opposite poles." Orthogonality is a necessary condition for uniformity, not a sufficient one. In practice, combined with reconstruction, it does the job: setting the weight above zero "prevented collapse scenarios where this term would tend to 1 otherwise."

A term tending to 1 means every pair of unrelated places has |cos| = 1 — total collapse onto a line. That is the failure this 0.05 is buying insurance against.

The honest footnote, which the paper supplies: they settled on b = 0.05, and a later sweep over b ∈ {0, 0.001, 0.005, 0.01, 0.1} found 0.005 to be optimal, with b = 0 and b = 0.1 the worst. The effect size varies a lot by evaluation: GLaNCE land-cover linear-probe balanced accuracy was ~66.6% at b = 0.005 versus ~64.7% at b = 0, but LUCAS land cover moved only from ~34.2% to ~34.7%. Two points on one benchmark, half a point on another, from a term nobody would think to include.

(c) Consistency: the same place, whatever the weather

Chapter 1's cloud arithmetic said Borneo gets 11 usable optical scenes and France gets 36. If the embedding for a place depends on how many scenes happened to be clear, then your embedding field encodes the weather during the satellite overpass, and every map you make from it has a cloud-climatology artefact baked in.

The fix is a teacher and a student that share parameters — the same weights, two forward passes. The teacher sees everything. The student's inputs are deliberately mutilated:

Both models are then asked for an embedding over the same summary period, and the loss is

which is a clean rescaling of cosine distance onto [0, 1]: identical directions give 0, orthogonal gives 0.5, antipodal gives 1.

Read strategies (b) and (c) again. Dropping the latter six months and then asking for a summary of those six months is forecasting. Dropping the former six and asking about them is backcasting. The consistency objective is quietly training the model to produce a stable embedding for a period it did not observe, matching what it would have produced had it observed it. That is where the extrapolation capability comes from, and it is trained for free as a side effect of robustness.

The paper flags the residual: "while considerably reduced, tile artifacts are still visible in our embedding fields layers resulting from irregular inputs, and these could be removed in future work with a more aggressive consistency objective term." A weight of c = 0.02 balances "match the teacher" against "actually reconstruct well" — push it too far and the easiest way to agree is for both to say the same bland thing.

(d) Text: a whisper of semantics

The smallest term. A frozen Gemini language model reads geotagged text — Wikipedia articles joined with GBIF species-occurrence records — associated with the training location and date range. An MLP decoder, conditioned on the language model's output and a summary period, produces a vector aligned to the teacher's embedding via a standard CLIP loss.

Weight d = 0.001. One thousandth of reconstruction. The intent, in the paper's words, is that "embeddings characterizing points on Earth's surface with similar semantics will cluster together" — a gentle pull that makes two ecologically similar places on different continents land closer than raw reflectance alone would put them.

The ablation is telling: adding the "Annotated" source group (NLCD plus Wikipedia) improved all land-use and land-cover evaluations — but "interestingly this does not lead to the biggest performance gain compared to adding radar and lidar data." Physics beat semantics. On this problem, knowing the canopy structure helps more than knowing what people wrote about the place.

The tensor walk, end to end

Here is one tile through the whole system, with shapes. This is the part to be able to reproduce from memory.

shapes# ── INPUT ──────────────────────────────────────────────
# One inference tile: 1.28 km x 1.28 km on the ground, 10 m cells
L = 128                       # 1280 m / 10 m
frames = 103                  # 65 Sentinel-2 + 17 Sentinel-1 + 21 Landsat
# per source i: (N_i, C_i, L, L) resampled to a common 10 m grid
# plus per-frame timestamps t_j (millisecond epoch) -> sinusoidal timecodes
# plus per-frame validity masks (swath edges, missing data, perturbations)

# ── INPUT PROJECTORS ───────────────────────────────────
# each source -> shared latent, downscaled to L/2
x = project(sources)          # (103, D, 64, 64)

# ── STP ENCODER, x15 blocks ────────────────────────────
#   space  path: (103, 1024,  8,  8)   ViT-style spatial self-attention
#   time   path: (103,  512, 16, 16)   time-axial attention, timecode-conditioned
#   precis path: (103,  128, 64, 64)   3x3 convolutions
#   blocks exchange state via learned Laplacian pyramid rescaling
x = stp(x)                    # (103, D_p, 64, 64)   one map per input frame

# ── TIME-CONDITIONAL SUMMARY ───────────────────────────
q = timecode_query(t_s, t_e)  # learned query from the VALID period
s = time_attn_pool(x, q)      # (D_p, 64, 64)   103 frames -> 1
s = learned_upsample(s)       # (D_p, 128, 128)

# ── vMF BOTTLENECK ─────────────────────────────────────
mu = mean_direction(s)        # (64, 128, 128), unit norm along dim 0
u  = sample_vmf(mu, kappa=8e3)

# ── PER-PIXEL IMPLICIT DECODERS (training only) ────────
for i in decoded_sources:                       # all 10 sources
    z = concat([u, timecode(t), sensor_meta_i]) # per pixel
    y_hat_i = mlp_i(z)                          # 2 hidden layers, width 512
    loss += w_i * f_i(y_i, y_hat_i)             # L1, or cross-entropy for NLCD

# ── INFERENCE OUTPUT ───────────────────────────────────
# quantise mu to int8, trim the 80 m overtiling margin
tile = quantize_s8(mu)[:, 8:120, 8:120]         # (64, 112, 112) int8
# 112 * 112 * 64 = 802,816 bytes per tile

Two things to notice. First, the decoders exist only during training — at inference you keep μ and throw the entire decoder stack away. Second, the output is dense: 112×112 independent embeddings per tile, not one. That density is what "embedding field" means and it is why no fine-tuning is needed to get pixel-level maps.

Training run

512 TPU v4 for 56 hours is roughly 28,700 chip-hours. That is a serious run, but it is not a frontier-LLM run — and it is paid once for a representation the whole field reuses. Compare it against the alternative: several hundred organisations each building bespoke pipelines, forever.

Note the warmup: 1% of the schedule ramping up, 99% linearly decaying. And note that the training set covers only 1.1% of land. The model has never seen 98.9% of the surface it is asked to embed. Generality is not optional here; it is the entire deployment assumption.

Sixty-four bytes: the quantisation, worked by hand

The embeddings come out of the network as 32-bit floats. Sixty-four of those is 256 bytes, which would make the annual global layer 381 TB. The released layers are int8, one byte per dimension, and this is the arithmetic that gets them there.

The obvious scheme is linear: multiply by a scale, round, clip. AlphaEarth uses something better. In their notation, with power and scale as parameters:

pythondef quantize(x, power, scale, min_value, max_value, dtype):
    sat     = jnp.abs(x) ** (1 / power) * jnp.sign(x)
    snapped = jnp.round(sat * scale)
    return jnp.clip(snapped, min_value, max_value).astype(dtype)

def dequantize(y, power, scale, dtype):
    rescaled = y.astype(jnp.float32) / scale
    return jnp.abs(rescaled) ** power * jnp.sign(rescaled)

For int8 the parameters are scale = 127.5, clipped to [−127, 127], and they chose power = 2. So encoding takes a square root and decoding squares it back. The paper's justification: "the exponentiation was introduced to preserve information in the least significant digits."

Let us verify that claim with actual arithmetic, because it is a lovely small result. Remember the components live on the unit 63-sphere, so a typical magnitude is around 1/√64 = 0.125, and many components are much smaller.

Case 1: a small component, x = 0.0100.

Square-root companding (power = 2): √0.0100 = 0.100000. Times 127.5 gives 12.750, which rounds to 13. Decoding: 13/127.5 = 0.101961; squared gives 0.010396. Error 0.000396, a relative error of 4.0%.

Plain linear: 0.0100 × 127.5 = 1.275, rounds to 1. Decoding: 1/127.5 = 0.007843. Error 0.002157, a relative error of 21.6%.

The linear scheme spent one code point on a value it got 22% wrong. The companded scheme spent thirteen code points on the same value and got it to 4%.

Case 2: a typical component, x = 0.1250. Companded: √0.1250 = 0.353553, ×127.5 = 45.078, rounds to 45; back: (45/127.5)² = 0.124567, error 0.000433. Linear: 0.1250 × 127.5 = 15.9375, rounds to 16; back: 0.125490, error 0.000490. Roughly a tie.

Case 3: a large component, x = 0.9000. Companded: √0.9 = 0.948683, ×127.5 = 120.957, rounds to 121; back: (121/127.5)² = 0.900638, error 0.000638. Linear: 114.75 rounds to 115; back: 0.901961, error 0.001961.

The pattern: companding buys a large factor on small components and loses nothing on large ones, because taking a square root stretches the crowded region near zero across more code points. Since embedding components on a 64-sphere are mostly small, that is exactly the right place to spend precision.

The paper's own verdict on the risk they took: "We did not initially assume that 8-bits ought to be enough for any evaluation given that quantization was not part of the learning process, we were nonetheless pleased to note little performance variability compared to the non-quantized embeddings." Quantisation was applied post hoc, with no quantisation-aware training, and the evaluations barely moved. That is a strong signal that the vMF noise channel had already smoothed the manifold enough to absorb one byte of rounding.

Making 1.5 trillion embeddings: the tiling system

Finally, how the field is actually produced. The world is divided into UTM zones, and each zone is tiled by 960 m × 960 m output tiles. But inference does not run on a 960 m tile. Each is buffered by 160 m on every side, giving a 1.28 km × 1.28 km input tile — 128×128 cells at 10 m, which is where the L = 128 in the tensor walk comes from. After the forward pass, the outer 80 m is trimmed and the remainder is rendered back onto the zone.

The arithmetic of that margin, in pixels: input 128×128; trim 8 pixels (80 m) from each edge leaving 112×112; place tiles on a 96-pixel stride. So neighbouring tiles overlap by 112 − 96 = 16 pixels. This is called overtiling, and its purpose is simple: a convolutional and attentional model has less context at the edge of its input, so edge predictions are worse. Throw the edges away and let a neighbour, for whom that ground was interior, supply the answer.

Now count the forward passes for one annual global layer:

One hundred and sixty-two million forward passes of a 480M-parameter model, per year of coverage, run entirely on Earth Engine, with the paper noting that "a great degree of care was taken to ensure our inference system respects the shared capacity of Earth Engine while scaling to hundreds of billions of observations."

This is the number that separates a paper from a product, and it is also why they picked the 480M variant over the 1B one. Doubling the parameters doubles a bill you pay 162 million times.

Chapter 3: Locations Without Pictures

AlphaEarth spends 28,700 chip-hours and 162 million forward passes to answer "what is at this place?" from imagery. SatCLIP asks a deliberately smaller question and answers it with about a million parameters and no imagery at all at query time: given only a latitude and longitude, what should I expect the ground to be like?

That sounds like cheating, and in an important sense it is — a coordinate cannot tell you whether this particular field was planted. But it can tell you an enormous amount, because places are not independent samples. Elevation, climate, biome, soil, and land-use history are all smooth-ish functions of position, and a downstream model that gets them for free needs fewer labels.

Why raw latitude and longitude is a bad input

Start with the naive thing and watch it fail. Feed (lat, lon) as two floats into an MLP.

Failure 1: the antimeridian. Two points on the equator, one at longitude 179.9° and one at −179.9°. On the ground they are about 22 km apart — effectively neighbours. In the input space they differ by 359.8. Any model with a smoothness prior will treat them as maximally distant, and any interpolation across the Pacific is nonsense.

Failure 2: the poles. At latitude 89.9°, all 360 degrees of longitude fit inside a circle about 22 km across. Longitude is nearly meaningless there, yet the network sees it varying over its full range. The input coordinate system's metric bears no relation to distance on the ground.

Failure 3: no useful basis. Even ignoring the topology, two raw numbers give a network nothing to build multi-scale structure out of. It has to learn everything, from continental gradients down to regional detail, from scratch.

The fix, step one: put the point back on the sphere

All three failures come from describing a sphere with two numbers that do not respect its geometry. So use three that do. With colatitude θ = 90° − latitude and longitude φ:

Work it for Nairobi, at latitude −1.29°, longitude 36.82°. Then θ = 91.29° and φ = 36.82°:

Check the norm: 0.800319² + 0.599151² + 0.022513² = 0.640511 + 0.358982 + 0.000507 = 1.000000. On the sphere, as promised.

Now re-run the antimeridian test. At latitude 0, θ = 90° so sinθ = 1, z = 0.

A separation of 359.8 in raw longitude became 0.003491 in Cartesian coordinates. The discontinuity is gone, entirely, for free, by choosing coordinates that respect the manifold. And at the pole, all longitudes converge to (0, 0, ±1) — the degeneracy is represented honestly instead of being faked.

The fix, step two: a full basis on the sphere

Three Cartesian coordinates fix the topology but give the network only the coarsest possible description: one linear function per axis. To let it express structure at many scales you want a basis — a family of functions on the sphere, from very smooth to very wiggly, that any reasonable function can be written as a combination of.

On a line that family is sines and cosines, and the combination is a Fourier series. On a sphere the analogous family is the real spherical harmonics Y_l^m(θ, φ), indexed by a degree l ≥ 0 and an order m with −l ≤ m ≤ l. They are orthogonal over the sphere, which is exactly the property that makes a Fourier basis useful: each one carries information the others do not.

The lowest ones are worth writing out, because they demystify the whole apparatus:

That second row is the punchline of this chapter. The degree-1 spherical harmonics are the Cartesian coordinates. Step one was not a separate hack — it was the first non-trivial term of the very series we are now writing down. Everything above degree 1 is refinement.

Count the basis functions for a truncation at L degrees (using l = 0 to L−1):

Verify for L = 3: 1 + 3 + 5 = 9 = 3². So SatCLIP's two published configurations give:

(Using ~111 km per degree of great-circle arc.) So L is a literal resolution dial with a physical unit attached. L = 10 cannot represent anything finer than a couple of thousand kilometres — continental-scale climate gradients and nothing else. L = 40 gets down to roughly the size of a small country.

Neither is anywhere near 10 m. That is not a defect; it is the design. SatCLIP is not trying to tell you about a field. It is trying to tell you about a region, cheaply, from a coordinate.

Siren: the network on top of the basis

The spherical harmonics are fixed — no parameters, just mathematics. The learned part is a small network on top. In the general form used across the location-encoder literature:

where PE is a non-parametric positional encoding (here, spherical harmonics) and NN is a small network. SatCLIP uses a Siren — a sinusoidal representation network, an MLP whose activation is sin(·) instead of ReLU. Two hidden layers, 512 wide, about a million parameters total, output 256 dimensions.

Sinusoidal activations are used because their derivatives are also sinusoids, so a Siren can represent high-frequency detail without the piecewise-linear blockiness a ReLU MLP produces. For an implicit representation of a signal over a continuous domain — which is exactly what a location encoder is — that matters.

The parameter budget is worth pausing on. One million parameters, 256-dimensional output, no imagery required at query time. You can evaluate this on a CPU in microseconds and ship the weights in an email attachment. Against AlphaEarth's 95 TB, that is the entire "implicit vs explicit" tradeoff in one comparison.

The training objective: CLIP, with coordinates on one side

Now, how do you train a coordinate encoder with no labels? Contrastively, against imagery. The dataset is pairs (c_i, I_i) — a coordinate and the satellite image taken there. Two encoders map both into a shared 256-dimensional space, and the loss is symmetric InfoNCE:

The bracket ⟨·,·⟩ is a normalised dot product — a cosine — and τ is a temperature. Both directions are included: "which image belongs at this coordinate?" and "which coordinate does this image come from?" A batch of N gives N correct pairs and N² − N incorrect ones with nobody annotating a single negative.

The pretraining data is S2‑100K: 100,000 Sentinel‑2 tiles, 256×256 pixels, 12 spectral channels, with their centroid coordinates. The critical property is not the size but the sampling: it is nearly uniform over global land mass, by construction.

Training details worth keeping:

The batch-size finding deserves a note. Contrastive learning usually wants enormous batches, because batch size is the number of negatives. Here 8k beat 32k. The plausible reason is specific to this problem: with 100,000 total tiles, a 32k batch contains a third of the dataset, so many "negatives" are geographic neighbours whose images genuinely do match the anchor coordinate. Pushing them apart teaches a falsehood. The optimal batch size is bounded above by how densely your negatives sample the space.

What SatCLIP buys, and what it cannot

Nine downstream tasks — air temperature, elevation, population density, median income, California housing, biomes, ecoregions, country codes, iNaturalist species. All take raw coordinates as input, transformed to embeddings, then an MLP. SatCLIP wins seven of nine by a large margin.

The two losses are diagnostic: California Housing (California only) and Median Income (continental US only). On both, GeoCLIP — trained on US-heavy MP‑16 — matches or beats it. A globally uniform encoder is worse on a single-region task than an encoder that overfits that region. Uniformity has a price and this is it.

The L ablation is even more informative:

Hold that against the scale of the problem: in the Africa ecoregion evaluation, training locations are on average 480 km from their nearest neighbour. At that sparsity a 512 km-resolution basis is fitting noise between samples, while a 2,220 km basis is doing honest interpolation. The right resolution is a function of your label density, not of how much detail exists in the world.

And then the limitation that ends the chapter. SatCLIP has no time axis. A coordinate is a coordinate; there is nowhere to put a date. The AlphaEarth paper therefore omits SatCLIP from every change-detection evaluation, noting these approaches "are location-only, i.e., have no time handling or way to differentiate between observations at different times," and expects "poor performance on evaluations where landscapes are dynamic (e.g. agriculture)."

The mirror-image weakness

Here is the detail that makes this chapter worth its length. AlphaEarth receives no coordinate information at all. It sees only pixels.

In the data-scaling study, the two models' weaknesses are exact mirrors. AlphaEarth beats every baseline at equal observation counts on nearly every evaluation — except US trees, tree-genus classification, where the paper reports it needs "an additional ~100x observations compared to SatCLIP," and speculates that this "is related to AEF receiving no coordinate information, therefore requiring more examples to learn climate gradients."

Of course. Tree genus is heavily determined by climate zone, and climate zone is nearly a function of latitude and elevation. SatCLIP is handed that for free in its first three basis functions. AlphaEarth has to infer climate from a hundred billion pixels of what the ground looked like across seasons.

Chapter 4: Clay & the Open Stack

AlphaEarth is a raster you download. SatCLIP is a function you evaluate. The third shape — and the one most of the open-source world has converged on — is a pretrained encoder you run yourself. Clay is the most complete example, and it is Apache‑2.0 all the way down.

This chapter covers Clay properly, then places it in the lineage that produced it, and then makes the argument that matters most in practice: most of your accuracy lives in the harness around the model, not the model.

Clay v1.5, from the outside in

Clay is a masked autoencoder (MAE) over satellite imagery. The MAE recipe is simple and worth restating from zero: cut the image into patches, hide most of them, feed only the visible patches to a Transformer encoder, and train a small decoder to reconstruct the hidden ones. If you can fill in three-quarters of an image you did not see, your encoder learned something about what images of that kind contain.

Clay v1.5's configuration, from the released model card:

Do the token arithmetic, because it explains why an MAE is affordable at this scale. A 256×256 input with patch size 8 gives

At 75% masking, only 256 of those go into the encoder. Attention is quadratic, so the encoder does 256² = 65,536 pairwise interactions per layer instead of 1,024² = 1,048,576 — a 16× saving, which is exactly (1/0.25)². The 24-layer, 1024-wide encoder only becomes trainable at all because it sees a quarter of the tokens. The decoder that reconstructs all 1,024 positions is deliberately tiny (depth 4, dim 512) so the pressure lands on the encoder.

The dynamic embedding block: any sensor, any bands

Here is Clay's real contribution, and it is a good one. A normal ViT has a fixed patch-embedding layer: a linear map from (patch × patch × C) to the model dimension. Fix C and you have fixed your sensor. Sentinel‑2 has 10 usable bands in Clay's spec, Landsat 6, Sentinel‑1 SAR 2, NAIP 4, LINZ 3, MODIS 7. Six different Cs.

Clay's dynamic embedding block generates the patch embedding as a function of the bands present and their central wavelengths. Instead of "channel 3 of my fixed layout," a band is described by the physical quantity it measures: this channel is centred at 665 nm, this one at 842 nm, this one is C‑band radar. The model learns a function of wavelength rather than a lookup table of channel indices.

The consequence is that Clay accepts, in its own words, "inputs of any size and any number of bands." Hand it a three-band aerial photo or a ten-band Sentinel‑2 stack and both work, and — more importantly — what it learned about the near-infrared from Sentinel‑2 transfers to the near-infrared of NAIP, because to the model they are nearby points in wavelength space rather than unrelated channel slots.

Its position encoding is similarly explicit. Clay adds encodings for:

Notice that Clay and AlphaEarth solve the same problem in opposite directions. AlphaEarth gives measurement metadata to the decoder so the embedding stays clean of it. Clay gives it to the encoder so the embedding is conditioned on it. Neither is wrong; they produce different objects. AlphaEarth's embedding aims to be invariant to how you looked; Clay's aims to be informed by it.

Clay's tensor walk

The shapes, so you could implement it. Take a Sentinel‑2 chip with 10 usable bands:

shapes# ── INPUT ──────────────────────────────────────────────
pixels      = (B, 10, 256, 256)   # C varies by sensor: 10 S2, 6 Landsat, 2 S1, 4 NAIP...
wavelengths = (10,)                # central wavelength of each band, in nm
gsd         = scalar               # 10.0 m for Sentinel-2, 30.0 for Landsat
latlon      = (2,)                 # chip centroid
timestamp   = (week_of_year, hour) # both cyclic -> encoded on a circle

# ── DYNAMIC EMBEDDING BLOCK ────────────────────────────
# patch weights are GENERATED from the wavelengths, not looked up by index
tokens = dynamic_patch_embed(pixels, wavelengths)   # (B, 1024, 1024)
#                              32*32 patches ─┘     └─ model dim

tokens += pos_encoding(gsd) + place_encoding(latlon) + time_encoding(timestamp)

# ── MASK 75% ───────────────────────────────────────────
keep   = tokens[:, visible_idx, :]                  # (B, 256, 1024)  <- 1/4 of them

# ── ENCODER: 24 layers, 16 heads, dim 1024 ─────────────
z = encoder(keep)                                   # (B, 256, 1024)

# ── DECODER: shallow on purpose (depth 4, dim 512) ─────
full   = scatter(z, mask_tokens)                    # (B, 1024, 512)
recon  = decoder(full)                              # (B, 1024, 8*8*10) pixel patches
loss   = 0.95 * l2(recon, hidden_patches) + 0.05 * rep_loss(z, dinov2(pixels))

# ── INFERENCE: no masking ──────────────────────────────
z_all = encoder(tokens)                             # (B, 1024, 1024) — all patches
emb   = z_all.mean(axis=1)                        # (B, 1024) — ONE of several options

Two lines deserve comment. The decoder is shallow on purpose: depth 4 against the encoder's 24. If the decoder were powerful it could reconstruct from a weak representation, and the pressure would leave the encoder entirely. Starving the decoder is what forces the encoder to do the work — the same logic as AlphaEarth's two-hidden-layer MLPs.

And that last line is the one the AlphaEarth authors had to make a judgement call on. A mean over 1,024 patch tokens is one choice; the class token is another; a mean over a subset is a third. They report evaluating a variant whose pooled dimensionality is 768 (Clay v1.5's encoder is 1024-wide, so the number reflects the specific variant and pooling they benchmarked). The value of "the Clay embedding" is not a property of the weights alone.

The DINOv2 teacher: 5% that is not decoration

Pure reconstruction has a well-known pathology: the easiest way to reduce pixel error is to get the texture statistics right, which does not require any semantic understanding. Clay counters with a representation loss against a frozen DINOv2 teacher, weighted at 5% against reconstruction's 95%.

DINOv2 is a self-supervised vision model trained on natural photographs. It has never seen a satellite image from above. What it has is a strong, general notion of "these two crops depict the same kind of thing," and pulling Clay's features toward that notion supplies a semantic gradient that pixel reconstruction does not.

Compare the weightings across the three anchors and a pattern appears: AlphaEarth puts 0.001 on its language term against 1.0 on reconstruction; Clay puts 0.05 on its teacher term against 0.95. Both are reconstruction models with a small semantic corrective. SatCLIP is the odd one out — pure contrastive, no reconstruction at all — and it is also the one with no time axis and 100× less data. The correlation is not proof, but it is suggestive: when your supervision is the physics of the signal, reconstruction is the objective that scales.

Training run and openness

The identical train and validation loss is worth a remark: at 70 million chips and 100 epochs, with 75% of every chip masked every time, the model is nowhere near memorising. This is a data-rich, not a capacity-rich, regime.

And note that Clay publishes precomputed embeddings as well as weights. That is the explicit-field idea arriving in the open ecosystem — the same insight AlphaEarth built its whole product around, offered as an option rather than the only interface.

The lineage

Clay's own documentation credits its ancestors, and they are the right ones to know. Each solved one problem that satellite imagery poses to a vision transformer:

The AlphaEarth paper's own framing of the field before it: geospatial foundation models "can be roughly characterized as derivatives of SatMAE or implicit models such as SatCLIP," and "none satisfy all of the following key properties: (1) multi-source or multi-modality, (2) inclusion of time into the modeling framework, or (3) spatial resolution at a precision useful for serving operational mapping use cases."

And then the sentence that should keep everyone honest: "existing learned featurization approaches don't always outperform designed featurization methods in scarce data regimes." Hand-built spectral composites and CCDC harmonics were, until 2025, frequently beating deep geospatial models on real sparse-label tasks. In the AlphaEarth evaluations, MOSAIKS — random convolutional features, not a trained deep model — is repeatedly the runner-up.

The harness is where the accuracy lives

Now the practical lesson, which comes from reading how the AlphaEarth authors evaluated everyone else's models rather than from any model card.

To score Clay on their suite, they had to decide: what patch size (256×256), what nominal scale (10 m for Sentinel, 30 m for Landsat), what cloud threshold (filter above 20% cover), how many frames (capped at 30, to fit a V100's memory), how to normalise (statistics from each evaluation's own training split), and — the big one — how to turn a grid of patch tokens into one vector.

On that last point the documentation offers several options: take the class token, or average the non-class tokens. The authors tried multiple variants and report the winner: "we average over per-source per-time spatial tokens for a final dimensionality of 768."

The same care applies to Prithvi: HLS L30 imagery, 224×224 at 30 m, cloud threshold 20% (raised to 50% for the oil-palm evaluation because imagery was too scarce otherwise), walking backwards in time by the revisit period until an image is found, and — for the under-1% of points where no image passes at all — substituting the mean embedding over the dataset.

Count the decisions. Patch size, scale, cloud threshold, frame cap, normalisation source, pooling strategy, missing-data fallback. Seven, each defensible, each swinging the result. A model card gives you none of them.

Choosing between the three shapes

A decision rule that survives contact with real projects: if your area of interest is on land, below 82°, and your question is annual, use the precomputed field — it is free, reproducible, and 64 bytes. If you need a sensor, resolution, or cadence the field does not cover — drone imagery, sub-weekly monitoring, a band nobody included — run Clay. If you have almost no labels and your target is a smooth environmental variable, concatenate SatCLIP; it costs a millisecond and 1 KB.

Chapter 5: Search the Planet by Hand

Everything so far has been about producing the field. This chapter is about the single operation that makes it useful, and we are going to do it entirely with pencil arithmetic on nine cells before scaling to 1.5 trillion.

The operation is: here is one place I care about; show me everywhere else that looks like it. No labels, no training, no model. Just the field and a dot product.

A toy field

Take a 3×3 patch of ground. Real embeddings have 64 dimensions and are not interpretable; ours will have 4 and will be, so we can check that the arithmetic agrees with intuition. Call the axes persistent green vegetation, bare soil / seasonal exposure, open water, and built structure.

Here is the raw field, before normalisation:

Step 1: normalise, and understand why

AlphaEarth's embeddings live on S⁶³, so every vector has length 1. Ours do not yet, so normalise. For the query cell:

Sanity check by squaring: 0.467532 + 0.467532 + 0.051948 + 0.012987 = 0.999999. Good.

Why does normalising matter? Suppose one cell were [12, 12, 4, 2] — exactly twice our query, same direction, same land cover, just a brighter or better-illuminated observation. Its raw dot product with q is 2 × 77 = 154, double everything else, and it would top any ranking by magnitude alone. Length is almost always a measurement artefact — illumination, atmosphere, gain. Direction is the content. Putting everything on the sphere deletes the artefact by construction, which is one of the three reasons Chapter 2 gave for the vMF bottleneck.

Step 2: cosine similarity, nine times

The cosine between the query and a cell is

and because the numerator is bilinear you can dot the raw integers and divide by the two norms at the end. Far less arithmetic. Take cell (0, 0), forest, v = [9, 1, 0, 1]:

And cell (1, 0), the neighbouring cropland, v = [6, 6, 1, 1]:

All nine, sorted:

Set a threshold at 0.95 and you retrieve exactly the three cropland cells. That is a map — a binary cropland mask over this patch — produced by one click and nine dot products, with no training and no labels.

Why a dot product is the right question to ask

It is worth being explicit about why this operation, of all operations, is the one the whole architecture is built around.

A dot product a · b is the length of a projected onto the direction of b, times the length of b. When both are unit vectors it is exactly the projection: how much of a points along b. So "cosine similarity" is not a similarity heuristic somebody invented — it is the answer to a precise question: if I decompose this cell's description into "the part that looks like my query" plus "everything else", how big is the first part?

Three properties make it the right primitive for a field:

The third is the practical one. Any similarity that is not a bilinear form — a learned comparison network, a cross-attention score — forces you to run something per candidate. A dot product lets you run one thing per query. Chapter 8 returns to this as the fundamental bargain behind every precomputed index.

What one byte per dimension does to the ranking

Chapter 2 quantised to int8 and asserted the error was negligible. Now we can check that claim on a ranking, which is what actually matters.

Round-trip the query through the shipped scheme (power = 2, scale = 127.5). Each component in turn: √0.683763 = 0.826901, times 127.5 is 105.43, rounds to 105. √0.227921 = 0.477411, times 127.5 is 60.87, rounds to 61. √0.113961 = 0.337581, times 127.5 is 43.04, rounds to 43. So the whole query is four bytes:

Decode by squaring: [0.678201, 0.678201, 0.228897, 0.113741]. Its norm is 0.992594 — slightly off the sphere, as promised — so renormalise to [0.683261, 0.683261, 0.230604, 0.114590]. The cosine between the original and the round-tripped query is

Do the same for every cell and re-rank:

Every score moves by less than 0.0022, and the top of the ranking is untouched — the three cropland cells stay first, second and third, which is the entire result you were going to use. That is the paper's claim of "little performance variability" reproduced at toy scale.

But look at rows four and five. Exactly, forest was 0.76304 and bare ground was 0.76190; after quantisation forest is 0.76147 and bare ground is 0.76163. Their order has flipped. The perturbation is a thousandth, and it was enough, because the gap it had to survive was a thousandth.

Step 3: read the failure in the numbers

Look at ranks 5 and 6. Mature forest scores 0.76304. Bare quarry ground scores 0.76190. They differ by 0.00114, about one part in seven hundred — and they are as different as two land covers get.

Why? Our query is cropland, which sits between forest and bare ground: it has real vegetation (like forest) and real exposed soil (like the quarry). Cosine similarity collapses a 4-dimensional relationship onto one number, and one number cannot distinguish "similar in the green direction" from "similar in the soil direction."

Three lessons follow, and all three bite in production:

Cosine or Euclidean? On a sphere it does not matter

A question that comes up immediately: should you rank by cosine similarity or by Euclidean distance? On the unit sphere the question dissolves. Expand the squared distance between two unit vectors:

Euclidean distance is a strictly decreasing function of cosine, so the two rankings are identical. Check it on two of our rows:

Same order, different units. This equivalence is exactly why forcing embeddings onto a sphere is such a friendly decision downstream: every vector index, every kNN library, every clustering algorithm agrees with every other one about what "near" means.

Scaling from 9 to 1,500,000,000,000

Nine cells is a table. The planet is 1.49 × 10¹² cells. What does the same operation cost there?

A brute-force scan is one dot product per cell: 64 multiply-adds. So a full-planet search is about 9.5 × 10¹³ multiply-adds — large, but embarrassingly parallel and, crucially, memory-bound rather than compute-bound. Which makes the byte count the number that matters:

Read the first row again. Sixty-four megabytes for a hundred square kilometres at 10 m — that fits in a browser tab. The same region in Clay embeddings does not fit in most laptops' RAM. And int8 dot products vectorise beautifully on ordinary CPU SIMD, several times faster per byte than float32.

So the 64-byte decision is not a storage footnote. It changes where the search can run: on a laptop, in a browser, at the edge, instead of in a cluster. That is the difference between a tool an ecologist uses and a service an ecologist requests.

Threshold, top-k, or an index?

Three ways to turn a similarity column into an answer, and they fail differently.

For a field, the brute-force scan is usually the right answer and the reason is the byte count again. An ANN index over 100 million vectors is a build step, a memory overhead, and a recall parameter to defend. A brute-force int8 scan over the same 100 million vectors touches 6.4 GB and finishes in seconds — and it is exact, which means "every cell above 0.95" is a claim you can actually make. Approximation is a tool for the regime where the field does not fit in memory, and 64 bytes pushes that regime a long way out.

The realisation: what the code actually is

pythonimport numpy as np

# field: the raster you downloaded, one int8 vector per 10 m cell
field = load_embeddings(bbox, year=2021)        # (H, W, 64) int8
H, W, D = field.shape

# dequantise exactly as the paper specifies: power=2, scale=127.5
r = field.astype(np.float32) / 127.5
X = np.abs(r) ** 2 * np.sign(r)                    # (H, W, 64) float32
X = X.reshape(-1, D)                                # (H*W, 64)
X /= np.linalg.norm(X, axis=1, keepdims=True)      # back onto the sphere

# the query: one cell you clicked on
q = X[row * W + col]                                # (64,)

# the entire search. one matrix-vector product.
sim = X @ q                                         # (H*W,) in [-1, 1]
mask = (sim > 0.95).reshape(H, W)                  # your map

Six meaningful lines. No model is loaded, no imagery is fetched, no GPU is touched. Note the renormalisation after dequantisation: the round-trip through int8 knocks vectors very slightly off the sphere, and if you skip the fix, cells whose components happened to round outward gain a small persistent advantage in every comparison — the exact magnitude artefact normalisation exists to kill.

Chapter 6: What the Field Unlocks

Chapter 5 did retrieval with zero labels. This chapter adds a handful of labels and gets maps, change masks, and continuous physical variables. Four capabilities, each derived, each with the paper's real numbers attached.

1. Thematic mapping with a linear probe

A linear probe is the smallest possible task head: one matrix multiply and a softmax, fitted on frozen features. For a C-class legend over D-dimensional embeddings:

Count the parameters, because in a sparse-label regime this is the entire argument for a small embedding. LCMAP land cover is a 6-class legend, and the evaluation's max-trial size is 300 labelled points per class:

At 4.62 samples per parameter the fit is over-determined and the solution is stable. At 0.39 it is under-determined — there are more free parameters than equations, infinitely many zero-training-error solutions exist, and which one you get is decided by your regulariser rather than by your data.

Now do the same for LUCAS land use, a 40-class legend, still 300 per class:

The ratio D_Clay/D_AEF = 12 is preserved exactly, because both the parameter count and the sample count scale linearly in C. Dimensionality is a fixed tax you pay on every task, and it is heaviest precisely where labels are scarcest.

Fitting a probe by hand — and watching it fail

Take the toy field from Chapter 5. Suppose you have labelled four cells: cropland at (1, 0) and (1, 2), not-cropland at (0, 0) forest and (2, 1) town. The cheapest possible probe is nearest class mean: average each class, normalise, assign each cell to whichever mean it is closer to.

The decision rule — "is x closer to ĉ_crop than to ĉ_other?" — is x · ĉ_crop > x · ĉ_other, which rearranges to a single dot product against a weight vector. So this is a linear probe, with

and it is interpretable: strongly positive on the soil axis, strongly negative on the built axis, mildly negative on persistent green. "Cropland is bare-ish and not a town." Reasonable.

Score the unlabelled cells:

Two failures out of four, and both are instructive. The quarry scores higher than the true cropland cell, because it is maximally soil-like and the probe learned "soil means crop." The river scores positive by default — not because it looks like cropland, but because it looks even less like the negative class, which was built entirely from forest and town.

Two lessons, neither about the embeddings:

Which is exactly why the paper's evaluation protocol looks the way it does: balanced samples per class, a minimum spacing of 1.28 km between sample points so neighbouring pixels do not leak between train and test, and transfer via "methods with minimal parameters: k-nearest neighbors, and linear layers fit to the features."

The paper's headline thematic-mapping result, across 11 classification evaluations: the largest error reductions come on the annual evaluations — LCMAP land cover, Descals oil palm, Africa crop mask, LCMAP land use. That is the continuous-time machinery paying off, since an annual valid period is exactly what those labels describe. All thematic evaluations except Ethiopia crops showed error reductions above 1.0× within the ~90% confidence interval.

Fitting the probe properly, and when zero error means nothing

The nearest-class-mean probe above was a teaching device. A real probe is fitted by minimising a loss, and for the regression case it has a closed form worth writing down because every dimension in it is a shape you now know:

with N = 1,800 sampled cells and D the embedding dimension. Two costs follow directly:

Seven million multiply-adds is instant on anything. A billion is a coffee break, and it recurs every time you retune. But the compute is not the real cost — the statistics are.

Here is the sharp version of Chapter 6's "label tax," and it comes from a 1965 result of Thomas Cover about how many patterns a hyperplane can shatter. For N points in general position and d = D + 1 free parameters (weights plus bias), the fraction of all 2^N possible labellings that a hyperplane can separate perfectly passes through one half exactly when N = 2d.

Read that as a warning about evidence. Below N = 2(D + 1), a linear separator with zero training error is more likely than not to exist for random labels, which means finding one tells you almost nothing about your features. So the number of labels at which "my probe fits perfectly" starts to be evidence is:

Now put the paper's three regimes against that column. At one shot (6 labels for a 6-class problem) and ten shot (60 labels), every one of these representations is far below its threshold — a perfect fit is guaranteed for any labelling, and zero training error is pure noise. At max trial (1,800 labels), the 64-dimensional field is 13.8× past its threshold and comfortably in the evidence regime, while the 768-dimensional one is only 1.17× past it, sitting right on the shoulder of the transition where margins are thin and generalisation is fragile.

2. Change detection as vector drift

Here is where a field beats a classifier decisively, because the operation needs no labels at all.

You have the same cell in two years. Both are unit vectors. Define drift as

Threshold d and you have a change mask. Let us compute four transitions using the toy vectors, and then look hard at the one that ruins everything.

Work the first one to see there is no magic. u·v on raw integers: 9×1 + 1×9 + 0×0 + 1×2 = 9 + 9 + 0 + 2 = 20. Norms √83 = 9.11043 and √86 = 9.27362, product 84.4867. So cos = 20/84.4867 = 0.236724 and d = 0.763276.

Set a threshold at d > 0.10. Real change (0.763, 0.542) is caught. Ordinary year-to-year variation (0.013) is not. Excellent — until you reach the last row.

The same crop, planted and harvested on a shifted calendar, drifts by 0.0997. That is 0.0003 below the threshold. A slightly warmer spring, a slightly earlier harvest, and this unchanged field is flagged as change. Meanwhile a subtle real conversion — natural forest to a young oil-palm plantation, which is still green and still tall — might drift less than 0.05 and be missed.

The paper measures precisely this. Two protocols on the same LCMAP change labels:

Read the last two rows carefully, because they are the honest ones. Supervision is worth roughly 7 to 8 points of balanced accuracy — that is the price of the phenology collision. And on unsupervised land-use change, an ImageNet-pretrained ViT beats AlphaEarth. The paper says so plainly, "suggesting the value of supervision for this use case."

Land use versus land cover is the reason. Cover is what the ground is made of, and a sensor can see it. Use is what people do with it, and a pasture and a hay meadow can be spectrally identical. An unsupervised drift detector cannot see a change in intent.

3. Search by example, at planetary scale

Chapter 5's operation, run over a continent. Click one illegal mining pit and get every cell on Earth whose 2021 embedding is within 0.05 of it. No labels, no training, no class definition — just "more like this."

The workflow this replaces is worth naming, because it is what most environmental monitoring actually consists of: an analyst who knows what a mining scar looks like, scrolling basemaps. Search-by-example turns their expertise from a scanning operation into a query. And because it needs no labels, it is the fastest way to build the labels for step 1 — retrieve candidates, have the expert confirm or reject a few dozen, then fit a probe on the confirmed set. Retrieval bootstraps supervision.

4. Regression: continuous physical variables

The most surprising capability, because the targets are not visible. Two evaluations:

Emissivity is a close race — three points of R² over random convolutional features. The paper reads the pattern of who does well: MOSAIKS and composites are strong, CCDC and coordinate-only baselines are weak, which "indicates that local spatial patterns and overall reflectance (as opposed to seasonality in reflectance) are important factors for this use case." Emissivity is about what the surface is made of, and a single good look tells you most of it.

Evapotranspiration is the opposite, and the result is dramatic: AlphaEarth is the only approach with R² above 0.2. Every other featurisation is nearly useless. That makes sense once you ask what evapotranspiration depends on: how much vegetation there is, how actively it is transpiring right now, how much water is available, and what the weather has been. It is a dynamical quantity. A model whose embeddings had to reconstruct ERA5 climate and GRACE water storage from optical and radar has been forced to encode exactly that, and a model built on a single composite has not.

Realisation: the entire downstream stack

pythonimport numpy as np
from sklearn.linear_model import LogisticRegression

# ---- 1. thematic map from sparse labels -----------------------
X = sample_embeddings(points, year=2021)     # (1800, 64) float32
y = point_labels                                # (1800,) in 0..5
probe = LogisticRegression(max_iter=2000).fit(X, y)   # 390 parameters

field = load_embeddings(bbox, year=2021)      # (H, W, 64)
flat  = field.reshape(-1, 64)
lc    = probe.predict(flat).reshape(field.shape[:2])   # your land-cover map

# ---- 2. change mask, no labels at all --------------------------
a = load_embeddings(bbox, year=2019).reshape(-1, 64)
b = load_embeddings(bbox, year=2021).reshape(-1, 64)
drift  = 1 - np.einsum('nd,nd->n', a, b)          # (H*W,)
change = (drift > 0.10).reshape(field.shape[:2])

# ---- 3. search by example --------------------------------------
q    = a[row * W + col]
hits = (a @ q > 0.95).reshape(field.shape[:2])

# ---- 4. regression on a physical variable ----------------------
from sklearn.linear_model import Ridge
et = Ridge(alpha=1.0).fit(X_gauges, y_et)         # 65 parameters

Four capabilities, one loaded raster, zero neural networks at query time, and the largest model in the file is a 390-parameter logistic regression. That is what "the field is the product" means when you actually sit down to use it.

Chapter 7: Where It Breaks

An embedding field is a very confident-looking object. It has a value at every cell, it never returns an error, and the values are dimensionless numbers with no units to sanity-check. That combination is dangerous, so this chapter is a catalogue of the ways it is wrong — almost all of it sourced from the authors' own admissions, which is the best sign of a serious paper.

Failure 0: measuring it wrong

Start with the failure that happens before any of the model's own. Every number in this lesson is a balanced accuracy or an R², never a plain accuracy, and there is a hard reason.

Land cover is wildly imbalanced. Take a 10,000-cell region where 500 cells (5%) are cropland. Now compare two models.

Model A never predicts cropland at all. Its confusion matrix: 0 true positives, 500 false negatives, 0 false positives, 9,500 true negatives.

Model B is a real, useful classifier. 400 true positives, 100 false negatives, 800 false positives, 8,700 true negatives.

The useless model scores four points higher. Plain accuracy on an imbalanced problem measures the class prior, not the classifier.

Balanced accuracy fixes it by averaging the per-class recalls, so every class contributes equally regardless of how rare it is:

Model A: ½(0/500 + 9,500/9,500) = ½(0 + 1) = 50.0% — exactly chance for two classes, which is the right verdict. Model B: ½(400/500 + 8,700/9,500) = ½(0.800000 + 0.915789) = 85.79%.

Which is why the paper reports balanced accuracy for every classification and change-detection evaluation, and why it balances the sample counts per class in the first place. Chance for a C-class problem is then always exactly 1/C, which makes the numbers comparable across a 4-class crop mask and a 40-class land-use legend.

Now the trap underneath the trap. Model B's precision is 400/1,200 = 33.3% — two thirds of the cells it calls cropland are not. Its F1 is 0.4706. And if you use its map to estimate area:

An 85.8% balanced accuracy and a 140% area error, from the same map. Because the negative class is nineteen times larger, even a small false-positive rate on it floods the positive class. This is the single most common way a good-looking remote-sensing map produces a wrong policy number, and no representation fixes it — it is a property of the class prior.

Failure 1: clouds, and what survives the fix

Chapter 1's arithmetic: 11 usable optical scenes a year in Borneo against 62 in central Australia. Chapter 2's answer: the consistency objective, teacher and student sharing parameters, sources and frames dropped at 30–50% rates.

It works, partially. The authors' own assessment: "while considerably reduced, tile artifacts are still visible in our embedding fields layers resulting from irregular inputs, and these could be removed in future work with a more aggressive consistency objective term."

Visible tile artefacts mean the embedding field carries a faint imprint of the satellite's acquisition pattern. Practically: if you threshold a similarity map tightly, you may see rectangles. If you compute regional statistics, some of your variance is orbital, not ecological. And the effect is geographically structured — it is worst where clouds are worst, which is the tropics, which is where most of the biodiversity and most of the deforestation are.

Failure 2: the bugs that shipped

The paper's supplementary material contains something rare — a list of what was wrong with the version whose results are published. AEF v2.0 produced the paper's numbers; the released layers are v2.1. Six changes:

Item 2 is the one to internalise. A supervision source with a 0.5 weight — a US-only human-labelled land-cover product — leaked enough of its own idiosyncrasies into the shared representation to degrade an unrelated downstream task in the same region. Nobody caught it internally; it took independent users.

Think about what that means for a general representation. Every training target is a claim about what matters, and those claims compete for 64 dimensions. NLCD's legend distinguishes things NLCD cares about, and pushing the embedding to reproduce it spent capacity that crop-type discrimination needed. The fix was not architectural — it was turning the weight down and adding a competing target.

Item 3 is the humbler kind of bug and the more common one: a date-handling error on one calendar boundary. It produced visible artefacts and it survived to a public release.

Failure 3: version drift, the operational trap

This one is not in the paper as a warning, but it follows directly and it is the single most important practical consequence of the whole design.

Embeddings are only comparable to embeddings from the same model. There is no canonical meaning to dimension 17. When AlphaEarth v2.0 became v2.1, every vector changed. Which means:

And a subtler version of the same trap within one model version: comparing a 2018 embedding to a 2023 embedding assumes both years saw comparable input coverage. Sentinel‑1B failed in December 2021, halving C-band radar availability for a period. A drift computed across that boundary contains a component that is purely about the constellation.

Failure 4: domain shift, and the resolution dial

Chapter 3 established the SatCLIP result: L = 40 wins at interpolation, L = 10 wins at generalising to a held-out continent, 8 task wins to 5. And L = 40 overfits more visibly on the validation curve.

The general statement: a representation's optimal resolution depends on the density of the labels you will use it with, not on how much detail exists in the world. With training points averaging 480 km apart in Africa, a basis that resolves 512 km is fitting between-sample noise.

AlphaEarth's version of this is different but rhymes. It has no coordinates at all, so it cannot overfit to place — but it also cannot use place, which is exactly why the paper reports it needing "~100x observations compared to SatCLIP" on tree genera, "related to AEF receiving no coordinate information, therefore requiring more examples to learn climate gradients." Neither design is safe; they fail in opposite directions.

Failure 5: temporal misalignment

The model supports arbitrary valid periods. The released layers are annual: one vector per cell per calendar year, 2017 through 2024.

A calendar year is a poor fit for most agriculture. A southern-hemisphere growing season straddles New Year, so a single crop cycle is split across two annual embeddings and neither contains a complete one. A double-cropped system runs two full cycles inside one annual vector, which averages them. And a label collected on a specific date — a field visit in March — is being matched against a summary of twelve months, eleven of which the surveyor did not observe.

This is precisely the gap the continuous-time machinery was built to close, and it is closed only for people who can run the model. Everyone using the public layers is stuck with calendar years. It is the clearest place where the product's convenience and the model's capability diverge.

Failure 6: coverage holes

Clay's list is admirably specific and the last two items are the interesting ones. "We do not explicitly include extreme events in the training data" means floods, fires and hurricanes are under-represented in what the model has learned to reconstruct — and those are exactly the events people most want to map. "At most 6 different times per location" bounds how much temporal structure any single place could have taught it.

And none of these models return anything resembling "I do not know." Ask AlphaEarth about a cell that was cloud-covered all year and you get a unit vector, the same shape and the same apparent confidence as any other. The field has no NaN.

Failure 7: dense predictions that were never measured

Chapter 1 flagged GEDI as a sparse sampler. Chapter 2 noted with pleasure that the implicit decoders produce "dense, superresolved LiDAR profiles from GEDI."

Be careful with that pleasure. Those dense profiles are a model's guess at what a LiDAR would have measured, based on optical and radar appearance plus whatever GEDI taught it about similar-looking places. Where the correlation between appearance and structure holds, they are useful. Where it does not — a young dense plantation that looks like mature forest from above but is a fifth the height — they will be confidently wrong, and they will look exactly like the correct ones.

The rule generalises to every decoded source: a reconstructed variable is an inference, not a measurement, and the field does not label which is which.

Failure 8: ill-posedness, and the source ablation

Chapter 1 quoted the decision to keep the input set minimal to "avoid ill-posed reconstruction problems where e.g. climatic information must inform reconstruction of radar data." Some cross-source mappings simply do not exist as functions, and asking a network to learn one produces confident noise.

The source-group ablation shows the more-is-better instinct also has limits. Groups were added cumulatively: Optical, Radar, LiDAR, Environmental, Annotated. The result: 11 of 15 evaluations were most performant with all groups — and therefore four were not. Different tasks prefer different physics, and the paper gives a clean example: Descals oil palm peaks with Optical + Radar + LiDAR, because those carry sub-canopy structure, "where climatic variables and free-form text annotations / land-cover labels are less informative."

Similarly, scaling training observations improves things monotonically for only 9 of 15 evaluations. Some saturate between 100 million and 1 billion observations; some show non-monotonicity the authors could not attribute to any obvious grouping or regional bias.

The pre-flight checklist

Chapter 8: Fields as a Product Shape

Strip the satellites away and something more general is left. AlphaEarth is one instance of a pattern that keeps reappearing wherever a domain has more raw data than labels, and it is worth naming precisely because the next place you see it will not involve orbit.

The pattern

Step 3 is the one people skip, and it is what converts a model into a product. A model is a capability; a materialised field is an asset. Assets can be indexed, cached, versioned, diffed, mirrored, and priced. Capabilities cannot.

Four preconditions

The pattern is not universally applicable. It pays when four things are true, and each one is checkable before you start.

Put numbers on the second one, since it is the economic argument. Producing one annual global layer costs about 162 million forward passes. Suppose one bespoke mapping project would need inference over 10,000 km² — roughly 10,900 tiles at 0.9216 km² each. Then:

Fifteen thousand district-scale projects and the global precomputation has broken even on raw compute alone — before counting the pipeline engineering each of those projects no longer has to write, which was four of the six stages in Chapter 0's table. For a domain with a global research community, that break-even is trivially cleared.

The same pattern, elsewhere

Every row shares the same economics: the encoder runs once per index entry, not once per query, and the query becomes an operation a database can do.

The contract a field product signs

Once a representation becomes an asset that other people build on, it acquires obligations a checkpoint never had. These are the terms, and every one of them was learned the hard way somewhere.

That last row is easy to skip and it is the one that made the crop-classification regression findable. The bug in Chapter 7 was caught by independent tests, which existed because there was a shared benchmark to run them against. A field with no public evaluation suite is a field whose regressions are invisible until somebody's policy report is wrong.

Building one for your own domain

The checklist, in the order the decisions actually bind:

Step 4 is the counterintuitive one and it is the one AlphaEarth got most obviously right. The instinct of every model builder is that more dimensions are strictly better, because they carry more information. That instinct is correct for the encoder and wrong for the product, because the product's users are label-bound. Sixty-four dimensions was not a compression compromise; it was a design target aimed at somebody with two hundred field points and a laptop.

Where the shape is wrong

Four situations where precomputing a field is the wrong call, each the negation of a precondition:

The third row is the deepest, and it is exactly the dual-encoder bargain that shows up in every retrieval system: you give up conditioning the representation on the query, and in exchange the representation becomes cacheable. AlphaEarth's version of that bargain is unusually generous, because the "query" — a valid period — is low-dimensional enough that they could keep a slice of it, via the time-conditioned attention query, without giving up precomputation.

The second-order effect: everyone sharing one coordinate system

There is a consequence of a public precomputed field that has nothing to do with accuracy, and it may end up being the important one.

Today, two teams mapping deforestation in adjacent countries build two pipelines, engineer two feature sets, and produce two maps whose disagreements are impossible to attribute. Was it different labels? Different cloud masking? Different composite windows? A different classifier? Every difference is confounded with every other, so the maps cannot be compared, only stacked.

If both teams start from the same embedding field, all of that collapses. The features are byte-identical. The only things that can differ are the labels and the probe — two small, inspectable, publishable objects. A disagreement becomes diagnosable, and a method becomes reproducible by sending someone 200 labelled points and 390 numbers instead of a repository and a cloud account.

Why this is not just "a foundation model"

The phrase "geospatial foundation model" covers all six systems in Chapter 4's lineage table, and it obscures the distinction that this lesson is about. Foundation model is a claim about training: one large model, self-supervised, transferable. Embedding field is a claim about delivery: the representation exists, materialised, for every entry in an index, before anybody asks.

The last row is the honest cost, and it is why Clay's weights and AlphaEarth's layers are complements rather than rivals. A field is a frozen answer to a question the publisher chose. When your question matches, it is free; when it does not, you need the model. What changed in 2025 is that for a very large fraction of real mapping work, the frozen answer is the right one.

What comes next for this particular field

Three gaps are visible from the paper itself, and each is a reasonable guess at the next release:

Chapter 9: Legacy & Cheat Sheet

Everything worth carrying out of this lesson, in one place.

The numbers

The five ideas

The lineage, in one line each

Where to go next on this site

References

1. Brown, C. F., Kazmierski, M. R., Pasquarella, V. J., Rucklidge, W. J., Samsikova, M., Zhang, C., Shelhamer, E., et al. "AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data," 2025 — arXiv:2507.22291. The paper this lesson is built on; v2 revised 8 September 2025.
2. Klemmer, K., Rolf, E., Robinson, C., Mackey, L., Rußwurm, M. "SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery," 2023 — arXiv:2311.17179. Chapter 3's anchor.
3. Clay Foundation. "Clay Foundation Model v1.5," model release 19 November 2024 — clay-foundation.github.io/model. Apache-2.0 weights and the model card quoted in Chapter 4.
4. Rußwurm, M., Klemmer, K., Rolf, E., Zbinden, R., Tuia, D. "Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networks," 2024. The Siren(SH(·)) location encoder SatCLIP adopts.
5. Sitzmann, V. et al. "Implicit Neural Representations with Periodic Activation Functions" (Siren), 2020 — arXiv:2006.09661.
6. Cong, Y. et al. "SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery," 2022 — arXiv:2207.08051.
7. Jakubik, J. et al. "Foundation Models for Generalist Geospatial Artificial Intelligence" (Prithvi), 2023 — arXiv:2310.18660.
8. Xiong, Z. et al. "Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation" (DOFA), 2024 — arXiv:2403.15356. Wavelength-conditioned dynamic band handling, credited by Clay.
9. Rolf, E. et al. "A generalizable and accessible approach to machine learning with global satellite imagery" (MOSAIKS), Nature Communications, 2021. Random convolutional features — the runner-up that keeps refusing to lose.
10. Radford, A. et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021 — arXiv:2103.00020. The objective SatCLIP and AlphaEarth's text term both use.
11. Oquab, M. et al. "DINOv2: Learning Robust Visual Features without Supervision," 2023 — arXiv:2304.07193. Clay's frozen teacher.
12. Zhu, Z. & Woodcock, C. E. "Continuous change detection and classification of land cover using all available Landsat data" (CCDC), Remote Sensing of Environment, 2014. The designed-feature baseline the deep models had to beat.