Biosignal Foundation Models: EEG, ECG, and PPG Embeddings

Chapter 0: The Label Famine

A hospital network in Toronto has 622,000 electrocardiograms sitting on a server. They span nine years and 211,000 patients across two acute-care hospitals. Every one of them was recorded because a real person walked into an emergency department with something wrong.

Now suppose you want a model that reads an ECG and predicts whether the patient's heart is pumping weakly — specifically, whether their left ventricular ejection fraction (LVEF, the fraction of blood the main pumping chamber squeezes out per beat) has fallen at or below 40%. That is a genuinely useful thing to get from a cheap, ubiquitous test, because the usual way to find out is an echocardiogram, which needs a sonographer, a machine, and an appointment.

How many of those 622,000 ECGs have an LVEF measurement attached? A small fraction. Only the patients who also got an echo, close enough in time for the pairing to mean anything. You have a mountain of signal and a teaspoon of labels.

This is the shape of the problem in every clinical signal domain. Recording is cheap and automatic. Labelling requires a specialist's time, a second test, or a follow-up that may take years. The ratio between "signal we have" and "signal a human has annotated" is not two-to-one or ten-to-one. In EEG it is closer to thousands-to-one, and the annotations that do exist were written for someone else's question.

Why the obvious fix does not work

The obvious fix is transfer learning: pretrain on a big labelled corpus, finetune on your teaspoon. That is what computer vision did with ImageNet. It fails here for a reason worth stating precisely.

There is no ImageNet for biosignals, and there cannot easily be one, because the datasets do not agree on their own shape. Consider what "an EEG recording" means across three plausible public corpora:

Property	Corpus A	Corpus B	Corpus C
Electrodes	19 channels, clinical 10–20 montage	62 channels, research cap	4 channels, consumer headband
Sampling rate	256 Hz	1000 Hz	250 Hz
Segment length	10 s windows	whole 40-minute session	2 s epochs
Task	normal vs abnormal	emotion rating	attention level
Label space	binary	continuous valence and arousal	4 ordinal classes

A conventional model trained on Corpus A has an input tensor of shape (19, 2560) and an output layer with two rows. It cannot ingest Corpus B, whose tensors are (62, 2400000), and its output layer means nothing there. So every dataset gets its own model, its own paper, its own leaderboard. LaBraM's authors put the consequence plainly: models are "typically designed for specific datasets and applications", which limits their scale "and thus diminishing their perceptual capabilities and generalizability."

Small models on small datasets is a stable, self-reinforcing equilibrium. It is also a ceiling.

The escape, in one sentence

Learn from the unlabelled signal itself, then attach the teaspoon of labels to whatever you learned.

That is the entire idea behind a foundation model for biosignals: a network pretrained with self-supervised learning (SSL) — training objectives that manufacture their own supervision out of the data's structure, with no human annotation — producing a fixed-length vector, an embedding, for any input segment. Downstream you fit something tiny on top: a logistic regression, a single linear layer, sometimes a light finetune.

The old loop

One dataset → one architecture → one task → one number. Labels are the bottleneck at every stage.

↓ separate "learn the signal" from "learn the task"

The foundation-model loop

Twenty datasets, no labels → one encoder → embeddings → a linear probe per task. Labels are needed only for the last and cheapest step.

Three papers, all from 2024, did this for the three most-recorded biosignals in the world. They did not coordinate. They landed on recognisably the same recipe with three genuinely different twists, and the differences are where the physiology lives.

Paper	Signal	Unlabelled scale	The distinctive move
LaBraM Jiang, Zhao & Lu, ICLR 2024	EEG — brain	~2,500 hours, ~20 datasets	Learn a discrete vocabulary of 8,192 neural codes by predicting the Fourier spectrum, then do masked-token prediction over it
ECG-FM McKeen et al., 2024/2025	ECG — heart	1.5 million 12-lead ECGs	Hybrid objective: masked contrastive prediction (wav2vec 2.0) plus "adjacent segments belong together" plus random lead dropout
PaPaGei Pillai et al., ICLR 2025	PPG — capillaries	57,641 hours, 20.7M segments	Define positive pairs by waveform morphology, not by patient identity, and regress three hand-derived shape indices

What you should be suspicious of

Before we build any of it, hold on to one uncomfortable question, because it will structure Chapters 6 and 7.

These models are evaluated by freezing the encoder, computing embeddings for a labelled dataset, fitting a linear classifier, and reporting accuracy or AUROC. When those numbers come out high, the natural conclusion is "the embedding captured the physiology." But a second explanation produces identical numbers: the embedding captured the patient.

Every biosignal is a biometric. Your ECG is as individual as your fingerprint — that is not a metaphor, it is the basis of a small industry in ECG-based authentication. So a model that learns to encode "who is this" will score beautifully on any task whose label is constant per person, which describes most clinical labels. If your evaluation split lets the same patient appear in both train and test, you cannot tell the two explanations apart. Hold that thought.

Chapter 1 starts where it should: with what these three signals physically are, because every architecture decision downstream is a response to a physical fact about the sensor.

Why can't we simply build "ImageNet for EEG" and use ordinary supervised transfer learning?

Because EEG signals are too noisy to learn anything from Because EEG datasets are not publicly available Because EEG corpora disagree on their own tensor shape — different electrode counts, sampling rates, and segment lengths — and on their label space, so a model's input and output layers are literally incompatible across datasets. There is no single shape to pretrain on Because EEG has fewer total recorded hours than there are images on the internet

Chapter 1: The Signals From Zero

Three sensors, three physical mechanisms, three completely different signal-to-noise regimes. If you skip this chapter the rest of the paper trio will look like arbitrary hyperparameter choices. It is not arbitrary. Almost every number in these three papers is downstream of a fact about physics.

ECG: the strongest signal you can get for free

Heart muscle cells are electrically excitable. When a wave of depolarisation sweeps across the heart, hundreds of millions of cells flip their membrane potential nearly in unison, and because they are physically aligned their tiny individual currents add instead of cancelling. The body is a salty conductor, so that summed current shows up as a voltage difference you can read at the skin.

How big? About 1–2 millivolts at the peak. That sounds small until you compare it to anything else measurable non-invasively. It is the loudest electrical event in the body.

The wave follows a fixed anatomical itinerary, and the ECG's shape is a direct trace of that itinerary. This is the P-QRS-T complex, and you need it in your head for Chapter 3:

Wave	What is happening	Typical duration	Typical amplitude
P	The atria (upper chambers) depolarise and contract, topping up the ventricles	~80–100 ms	~0.1 mV, gentle bump
PR segment	The signal pauses at the AV node — a deliberate delay so the atria finish before the ventricles start	~50–120 ms	flat
QRS	The ventricles depolarise. The biggest muscle mass, activated fastest	~80–100 ms	1–2 mV, sharp spike
ST segment	Ventricles fully depolarised, plateau phase — this is where ischaemia shows up	~80–120 ms	flat, at baseline
T	The ventricles repolarise, resetting for the next beat	~160 ms	~0.1–0.5 mV, broad bump

Twelve "leads" does not mean twelve wires. It means ten electrodes combined arithmetically into twelve different viewing angles on the same three-dimensional electrical dipole. Lead II looks along the heart's main axis from upper-right to lower-left; V1 sits over the right ventricle; V6 over the left lateral wall. Each lead sees the same event projected onto a different direction, which is why an infarct in the left anterior wall shows up dramatically in V2–V4 and barely at all in lead III.

Concept to realization. A 12-lead ECG tensor is (12, T), and the 12 rows are not independent channels the way RGB pixels are. Leads I, II, and III satisfy Einthoven's triangle exactly: III = II − I. The augmented leads aVR, aVL and aVF are further linear combinations. So of the twelve rows only about eight carry independent information. That redundancy is not a nuisance — ECG-FM turns it into a training decision in Chapter 3.

EEG: the same idea, a hundred times weaker, through bone

Neurons are electrically active too, but three things conspire against you.

First, amplitude. Scalp EEG runs 10–100 microvolts — roughly a hundredth of an ECG. LaBraM's preprocessing states the working range as ±0.1 mV and rescales so the numbers land near ±1.

Second, the skull. Bone is a poor conductor and acts as a spatial low-pass filter, smearing each cortical source over several centimetres of scalp. Every electrode sees a blurred mixture of many sources. This is why EEG has superb time resolution (sub-millisecond) and terrible spatial resolution (centimetres).

Third, geometry. Cortical neurons only sum coherently when they are aligned. The pyramidal cells in cortical gyri are aligned, so they dominate; sources in sulcal walls or deep structures largely cancel. You are not measuring "brain activity", you are measuring the projection of a particular subset of it onto a particular sensor geometry.

The result is a signal with a genuinely low signal-to-noise ratio, non-stationary statistics (the same electrode's spectrum changes minute to minute), and nonlinear dynamics. LaBraM's authors report a concrete consequence we return to in Chapter 2: when they tried to train their tokenizer by reconstructing raw EEG, the loss failed to converge. You cannot ask a model to predict noise.

The structure people do find in EEG lives in the frequency domain: delta (0.5–4 Hz, deep sleep), theta (4–8 Hz, drowsiness), alpha (8–13 Hz, the relaxed eyes-closed occipital rhythm), beta (13–30 Hz, active thinking), gamma (above 30 Hz, contested). Remember this — it is why LaBraM's tokenizer predicts a spectrum rather than a waveform.

PPG: not electrical at all

PPG (photoplethysmography) is the green light on the back of your watch. An LED shines into your skin and a photodiode measures how much comes back. Haemoglobin absorbs that wavelength, so when a pressure pulse pushes a bolus of blood into the capillary bed under the sensor, more light is absorbed and the returned intensity dips.

A PPG waveform is therefore a volume trace, not a voltage trace. Its shape encodes the mechanical behaviour of your arteries, which is what makes it interesting and fragile at once.

Feature of a PPG beat	What it means physically
Systolic peak	The forward pressure wave from the heart's ejection arriving at the finger or wrist
Dicrotic notch	The aortic valve slamming shut, plus the wave reflected back from the arterial periphery. Its depth and timing depend on how stiff the vessels are
Diastolic (second) peak	The reflected wave itself. In young elastic arteries it arrives late and distinct; in stiff arteries it arrives early and merges into the systolic peak
Slow baseline drift	Respiration, venous pooling, sensor pressure, motion

PPG's failure modes are entirely different from ECG's. Motion moves the sensor relative to the skin and swamps the pulsatile component. Ambient light leaks in. Cold fingers vasoconstrict and the pulse amplitude drops toward the noise floor. And melanin absorbs green light, so darker skin returns less signal to begin with — PaPaGei's authors treat this as a first-class evaluation axis rather than a footnote, and we look at their result in Chapter 4.

Signal anatomy explorer

Switch signals to see the same four-second window through each sensor, with the landmarks labelled. Then turn on the artifacts every real recording contains. Notice how differently each one degrades: ECG survives baseline wander with its landmarks intact, EEG is genuinely swallowed by a single eye blink, and PPG's beat shape — the exact thing PaPaGei's objective depends on — is destroyed by wrist motion.

Signal:

Worked example: what the tensor actually looks like

Let us turn all of that into shapes, because the shapes are what the three papers argue about.

ECG-FM's input. The authors resample every recording to 500 Hz and cut non-overlapping 5-second segments. One segment is therefore:

12 leads × (5 s × 500 Hz) = 12 × 2500 = 30,000 floats

That tensor goes into a convolutional feature extractor of four blocks, each a convolution with kernel length 2 and stride 2. Track the length through the stack by hand. With kernel k and stride s, an input of length L becomes ⌊(L − k)/s⌋ + 1:

2500 → 1250 → 625 → 312 → 156

Check the third step, because it is the one that rounds: ⌊(625 − 2)/2⌋ + 1 = ⌊311.5⌋ + 1 = 311 + 1 = 312. Then ⌊(312 − 2)/2⌋ + 1 = 155 + 1 = 156.

So a 5-second, 12-lead ECG becomes 156 tokens, each of dimension 768 once the transformer has run. The total stride is 2⁴ = 16 samples, and at 500 Hz that is:

16 samples ÷ 500 Hz = 0.032 s = 32 milliseconds per token

Now the number means something. A QRS complex lasts 80–100 ms. At 32 ms per token, the entire QRS occupies about three tokens. The model's "words" are finer than the heart's most important event, which is exactly what you want: the tokenization is not throwing away diagnostic morphology. Hold this — in Chapter 3 we compute how many tokens a single mask span swallows, and the answer is what makes the pretext task hard.

LaBraM's input. Different problem, different answer. EEG's channel count varies by dataset, so LaBraM refuses to treat "the montage" as a fixed input dimension. It resamples to 200 Hz and cuts each channel independently into 1-second patches (window w = 200 samples). The number of patches for C channels over a recording of t samples is:

N = C × ⌊t / w⌋

The transformer's sequence length is capped at 256 patches for memory. That single cap generates the paper's odd-looking data recipe:

Montage	Patches per second	Seconds that fit in 256 patches
64 channels	64	256 ÷ 64 = 4 s
32 channels	32	256 ÷ 32 = 8 s
19 channels (clinical 10–20)	19	⌊256 ÷ 19⌋ = 13 s

This is the crux of LaBraM's cross-dataset trick. A patch is one channel for one second — a unit that exists in every EEG dataset ever recorded, regardless of montage. Channel identity is not baked into the tensor shape; it is supplied separately as a learned spatial embedding, exactly the way a transformer supplies position. Add a channel, add patches. Remove a channel, remove patches. No weight matrix changes size.

PaPaGei's input. Single channel, so the shape question is trivial and the interesting choices are elsewhere: a 4th-order Chebyshev bandpass from 0.5 to 12 Hz, 10-second windows, drop any window that is more than 25% flatline, z-score, resample to 125 Hz. One segment:

1 × (10 s × 125 Hz) = 1250 floats → encoder → a 512-dimensional embedding

Why 125 Hz, when one of their three source datasets records at 500 Hz? Because MIMIC-III's waveforms are at 125 Hz, and they downsample everything to the lowest common rate. You cannot invent bandwidth a corpus never had, and a model trained on a mixture of rates learns the rate as a dataset fingerprint. Pinning the rate removes that shortcut. Why 0.5–12 Hz? Below 0.5 Hz is baseline drift and respiration; above 12 Hz there is essentially no pulsatile content left, only noise. The filter is a statement about where the physiology lives.

The pattern to notice. All three papers spend their preprocessing budget on removing degrees of freedom that describe the recording rather than the body — sampling rate, amplitude units, channel count, mains hum. Every one of those is a feature a model would happily learn, and every one of them is a feature that will not transfer to the next hospital.

LaBraM's version of the same instinct: bandpass 0.1–75 Hz, then a 50 Hz notch filter. Fifty hertz is European and Asian mains frequency. It is the single loudest periodic component in most raw EEG and it carries exactly zero information about the brain — but it does carry information about which continent the recording was made on. Notch it out and a model pretrained in Shanghai has one less irrelevant thing to memorise.

LaBraM caps its transformer sequence at 256 patches, where a patch is one channel for one second. Why does this let it train on datasets with different electrode counts, when a conventional EEG CNN cannot?

Because 256 is larger than any real montage, so every dataset fits Because channel count is moved out of the tensor shape and into the sequence length. Channel identity is supplied as a learned spatial embedding, the way position is. A 62-channel recording is just a different-length sequence of the same one-channel-one-second unit, and no weight matrix changes size Because all EEG datasets are resampled to 200 Hz, which makes their channel counts equal Because the model averages across channels before the transformer sees them

Chapter 2: Tokenizing Biology

Masked modeling is the workhorse of self-supervised learning: hide part of the input, predict what was hidden, and to succeed the model must internalise how the parts relate. BERT hides words. It works because a word is already a token — a discrete symbol from a fixed vocabulary — so "predict the hidden thing" is a classification problem with a clean cross-entropy loss.

EEG has no vocabulary. A one-second patch is 200 floating-point numbers. So what exactly do you ask the model to predict?

The obvious answer, and why it fails

Predict the numbers. Mask the patch, have the model output 200 values, minimise mean squared error against the truth. This is masked autoencoding, and in images it works beautifully.

LaBraM's authors tried it. They report, without hedging, that "the loss fails to converge while directly reconstructing raw EEG signals."

Here is why, and it is worth deriving because the same trap sits under every attempt to do generative SSL on a noisy signal.

Split any patch into two parts: a component that is predictable from the surrounding context, and a component that is not — thermal noise in the amplifier, unmodelled muscle activity, the stochastic part of cortical dynamics. Call them s and n. The MSE-optimal prediction of s + n given context is E[s + n | context] = s, and the residual loss floor is the variance of n. When EEG's SNR is low, that floor is most of the total variance, so nearly all of the loss you are minimising is irreducible. The gradient is dominated by noise the model can never fit.

Now make it concrete with the sharpest failure. Take a patch that is a clean 10 Hz alpha oscillation, x(t) = A sin(ωt). Suppose the model gets the frequency right, gets the amplitude exactly right, and is off by a quarter cycle in phase: x̂(t) = A sin(ωt + φ). What does MSE say about this excellent prediction?

E[(x − x̂)²] = A²(1 − cos φ)

Work it out with the standard expectations over a full cycle: E[sin²] = ½ for both terms, and the cross term is 2 · ½ cos φ. So the total is ½ + ½ − cos φ, times A².

At φ = 90°, cos φ = 0, so the error is A². Now compare against the laziest possible prediction, a flat line at zero, whose error is just the signal's own power:

E[x²] = A²/2

Predicting the right oscillation at the wrong phase scores twice as badly as predicting nothing at all. A² versus A²/2. And EEG phase, one second into the future, across a skull, is very close to unpredictable. So the loss landscape actively pushes the model toward the flat-line solution, which is exactly the non-convergence the authors observed. The model is not failing to learn. It is correctly optimising a badly posed target.

The fix: predict the spectrum, and quantize it

LaBraM's answer has two moves, and they solve two different halves of the problem.

Move one: change the target from waveform to spectrum. Apply the Discrete Fourier Transform to each patch. The DFT re-expresses 200 time samples as a set of complex coefficients, one per frequency, each of which can be written in polar form as an amplitude and a phase:

A^m = √( Re(X^m)² + Im(X^m)² ), φ^m = arctan( Im(X^m) / Re(X^m) )

The tokenizer's decoder gets two separate heads, one regressing amplitude and one regressing phase, each with its own MSE term. This is not a lossy transform — amplitude and phase together contain exactly the same information as the waveform. What changes is the parametrisation of the loss, and that is the whole point. Go back to the alpha-oscillation example: with separate heads, a model that nails the 10 Hz amplitude and misses the phase gets full credit on the amplitude term and takes its beating only on the phase term. In the time domain, the phase error contaminated the amplitude credit and the two were inseparable.

And this maps onto the physiology. "How much 10 Hz power is in this patch" is a stable, meaningful, learnable quantity — it is alpha rhythm, it tracks arousal, it differs between electrodes and states. "Where in its cycle is that oscillation right now" is mostly a coin flip. Splitting the target lets the model learn the first without being punished into silence by the second.

Move two: make the prediction target discrete. This is where the neural codebook comes in, and it is the idea worth slowing down for.

Vector quantization, derived

A codebook is a matrix V of K learned vectors, each of dimension D. LaBraM uses K = 8192 and D = 64. Given a patch representation p from the tokenizer's encoder, quantization means: find the nearest codebook entry and replace p with its index.

z = argmin_j ‖ ℓ₂(p) − ℓ₂(v_j) ‖₂

Notice the ℓ₂ on both sides: both vectors are normalised to unit length before the distance is taken. The paper flags this as deliberate — it makes the search equivalent to finding the highest cosine similarity, and it "improves the codebook utilization."

That claim deserves a worked example, because it is not obvious why normalisation should matter to a nearest-neighbour lookup.

Worked example: why normalisation changes the answer

Work in three dimensions so you can check every number by hand. Let the patch representation be

p = [3, 4, 0], ‖p‖ = √(9+16) = 5, ℓ₂(p) = [0.6, 0.8, 0]

and let there be three candidate codes:

Code	Vector	Norm	Raw ‖p − v‖²	Cosine with p	Normalised distance² = 2 − 2cos
A	[9, 12, 0]	15	6²+8² = 100	1.000	0.000
B	[4, 3, 0]	5	1²+1² = 2	0.960	0.080
C	[0, 0, 5]	5	3²+4²+5² = 50	0.000	2.000

Check code A's cosine: p · v_A = 3(9) + 4(12) = 27 + 48 = 75, and ‖p‖‖v_A‖ = 5 × 15 = 75, so the cosine is exactly 1. Code A points in precisely the same direction as the patch; it is the same shape, three times louder.

Check code B: p · v_B = 12 + 12 = 24, over 5 × 5 = 25, giving 0.96.

Now read the two rankings against each other:

Without normalisation

B wins (distance 2), A loses badly (distance 100). The code that happened to have the right magnitude beats the code with the right shape.

↓ project both onto the unit sphere first

With normalisation

A wins with a perfect match. Magnitude is no longer part of the competition; only direction is.

This is why unnormalised codebooks collapse. In raw distance, a code's chance of ever being selected depends on its length. Codes that happen to sit near the typical magnitude of the data hoover up assignments; codes that are long or short are almost never nearest to anything, get no gradient, drift nowhere, and stay dead forever. With 8,192 entries, most of the codebook can quietly die. Normalising removes magnitude from the competition entirely, so all 8,192 codes compete purely on waveform shape — which is the axis you actually wanted them to specialise along.

Worked example: measuring codebook collapse

"Utilization" needs a number. The standard one is perplexity: exponentiate the entropy of the code-usage distribution, which gives you the effective number of codes in use.

Take a toy codebook of four entries and 100 quantized patches, with usage counts 70, 20, 7, 3. The probabilities are 0.70, 0.20, 0.07, 0.03. Compute the entropy in nats, term by term:

Code	p	ln p	−p ln p
1	0.70	−0.3567	0.2497
2	0.20	−1.6094	0.3219
3	0.07	−2.6593	0.1862
4	0.03	−3.5066	0.1052
Entropy H			0.8629 nats

perplexity = e^H = e^0.8629 = 2.37 effective codes out of 4

Forty percent of the nominal vocabulary is dead weight. Scale that ratio to 8,192 and you can see why the ℓ₂ detail earns its sentence in the paper. A collapsed codebook makes the downstream pretext task trivially easy — if only 20 codes are ever used, predicting the masked code is a 20-way problem, and the transformer learns almost nothing.

Patch → code: the quantizer, live

Left: patch representations (dots) and codebook entries (rings) in a two-dimensional stand-in for the 64-dimensional space, with each patch tied to the code it is assigned. Right: the resulting usage histogram and perplexity. Toggle normalisation and watch assignment flip from "nearest in magnitude" to "nearest in direction" — and watch the dead codes come back to life.

The gradient problem, and the two commitment terms

There is one more obstacle. argmin has zero derivative almost everywhere. If quantization sits in the middle of your network, no gradient reaches the encoder.

The VQ-VAE solution, which LaBraM inherits, is a straight-through estimator plus two auxiliary terms that use a stop-gradient operator sg(·) — an identity function forward, a zero function backward. The tokenizer's full objective is:

ℒ_T = ∑ ‖o^A − A‖² + ‖o^φ − φ‖² + ‖sg(ℓ₂(p)) − ℓ₂(v_z)‖² + ‖ℓ₂(p) − sg(ℓ₂(v_z))‖²

Read it as four jobs:

Term	Who it trains	What it says
‖o^A − A‖²	encoder + decoder	"The code must retain enough information to reconstruct the amplitude spectrum"
‖o^φ − φ‖²	encoder + decoder	"...and the phase spectrum, scored separately"
‖sg(p) − v_z‖²	the codebook only	"Codes, move toward the patches assigned to you" — this is online k-means
‖p − sg(v_z)‖²	the encoder only	"Encoder, do not wander away from the code you were assigned" — the commitment term that keeps the two from chasing each other

The two stop-gradients split one symmetric-looking distance into two one-way forces. Without the split, encoder and codebook can co-drift arbitrarily and the quantization boundaries never settle.

How much does the code throw away?

A patch is 200 float32 numbers, so 200 × 32 = 6,400 bits. A code index into an 8,192-entry codebook is log₂(8192) = 13 bits.

6400 / 13 ≈ 492× compression

Nearly five hundred to one. That is aggressive, and it is the point: the tokenizer is a lossy bottleneck by design, and what it is designed to lose is precisely the unpredictable component we showed above was poisoning the reconstruction loss. What survives is a coarse description of the patch's spectral shape — roughly, "quiet alpha-dominant patch", "high-amplitude slow wave", "muscle-contaminated segment". That is the vocabulary the big model gets to reason over.

Stage two: masked EEG modeling

With a vocabulary in hand, LaBraM's pretraining becomes an exact analogue of BERT.

1 · patch

Cut every channel into 1-second patches → N patches, N ≤ 256

↓

2 · tokenize (frozen)

Each patch → one of 8,192 code indices. These are the labels. No human wrote them

↓

3 · mask

Replace 50% of patch embeddings with a learned mask token e_M; add temporal + spatial embeddings

↓

4 · predict

Transformer → linear head → softmax over 8,192 → cross-entropy against the true codes

↓ and then again with the mask inverted

5 · symmetric masking

Run the same batch with mask 1 − M. Two views per tokenizer pass. ℒ = ℒ_M + ℒ_M̃

Symmetric masking is a small idea with two justifications the authors state directly: the tokenizer forward pass is an extra cost per sample, so reusing its output for a second masking pattern doubles what you get per unit of compute; and the two complementary views give the batch more diversity. Their ablation lands on 0.5 as the best or near-best ratio, and notes it also gives the smallest variance across seeds — which matters more than a decimal point of mean when your downstream datasets are this small.

Did the vocabulary earn its complexity?

The honest test is the ablation, and the result is more interesting than a clean win. The authors compare LaBraM against a variant that skips the tokenizer and just reconstructs raw patches, and another that reconstructs the Fourier spectrum directly.

Benchmark	What the task is	Effect of the neural tokenizer
TUAB	Binary: is this 10-second clinical EEG normal or abnormal? 409,455 samples	Naive reconstruction is slightly better
TUEV	Six-way event type: spike-and-sharp-wave, GPED, PLED, eye movement, artifact, background. 112,491 samples	The tokenizer wins decisively

The authors' explanation is the right one: TUAB is a low-level task where normal and abnormal segments differ visibly in raw waveform statistics, so a model that learned to reproduce waveforms has everything it needs. TUEV asks the model to name an event category — a semantic judgement — and semantics is exactly what a learned discrete vocabulary supplies. The codes are not waveform templates; they are cluster identities over spectral shapes, and a cluster identity is one step closer to a name.

The headline numbers, all from LaBraM-Base at 5.8M parameters, against BIOT (3.2M), the strongest prior model:

Model	TUAB balanced acc.	TUAB AUROC	TUEV balanced acc.	TUEV Cohen's κ
BIOT	0.7959	0.8815	0.5281	0.5273
LaBraM-Base (5.8M)	0.8140	0.9022	0.6409	0.6637
LaBraM-Large (46M)	0.8226	0.9127	0.6581	0.6622
LaBraM-Huge (369M)	0.8258	0.9162	0.6616	0.6745

Two things to read here. TUAB moves by about two points from BIOT to Base and then by less than two more points across a 64× increase in parameters — a task near its ceiling. TUEV moves by eleven points from BIOT to Base, and then barely at all with scale. The pretraining objective bought the jump; the parameters did not.

The data-scaling result is bluntest of all. LaBraM-Base pretrained on 500 hours beats LaBraM-Base pretrained on all 2,500 hours on TUAB. Only the Huge model shows a clean monotone appetite for more data. At 5.8M parameters there is simply not enough capacity to convert five times the signal into anything, and the extra data mostly changes the mixture of recording conditions the model has to reconcile.

LaBraM's tokenizer predicts the Fourier amplitude and phase with two separate heads rather than reconstructing the raw waveform. Since amplitude+phase contains exactly the same information as the waveform, why does this change anything?

It compresses the signal, so there is less to predict It removes the high-frequency noise, since the DFT is a low-pass filter It makes the task differentiable, which raw waveform reconstruction is not It changes the loss parametrisation, not the information. In the time domain a correct-amplitude, wrong-phase prediction scores worse than predicting zero (A² vs A²/2), so the optimiser is pushed to the flat-line solution. With separate heads, the model banks full credit for the learnable amplitude structure and takes its loss only on the unlearnable phase

Chapter 3: Inpainting a Heartbeat

ECG-FM faces the opposite situation from LaBraM. The signal is loud, stereotyped, and periodic. A cardiologist can read it. So the interesting question is not "how do we get a learnable target out of noise" but "how do we make the pretext task hard enough to teach anything".

Because here is the trap: if you mask 30 milliseconds of an ECG, the model can fill it in by linear interpolation. It has learned that ECGs are smooth. That is not cardiology.

The architecture, in shapes

ECG-FM is wav2vec 2.0 — the speech model — applied to 12-lead ECG, at 90.9 million parameters. Two stages:

Stage	Configuration	Tensor
Input	5 s, 500 Hz, 12 leads, z-scored	(12, 2500)
CNN feature extractor	4 blocks: conv(256 ch, kernel 2, stride 2) + LayerNorm + GELU	(256, 156)
Positional conv	128 filters, 16 groups, added to the latents (relative positions, not absolute)	(256, 156)
Transformer encoder	BERT-Base config: 12 layers, dim 768, 12 heads, FFN 3072	(768, 156)
Output	156 contextual vectors c_t, pooled for downstream use	(768, 156)

We computed the 156 in Chapter 1: four stride-2 layers is a total stride of 16, and 16 samples at 500 Hz is 32 ms per token.

Masking: derive the 49%

The masking rule sounds arbitrary until you compute what it does. Each of the 156 token positions is independently selected as a span start with probability 0.065, and each selected start masks the following 10 tokens. Spans overlap freely.

The naive expectation is 0.065 × 10 = 65% masked. The paper reports approximately 49%. The gap is overlap, and the correct calculation is easier through the complement.

A token is unmasked only if none of the 10 start positions that could cover it was selected. Each of those 10 independent draws misses with probability 1 − 0.065 = 0.935, so:

P(unmasked) = 0.935¹⁰

Compute it by repeated squaring so you can check every digit:

Power	Value
0.935²	0.874225
0.935⁴ = (0.935²)²	0.764269
0.935⁸ = (0.935⁴)²	0.584107
0.935¹⁰ = 0.935⁸ × 0.935²	0.510630

P(masked) = 1 − 0.5106 = 0.489 → 48.9% ≈ the paper's "approximately 49%"

Our derivation reproduces the paper's number to the tenth of a percent, which is a good sign we have read the rule correctly.

What a 10-token span actually destroys

Now combine two numbers we computed independently: 32 ms per token, and spans of 10 tokens.

10 tokens × 32 ms = 320 milliseconds

At 75 beats per minute a cardiac cycle is 800 ms. So a single mask span removes forty percent of a heartbeat — comfortably enough to swallow the entire QRS complex plus the ST segment plus most of the T wave.

This is what makes the task cardiology instead of interpolation. Confronted with a gap that starts after the P wave and ends in the T wave, a model has no local smoothness to lean on. To fill it, it must know: that a P wave is followed by a fixed conduction delay; that the delay ends in a sharp, narrow, high-amplitude ventricular deflection; that the deflection's polarity depends on which lead you are looking at; that repolarisation follows with a characteristic broad shape; and that the whole package repeats at a rate visible in the surrounding beats. Every one of those is a fact about the conduction system. The pretext task is a forced march through the physiology.

Masked ECG inpainting — what the model must infer

Every token is a 32 ms slice. Drag the span-start density, then look at what is hidden. At the paper's setting a typical span erases a whole ventricular complex — the model must reconstruct it from rhythm and the surrounding beats' morphology, not from local continuity. Switch to short spans and watch the task collapse into "draw a smooth line".

Span length:

The target: quantize, so the task cannot be cheated

Once a span is masked, what is the model predicting? Not the raw latents z_t. Following wav2vec 2.0, ECG-FM quantizes them first, with a specific rationale the authors state: quantization "help[s] remove detailed artifacts which would otherwise make the task easier, thereby hurting generalizability."

Read that carefully, because it is the same insight as LaBraM's Chapter 2, arrived at from the other side. LaBraM discretised because the raw target was too noisy to learn. ECG-FM discretises because the raw target is too easy to match on irrelevant detail. A specific electrode's contact noise or a specific machine's filter ringing is perfectly predictable within a recording, so an un-quantized objective rewards the model for memorising the recording's idiosyncratic fingerprint. Quantization coarsens the target until only the shared, physiological content survives as something worth predicting.

The quantizer uses two codebooks of 320 entries each, and the code for a latent is the concatenation of one entry from each — product quantization. The effective vocabulary is:

320 × 320 = 102,400 distinct codes, from only 640 learned vectors

That is the point of the product structure: 102,400 codes stored as 640 vectors instead of 102,400 vectors. A codebook diversity loss pushes usage toward uniform across entries — the same anti-collapse concern we quantified with perplexity in Chapter 2, addressed with an explicit penalty rather than with normalisation.

The loss: contrastive, not cross-entropy

Here LaBraM and ECG-FM genuinely diverge. LaBraM classifies over its 8,192 codes with cross-entropy. ECG-FM instead uses a contrastive objective: for each masked position, pull the contextual representation c_t toward its own quantized target q_t and push it away from a set of distractors sampled from other masked positions in the same utterance.

ℒ = −log [ exp(cos(c_t, q_t)/τ) / ∑_{q̃ ∈ Q_t} exp(cos(c_t, q̃)/τ) ]

Worked example: what the temperature does

Take one masked position with four candidates — the true target and three distractors — with these cosine similarities to c_t:

cos(c, q_true) = 0.62, distractors: 0.51, 0.30, 0.11

With τ = 0.1, divide every similarity by 0.1 and exponentiate:

Candidate	cos	cos/τ	exp(·)
true	0.62	6.2	492.75
distractor 1	0.51	5.1	164.02
distractor 2	0.30	3.0	20.09
distractor 3	0.11	1.1	3.00
sum			679.86

p(true) = 492.75 / 679.86 = 0.725 → loss = −ln(0.725) = 0.322

Now repeat with τ = 1.0. The exponentials become e^0.62 = 1.859, e^0.51 = 1.665, e^0.30 = 1.350, e^0.11 = 1.116, summing to 5.990:

p(true) = 1.859 / 5.990 = 0.310 → loss = 1.170

Same embeddings, same ranking, loss 3.6× larger. And look at where the loss lives: at τ = 0.1 the denominator is 72% true target and 24% the single hardest distractor, with the two easy distractors contributing 3% between them. At τ = 1.0 all four candidates contribute comparably. Low temperature turns the sum into a competition with the hardest negative; high temperature averages over all of them. That is the entire mechanism, and it is why temperature is the hyperparameter contrastive methods are most sensitive to.

CMSC: the objective that needs no augmentation

Masked prediction teaches local structure. It does not, on its own, teach the model that two different five-second windows of the same person's heart should land near each other. For that you want a contrastive objective at the segment level — and the usual way to build one is augmentation: take a segment, distort it twice, declare the two distortions a positive pair.

Augmentation is dangerous here in a specific way the ECG literature calls faulty alignment. Consider the standard time-series augmentations and what each one destroys:

Augmentation	What it changes	Why that is a diagnosis
Amplitude scaling	Voltage magnitudes	Left ventricular hypertrophy is defined by voltage thresholds. Scaling turns a positive into a negative
Time warping	Interval durations	A prolonged QT interval is a drug-toxicity and arrhythmia-risk marker measured in milliseconds
Lead permutation	Which view is which	Lead misplacement is itself an artifact category the model is asked to detect
Adding noise	Signal quality	"Poor data quality" is one of the labels in ECG-FM's own downstream task

CMSC (Contrastive Multi-Segment Coding, from the CLOCS family) sidesteps all of it. A standard ECG recording is 10 seconds; ECG-FM cuts it into two non-overlapping 5-second segments. Those two halves are declared a positive pair. No distortion is applied at all.

The physiological assumption is explicit and checkable: cardiac function is stable over a few seconds. Whatever is true of the heart in seconds 0–5 is almost certainly true in seconds 5–10. The supervision is free because the recording protocol already provided it.

Notice the design pattern. All three papers find their positive pairs in a structure that already exists in the data rather than in a distortion they invent. ECG-FM uses temporal adjacency within a recording. PaPaGei, as we will see in Chapter 4, uses shared waveform morphology across strangers. Invented augmentations require you to know which transformations preserve the label — and in medicine you frequently do not.

RLM: random lead masking, and a shortcut it closes

The third ingredient is Random Lead Masking: during pretraining, leads are stochastically zeroed out. The authors' stated purpose is deployment flexibility — "by exposing the model to diverse lead combinations during pretraining, their model can be finetuned using arbitrary reduced lead sets of the standard 12-lead ECG." That matters practically, because an ambulance monitor or a smartwatch gives you one to three leads, not twelve.

There is a second consequence worth working out, which follows from Chapter 1's note that the twelve leads are not independent. The limb leads are defined from three electrode potentials:

I = LA − RA, II = LL − RA, III = LL − LA

Subtract the first from the second: II − I = (LL − RA) − (LA − RA) = LL − LA = III. That is Einthoven's triangle, and it is an exact algebraic identity, not an approximation.

So if a mask hides a stretch of lead III while leaving I and II visible at the same timestamps, a sufficiently clever model can reconstruct the hidden stretch with zero error by subtraction — and learn nothing whatsoever about hearts. RLM removes entire leads, so a lead is sometimes missing along with the leads that would determine it, and the shortcut is unavailable. (The paper motivates RLM by deployment flexibility; this shortcut-closing reading is ours, but it follows directly from the linear dependence they themselves note in the 12-lead system.)

The data decision that most papers get wrong

ECG-FM's corpus is 1.5 million 12-lead ECGs. It comes from three places, and how the authors used them is the most quietly rigorous thing in the paper:

Source	What it is	Used for pretraining?	Used downstream?
PhysioNet 2021 (CPSC, CPSC-Extra, PTB-XL, Georgia, Ningbo, Chapman)	Public, multi-country	Yes	No
MIMIC-IV-ECG	Public, Beth Israel Deaconess, 10 s recordings	Yes	Yes (released benchmark)
UHN-ECG — 622k ECGs, 211k patients, 2010–2018, Toronto General + Toronto Western	Private, institutional, every ECG cardiologist-over-read	No	Yes (interpretation + LVEF)

Their reasoning, in their own words: "We did not pretrain on the UHN-ECG dataset so that we may release our pretrained model while respecting patient privacy, as well as to explore the cross-dataset generalizability of ECG-FM."

Two goals, one decision. Weights that never saw the private data cannot memorise it, so the model can be open-sourced. And a downstream benchmark drawn entirely from hospitals the pretraining never touched is a real out-of-distribution test rather than a rehearsal. Chapter 6 is about why that second point is not optional.

One more detail that tells you this is a clinical dataset and not a benchmark: 12.8% of UHN-ECGs are annotated as poor data quality — artifacts, muscle interference, lead misplacement and reversal. The authors kept every one of them, and made "poor data quality" a prediction target, because "interpretation may be attempted regardless" in the real emergency department. A model trained on the clean 87% would meet the other 13% for the first time in production.

ECG-FM masks spans of 10 tokens rather than single tokens. Given that each token is 32 ms, what does that choice accomplish?

It reduces compute by making fewer prediction targets It matches the sampling rate of the original recordings A 320 ms gap is about 40% of a cardiac cycle — enough to erase an entire QRS-ST-T. Local interpolation cannot fill it, so the model is forced to learn conduction structure: that a P wave implies a delayed ventricular complex of a particular shape and polarity in that lead It guarantees exactly 49% of tokens are masked

Chapter 4: PPG at Population Scale

PaPaGei is trained on 57,641 hours of photoplethysmography — 20,751,206 ten-second segments from 13,517 people. All of it public. That is the largest open PPG corpus assembled, and it comes from three places that could hardly be less alike:

Dataset	Where the sensor was	Native rate	Participants	Segments	Hours
VitalDB	Finger, during surgery	500 Hz	5,866	6,248,100	17,355
MIMIC-III waveform	Fingertip, ICU monitor	125 Hz	5,596	7,196,401	19,990
MESA sleep study	Fingertip, polysomnography	256 Hz	2,055	7,306,705	20,296
Total			13,517	20,751,206	57,641

Anaesthetised surgical patients, critically ill ICU patients, and sleeping adults in a cardiovascular cohort study. Three physiological states, three monitors, three sampling rates. The whole design problem of this paper is: what training objective makes an encoder that generalises across that?

The obvious objective, and its specific poison

The default answer in 2024 was subject-level contrastive learning: take two segments from the same person, call them a positive pair, push them together; push different people apart. Apple's work on Apple Watch PPG used exactly this, and PaPaGei implements it as their own baseline, PaPaGei-P, with the standard NT-Xent loss.

It works. It also has a defect that is easiest to see by asking what the objective literally optimises.

"Same person → near" is the definition of a biometric. If you train an encoder so that any two segments from person A are close and any segment from person B is far, you have not built a physiology encoder. You have built a face-recognition system for capillaries. Any downstream health signal it carries is a side effect of the fact that health correlates with identity. Chapter 7 is about what that means for deployment; here, notice what it costs you in accuracy: two people with identical arterial stiffness are pushed apart, because the objective's only notion of similarity is who you are.

There is a second, subtler cost, and it is about augmentations. Contrastive learning needs distorted views, and the standard time-series menu — cropping, negation, time flipping, magnitude scaling, Gaussian noise — interacts badly with PPG. PaPaGei-P uses cropping (p = 0.50), negation (0.20), flipping (0.20), and scaling (0.40). Consider what flipping a PPG segment in time does: it puts the dicrotic notch before the systolic peak. The result is not a PPG signal at all; it is a waveform no artery has ever produced. Asking the encoder to map it to the same point as the original is asking it to be invariant to the exact feature that carries the diagnosis.

The move: define similarity by shape, not by person

PaPaGei-S replaces "same subject" with "same waveform morphology". Two segments from two strangers who happen to have similar vascular behaviour become a positive pair. Two segments from the same person at different moments — before and after exercise, say — may become negatives.

To do that you need morphology as a number. The authors compute three, all classical and all cheap, from the signal-processing literature rather than learned:

Index	What it measures	Physiological reading	Role in the loss
sVRI stress-induced Vascular Response Index	Ratio of mean signal after the systolic peak to mean signal before it	Amplitude behaviour — how the pulse decays, which tracks peripheral resistance	Discretised into 8 bins; defines the positive pairs
IPA Inflection Point Area ratio	Ratio of the area under the systolic portion to the area under the diastolic portion, split at the dicrotic notch	Signal width — how much of the beat is forward wave versus reflected wave	Regressed by a mixture-of-experts head, MAE loss
SQI Signal Quality Index	Skewness of the samples, averaged over 5-second windows	Whether the waveform has a real pulsatile shape at all	Regressed by a second mixture-of-experts head, MAE loss

The authors justify the trio by complementarity: "sVRI captures variations in amplitude, while IPA measures signal width." SQI is the safety net — when a segment is too noisy or too atypical for the dicrotic notch to be located, IPA is meaningless, and SQI is what tells the model so. They report empirically that SQI is significantly higher (p < 0.05) in signals that do have a visible dicrotic notch, which is the check that makes the safety net trustworthy.

Worked example: computing all three by hand

Take a ten-sample toy beat — absurdly short, but every number below is checkable with a pencil. Sample 3 is the systolic peak; sample 6 is the dicrotic notch.

x = [0.10, 0.50, 1.00, 0.80, 0.50, 0.35, 0.45, 0.35, 0.20, 0.10]

sVRI. The definition, with sys the index of the systolic peak and n the length:

sVRI = [ sys · ∑_i=sysⁿ x_i ] / [ (n − sys) · ∑_i=1^sys x_i ]

Sum the post-systolic part (samples 3 through 10): 1.00 + 0.80 + 0.50 + 0.35 + 0.45 + 0.35 + 0.20 + 0.10 = 3.75. Sum the pre-systolic part (samples 1 through 3): 0.10 + 0.50 + 1.00 = 1.60. With sys = 3 and n = 10:

sVRI = (3 × 3.75) / (7 × 1.60) = 11.25 / 11.20 = 1.004

Now change the physiology. Stiffen the arteries: the reflected wave comes back early and merges into the pulse, so the beat's tail stays high instead of falling away.

x′ = [0.10, 0.50, 1.00, 0.85, 0.70, 0.60, 0.60, 0.50, 0.35, 0.20]

Post-systolic sum: 1.00 + 0.85 + 0.70 + 0.60 + 0.60 + 0.50 + 0.35 + 0.20 = 4.80. Pre-systolic sum unchanged at 1.60.

sVRI′ = (3 × 4.80) / (7 × 1.60) = 14.40 / 11.20 = 1.286

IPA. Area before the notch over area after it. For the first beat, samples 1–6 sum to 0.10 + 0.50 + 1.00 + 0.80 + 0.50 + 0.35 = 3.25, and samples 7–10 sum to 0.45 + 0.35 + 0.20 + 0.10 = 1.10:

IPA = 3.25 / 1.10 = 2.955 IPA′ = 3.75 / 1.65 = 2.273

Look at what the two indices did. sVRI went up (1.00 → 1.29) and IPA went down (2.96 → 2.27). They are not measuring the same thing with different arithmetic; one responds to the amplitude of the tail, the other to how much of the beat's area sits in the diastolic half. That is exactly the complementarity the authors claimed, and you just verified it on ten numbers.

SQI. Skewness is the third standardised moment, m₃ / m₂^3/2. For the first beat the mean is 4.35 / 10 = 0.435. Deviations, squares, and cubes:

x	d = x − 0.435	d²	d³
0.10	−0.335	0.112225	−0.037595
0.50	0.065	0.004225	0.000275
1.00	0.565	0.319225	0.180362
0.80	0.365	0.133225	0.048627
0.50	0.065	0.004225	0.000275
0.35	−0.085	0.007225	−0.000614
0.45	0.015	0.000225	0.000003
0.35	−0.085	0.007225	−0.000614
0.20	−0.235	0.055225	−0.012978
0.10	−0.335	0.112225	−0.037595
sum		0.755250	0.140146

So m₂ = 0.0755250 and m₃ = 0.0140146. Then m₂^3/2 = √0.075525 × 0.075525 = 0.274818 × 0.075525 = 0.020757:

SQI = 0.0140146 / 0.020757 = 0.675

Now feed it a segment that is symmetric noise instead of a pulse — say [0.50, 0.30, 0.70, 0.50, 0.40, 0.60, 0.50, 0.45, 0.55, 0.50], whose mean is exactly 0.50. The deviations are 0, −0.2, +0.2, 0, −0.1, +0.1, 0, −0.05, +0.05, 0. Every deviation is matched by its negative, and cubing preserves sign, so the sum of cubes is exactly zero:

SQI_noise = 0 / m₂^3/2 = 0.000

0.675 versus 0.000. A real pulse is strongly right-skewed — a sharp tall systolic peak with a long low tail — while noise is symmetric. That is the entire physical content of "skewness as a quality index", and it costs three passes over the array.

PPG morphology — the three indices, live

Stiffen or relax the arteries and watch the beat change shape. The three indices are recomputed from the drawn waveform on every frame with the formulas above — nothing is hard-coded. The bin strip at the bottom shows which of PaPaGei-S's eight sVRI buckets this beat falls into: every segment in the world sharing that bucket is a positive pair for it, regardless of whose body it came from.

Vascular tone:

The full objective

With the three indices computed for every segment before training begins, PaPaGei-S's loss has three parts.

Part one: contrastive on sVRI bins. Discretise sVRI into b = 8 bins. Positives are pairs sharing a bin. The paper states the consequence explicitly: "positive pairs are not defined based on participants." The loss is standard NT-Xent over cosine similarities with temperature τ.

ℒ_svri = −log [ exp(sim(z_i, z_j)/τ) / ∑_k≠i exp(sim(z_i, z_k)/τ) ]

Part two and three: regress IPA and SQI. Two mixture-of-experts heads sit on the same encoder embedding. Each is three parallel small feed-forward networks whose outputs are combined by softmax-weighted sum, trained with mean absolute error:

ℒ_ipa = (1/N) ∑ |y^ipa − ŷ^ipa|, ℒ_sqi = (1/N) ∑ |y^sqi − ŷ^sqi|

Combined, with α = 0.6 in their configuration:

ℒ_s = α ℒ_svri + (1 − α)( ℒ_ipa + ℒ_sqi )

Why mixture-of-experts for two scalar regressions, which sounds like overkill? The authors' stated rationale is that "each expert can specialize in learning distinct properties that contribute to the overall index." IPA is a ratio of two areas whose split point is a landmark that may be sharp, shallow, or absent. A single feed-forward head must learn one function covering all three regimes; three experts with a learned gate can partition the input space and specialise.

The augmentation list is the tell. PaPaGei-P uses cropping, negation, flipping, and scaling. PaPaGei-S uses only cropping (0.25) and Gaussian noise (0.25), and the paper says why in one clause: PaPaGei-S "avoids augmentations that alter PPG's morphology." Once your positive pairs are defined by morphology, any augmentation that changes morphology is not a view of the same thing — it is a mislabelled pair. The objective and the augmentation policy have to agree about what is being held invariant, and most contrastive papers never state that contract out loud.

The architecture, and a number worth staring at

The encoder is a 1-D ResNet: 18 convolutional blocks, 32 filters to start, doubling every 4 blocks. A single fully-connected projection produces the 512-dimensional embedding. The two expert heads produce 128-dimensional intermediates and are discarded at inference, so the deployed feature extractor is about 5 million parameters.

Now compare against the general-purpose time-series foundation models the paper benchmarks against — Moment (1024-d embeddings) and Chronos (768-d) — and read the abstract's claim carefully: PaPaGei improves classification metrics by 6.3% and regression metrics by 2.9% in at least 14 of 20 tasks, "while being more data- and parameter-efficient, outperforming models that are 70× larger."

Seventy times smaller and better. That is the single most useful number in the biosignal foundation-model literature, and it is an argument about inductive bias, not about scale. A generic time-series transformer has to discover from data that a pulse has a systolic peak and a dicrotic notch and that the ratio between their areas is meaningful. PaPaGei is handed that structure in its loss function, by way of three formulas from the 2009–2016 signal-processing literature. Domain knowledge, injected as an objective rather than as a feature, buys you nearly two orders of magnitude of parameters.

The bias evaluation most papers skip

PPG is optical, and melanin absorbs green light. This is not a hypothetical concern — it is the mechanism by which pulse oximetry has documented accuracy problems in darker-skinned patients.

PaPaGei's authors evaluate on Vital Videos, 231 participants from Europe and Sub-Saharan Africa, most with Fitzpatrick skin tone 5 or 6, predicting systolic and diastolic blood pressure. Their reported finding is honest and unflattering in the right places: PaPaGei-S beats PaPaGei-P in most cases, is strongest for lighter tones (1–2 for systolic, 2–3 for diastolic), is "comparable to other models" at tones 4 and 5, and is "not the best for darker skin tones", with the authors writing that "additional future work is needed."

The value here is not the result. It is that the result is reported at all, disaggregated by skin tone, in a paper that could have printed one average and moved on. An aggregate metric over a cohort that is 80% one group tells you almost nothing about the other 20%, and the model will meet that 20% in production regardless.

PaPaGei-S defines positive pairs by shared sVRI bin rather than by shared participant. What is the substantive consequence for the learned embedding?

It makes training faster, since sVRI is cheaper to compute than a participant lookup It guarantees the embedding is invariant to skin tone The embedding is organised by vascular physiology instead of by identity. Two strangers with matched arterial behaviour become neighbours, while one person's pre- and post-exercise segments may be pushed apart — so the geometry encodes the mechanism the downstream tasks are about, rather than encoding who the patient is It removes the need for a contrastive loss entirely

Chapter 5: The Embedding Payoff

Three encoders exist. None of them can do anything by itself — an embedding is not a prediction. This chapter is about the three ways you cash one in, and about reading the resulting numbers without fooling yourself.

Cash-out 1: the linear probe

Freeze the encoder. Push your labelled dataset through it once, storing the vectors. Fit a logistic regression on those vectors. That is a linear probe, and it is both the cheapest deployment and the sharpest scientific instrument in the whole toolkit.

Why sharp? Because a linear model cannot construct features. If a linear function of the embedding separates your classes, then the embedding already contained that separation as a direction in space. The probe discovers it; it does not build it. Probe accuracy is therefore a measurement of the encoder, in a way that finetuned accuracy is not.

The cost accounting is worth doing once, because it explains why this is what actually ships:

	Linear probe	Full finetune
Parameters updated	D + 1 (e.g. 513)	All of them (90.9M for ECG-FM)
Passes through the encoder	One, ever — then cache	One per epoch, forward and backward
Hardware	A laptop	An A100, days
Labels needed	Tens to hundreds	Thousands
New task cost	Refit on cached vectors, seconds	Another full training run
Ceiling	Whatever is linearly available	Higher, given enough data

PaPaGei evaluates exclusively this way — frozen embeddings into logistic regression, ridge regression, or a random forest, hyperparameters chosen by cross-validated grid search, across all 20 tasks. That is a deliberate methodological stance: it measures the encoder rather than the finetuning recipe, so comparisons across encoders mean something.

Worked example: reading an AUROC by hand

Every result in these three papers is reported as AUROC or AUPRC. If those are just letters to you, the numbers cannot inform a decision. So compute one from scratch.

Run a probe on seven held-out patients. Three have the condition, four do not. The probe outputs these scores:

positives: 0.91, 0.62, 0.44 negatives: 0.71, 0.55, 0.30, 0.12

AUROC equals the probability that a randomly chosen positive scores above a randomly chosen negative. That definition is not an approximation; it is exact, and it turns the computation into counting. There are 3 × 4 = 12 positive-negative pairs. Count the wins:

Positive	Beats	Loses to	Wins
0.91	0.71, 0.55, 0.30, 0.12	—	4
0.62	0.55, 0.30, 0.12	0.71	3
0.44	0.30, 0.12	0.71, 0.55	2
total			9 / 12

AUROC = 9 / 12 = 0.750

Two properties fall straight out of that construction, and both matter later. AUROC depends only on the ranking, so any monotone rescaling of the scores leaves it unchanged — which means AUROC says nothing about whether the probabilities are calibrated. And it weights both classes equally regardless of prevalence, which means a rare-disease model can post a beautiful AUROC while being useless in the clinic. We cash that second point in below.

Cash-out 2: finetuning, and how much pretraining actually bought

ECG-FM runs the experiment that answers the question everyone should ask: how much of the performance is the pretraining, and how much is just having a big model and a lot of labels?

They compare four conditions at 1%, 10%, 50%, and 100% of the downstream training set:

Condition	What it is	What it isolates
Full	Pretrained weights, all parameters finetuned (lr 1×10⁻⁶)	The best number the method can produce
Random Init.	Same architecture, random weights, full finetune (lr 1×10⁻⁵)	The value of pretraining specifically — weight initialisation is the only difference from Full
Linear	Frozen embeddings, one linear layer	What is linearly available in the representation
ResNet baselines	Two task-specific models competitive in the PhysioNet 2021 challenge	What a good conventional model does

Their findings, in their words and in this order. The Linear results "demonstrate that our pretrained model embeddings encode rich, task-relevant information." Random Init. "performance is relatively poor with few training samples, confirming that ECG-FM's pretraining is responsible for its superior data efficiency." And at the smallest training-set sizes, Linear "outperforms the baselines and performs comparably to Full" on two of the three tasks — "however, its performance plateaus because it lacks the representational capacity necessary to exploit additional downstream data."

That is the complete decision rule, and it is not "always finetune". Few labels: the linear probe on frozen embeddings beats a task-specific network trained from scratch, and roughly matches full finetuning, for a rounding error of the compute. Many labels: finetuning pulls ahead because the probe's linearity becomes the binding constraint. The crossover is a property of your label budget, and you can locate it empirically in an afternoon by fitting probes on 1%, 10%, and 100% and watching where the curves cross.

Probe-transfer matrix

Rows are what you put on top of the signal; columns are label budget. Cells show the shape of the published pattern: a frozen encoder plus a linear probe dominates in the label-starved regime and plateaus, a from-scratch model is helpless there and catches up late, and full finetuning tracks the probe early then keeps climbing. Click a cell to read it. The numbers are illustrative of the reported ordering — the ordering is what the papers establish.

Cash-out 3: retrieval, which needs no training at all

The third use is the one that gets least attention and may be the most clinically natural. You have an embedding for every ECG in your archive. A new patient arrives. Compute their embedding, and ask: which of the last 600,000 patients does this person's heart most resemble?

That is a nearest-neighbour query, and it needs no labels, no training, and no task definition:

sim(a, b) = (e_a · e_b) / (‖e_a‖ ‖e_b‖) → return the top k

The independent evaluation ECG-FM cites is the cleanest demonstration of why frozen embeddings are worth having. A separate group applied the released ECG-FM weights to MEDIC, a private prehospital ambulance dataset of 5,813 ten-second ECGs with 20% prevalence of acute coronary syndrome. They did not finetune. They extracted frozen embeddings and fitted XGBoost.

Frozen encoder	ACS detection AUPRC
ST-MEM (generative-only SSL)	0.594 ± 0.013
ECG-FM (hybrid generative + contrastive)	0.667 ± 0.024

Ambulances, not hospitals. A different country's protocol, a different machine, a different acuity mix, a task nobody trained for. The embeddings still carried usable structure — and the hybrid objective beat the purely generative one by more than three standard deviations, which is the sharpest available evidence that the contrastive half of ECG-FM's loss is doing real work rather than decorating the abstract.

Reading the headline numbers without being fooled

ECG-FM's abstract reports AUROC 0.996 for atrial fibrillation. That number is true and it is not what most readers will take from it. Here is the full row from their UHN-ECG interpretation table:

Metric	Value	What it means
AUROC	0.996	A random AF case outranks a random non-case 99.6% of the time
AUPRC	0.968	Precision-recall area, which does respond to prevalence
Recall (sensitivity)	0.957	96% of true AF is caught
Specificity	0.980	2% of non-AF patients are falsely flagged
Precision (PPV)	0.814	81% of alarms are real — on this cohort

Worked example: the same model in a different population

Precision is the metric clinicians actually feel, and it is not a property of the model alone. Take the model's sensitivity (0.957) and specificity (0.980) as fixed — they are properties of the model at that threshold — and deploy it in outpatient screening where AF prevalence is 2%. Ten thousand patients:

Group	Count	Flagged	Arithmetic
Has AF	200	191.4	200 × 0.957
No AF	9,800	196.0	9,800 × (1 − 0.980)
Total alarms		387.4	191.4 + 196.0

PPV = 191.4 / 387.4 = 0.494

AUROC 0.996, and slightly more than half your alarms are false. Nothing about the model changed — same weights, same threshold, same sensitivity and specificity to three decimals. Only the population changed. You can even run the arithmetic backwards to find the prevalence implied by their reported precision of 0.814: solving 0.814 = 0.957π / (0.957π + 0.020(1 − π)) gives π ≈ 0.084. About 8% AF prevalence — entirely plausible for ECGs ordered in an emergency department, and four times the screening rate. Precision is a statement about the room the model is standing in.

None of this is a criticism of ECG-FM, whose evaluation is unusually careful. It is a criticism of how these numbers travel. "AUROC 0.996 for AF" will be quoted in a slide deck for a screening product, and the arithmetic above is what that deck is not showing.

A frozen-encoder linear probe beats a task-specific network trained from scratch when labels are scarce, but plateaus as labels grow while full finetuning keeps improving. What causes the plateau?

The probe overfits the small label set and cannot recover The encoder's weights slowly degrade as more data passes through it Logistic regression cannot be trained on more than a few thousand examples The probe can only exploit structure that is already linearly available in the frozen embedding. Extra labels cannot add representational capacity, because no encoder parameter moves. Finetuning can reshape the representation itself, so it converts additional labels into additional capability

Chapter 6: Evaluation Traps

Everything so far has been construction. This chapter is destruction, and it is the most useful one, because a biosignal foundation model is far easier to appear to evaluate than to actually evaluate. Three traps, in increasing order of how often they are stepped in.

Trap 1: subject leakage

You have 100 participants, each contributing 60 ten-second segments. That is 6,000 segments. You shuffle them and take an 80/20 split. It feels rigorous — the test segments were never trained on.

Compute what that split actually did. For any particular subject, the probability that none of their 60 segments landed in the training set is:

0.2⁶⁰ ≈ 1.2 × 10⁻⁴²

Check the magnitude: log₁₀(0.2⁶⁰) = 60 × log₁₀(0.2) = 60 × (−0.699) = −41.9. So it is a number with 41 zeros after the decimal point. It will never happen. In fact the expected number of same-subject segments sitting in the training set, for each test segment, is:

59 × 0.8 = 47.2 twins per test segment

Every test segment has roughly forty-seven near-copies of itself in training, recorded from the same body, on the same device, in the same session, minutes apart.

Now recall Chapter 0's warning: these signals are biometrics. And recall Chapter 4: PaPaGei-P's objective is literally "same subject → near". So the model has both the means and the motive to solve your task by retrieval instead of by physiology.

Worked example: sizing the illusion

Suppose the honest, physiology-only accuracy on your task is 68%. Suppose also that identity matching from a 10-second segment succeeds 92% of the time, which is conservative for ECG or PPG within a single session. Your label — smoker, hypertensive, diagnosis — is constant per subject, as most clinical labels are. Then the leaked model's accuracy is:

Path	Probability	Accuracy on that path	Contribution
Identity matched → copy that subject's label	0.92	1.00	0.920
Identity missed → fall back on physiology	0.08	0.68	0.054
Reported accuracy			0.974

97.4% reported, 68% real. A 29-point illusion, produced by a split that looked reasonable, on a model that learned nothing wrong — it optimised exactly what you asked. And the illusion is invisible from inside: the loss curve is beautiful, the test metric is stable across seeds, the confusion matrix looks sensible. The only way to see it is to change the split.

The fix is one line and it is not optional: group the split by subject. Every segment from a participant goes entirely into train, or entirely into test, never both. Watch what the three papers do:

Paper	What they did about it
ECG-FM	Split UHN-ECG by patient and by time, and refused to pretrain on it at all, so downstream evaluation touches a hospital network the encoder has never seen
PaPaGei	Marks the out-of-domain evaluation datasets explicitly in their table — for the datasets that were used in pretraining, they hold out test sets and labels; the remainder were never seen during training
LaBraM	Ran an ablation comparing pretraining with and without the downstream datasets included, to measure how much of the result came from having seen that distribution

That LaBraM experiment deserves a note. It is not a split-hygiene question, it is a pretraining contamination question, one level up: even with clean downstream splits, if TUAB's distribution was part of your unlabelled pretraining mixture, your "generic representation" has had a rehearsal. The authors explicitly wanted "general EEG representations that are not specific to any particular task", and the only way to check is to remove the dataset from pretraining and see what happens.

The split that lies

Twelve subjects, each with several segments; colour is the subject, shape is the label. Switch between a random segment split and a subject-wise split, and read the two numbers at the bottom: how many test segments have a same-subject twin in training, and what the reported score becomes when the model can exploit them.

Trap 2: dataset shift, and why one hospital is not a distribution

Assume you fixed the split. Your model now generalises to unseen patients — recorded on the same machine, by the same technicians, under the same protocol, in the same building, on the same case mix.

Enumerate what changes when you move it:

Shift	Concrete instance	What breaks
Device	A different ECG cart with a different anti-aliasing filter and mains notch	Spectral content the encoder learned as "normal" moves
Sampling rate	250 Hz cart instead of 500 Hz — 11.2% of UHN's own ECGs are the former	Resampling artifacts change the fine morphology
Placement	Wrist PPG instead of fingertip; different melanin, different perfusion	Beat morphology — PaPaGei's entire objective — changes shape
Prevalence	Emergency department (~8% AF, derived in Chapter 5) vs outpatient screening (~2%)	Precision collapses even though the model is unchanged
Case mix	ICU patients under sedation vs healthy adults asleep	The physiological range itself is different
Protocol	Which patients get an ECG at all, and when	Label meaning shifts — "normal" in an ED is not "normal" in a clinic

This is why PaPaGei's design is what it is. Their contribution list names "insufficient exploration of out-of-domain generalization" as one of three limitations in prior work, and they answer it structurally: pretrain on three datasets, evaluate on ten, with the unseen ones marked, spanning cardiovascular health, sleep disorders, pregnancy monitoring, and wellbeing. Twenty tasks is not padding — it is the only way to distinguish "this encoder is general" from "this encoder suits VitalDB."

The single most informative experiment in this whole literature is ECG-FM's: pretrain on public data, evaluate on a private hospital network the encoder has never seen, and then let an independent group evaluate the released weights on prehospital ambulance data from a different study entirely. The AUPRC of 0.667 on that ambulance dataset is worth more than any in-distribution number in the paper, because nobody involved could have tuned anything toward it.

Aggregation, a quieter shift

ECG-FM's default metrics are computed by randomly sampling a single 5-second segment per ECG. That is a deliberate choice — using all segments would put correlated halves of the same recording into the same evaluation, inflating the effective sample size.

But they also report what happens if you aggregate a recording's segments before predicting. For most labels the gain is fractions of a percent. For premature ventricular contractions it is not:

Label	Aggregation	AUPRC change
Sinus rhythm	mean	+0.03%
Atrial fibrillation	mean	+0.51%
PVC	max	+16.65%

The pattern is physiological, and once you see it you cannot unsee it. Sinus rhythm and AF are properties of the whole recording — averaging over segments reduces variance and nothing else. A PVC is a single ectopic beat that may occur in one 5-second half and not the other. Averaging dilutes it; taking the maximum finds it. The right aggregation function is a statement about whether your label describes a state or an event, and getting it wrong costs 17 points of AUPRC on exactly the labels clinicians care about spotting.

Trap 3: calibration, and what AUROC refuses to tell you

Recall from Chapter 5 that AUROC is a pure ranking statistic. Apply any monotone function to every score and it does not move. That has a consequence people find surprising: a model can have excellent AUROC and probabilities that are systematically wrong.

The standard measure is Expected Calibration Error. Bin predictions by confidence, and in each bin compare the average confidence to the observed frequency:

ECE = ∑_b (n_b/N) · |accuracy_b − confidence_b|

Worked example

A thousand held-out ECGs, three confidence bins:

Bin	n	Mean predicted p	Observed rate	\|gap\|	Weighted
[0.00, 0.33)	500	0.15	0.10	0.05	0.5 × 0.05 = 0.025
[0.33, 0.66)	300	0.50	0.38	0.12	0.3 × 0.12 = 0.036
[0.66, 1.00]	200	0.88	0.71	0.17	0.2 × 0.17 = 0.034
ECE					0.095

Nearly ten points of average miscalibration, with the worst error in the high-confidence bin: when this model says 88%, the truth is 71%. And notice that every bin's ordering is intact — higher predicted probability still means higher observed rate — so the AUROC is untouched. The ranking is good. The numbers are lies.

Why this is not an academic complaint. A ranking is enough when a human reads the top of a worklist. A probability is required the moment the output enters a decision rule: "flag if p > 0.9", "escalate if risk exceeds 15%", or any cost-weighted threshold. Clinical deployment always ends up in the second regime, because that is what a protocol is. And overconfidence is the dangerous direction — a model that says 88% when it means 71% will have its errors trusted.

The good news is that the standard fix is one parameter. Temperature scaling: fit a single scalar T on a validation set and divide the logits by it before the softmax. It is the same temperature knob from Chapter 3's contrastive loss, used for the opposite purpose — there it sharpened the competition, here it softens overconfident outputs. Because dividing all logits by a positive constant is monotone, AUROC is mathematically unchanged, so calibration here is free: you cannot lose ranking performance by fitting it.

The checklist

Question to ask of any biosignal result	Why
Was the split grouped by subject?	Otherwise the number may be an identity lookup (29 points, above)
Was any evaluation dataset in the pretraining mixture?	Contamination one level up; LaBraM measures this explicitly
What is the prevalence in the evaluation cohort, and in deployment?	Precision moves with prevalence even when the model does not (0.814 → 0.494)
Is the metric AUROC alone?	Then you know nothing about calibration and little about the rare class — ask for AUPRC too
How were multiple segments per recording aggregated?	mean vs max is worth 17 AUPRC points on event-type labels
Was the model evaluated on a device or site it never saw?	Everything else is in-distribution rehearsal
Are results disaggregated by skin tone, sex, age, device?	An average over an unbalanced cohort hides the subgroup you will fail

You split 6,000 segments from 100 subjects randomly, 80/20. Your test accuracy is 97%. A colleague re-splits the data by subject and gets 68%. What most likely happened?

The subject-wise split has fewer training samples, so the model underfits The random split put ~47 same-subject twins in training for every test segment. Since biosignals are biometrics and the label is constant per subject, the model could reach 97% by recognising who the segment came from and copying that person's label — a shortcut the grouped split removes. 68% is the honest number The subject-wise split is incorrect; random splitting is the standard protocol The difference is normal seed variance and both numbers are valid

Chapter 7: Embeddings Leak Identity

Chapter 6 treated identity encoding as a measurement problem — it corrupts your test numbers. This chapter treats it as what it also is: a disclosure problem.

The setup that makes it concrete. You have deployed a wearable. Raw PPG never leaves the device; only the 512-dimensional embedding is uploaded, and the privacy notice says so in good faith. The intuition behind that design is that an embedding is an abstraction — a summary, a compressed gist, something less than the data.

Let us find out whether that intuition survives arithmetic.

The capacity argument

A PaPaGei embedding is 512 float32 numbers:

512 × 32 bits = 16,384 bits

The input it summarises is a 10-second segment at 125 Hz. Quantise each sample to 8 bits, which is generous for a z-scored waveform whose diagnostic content is its shape:

1250 samples × 8 bits = 10,000 bits

The embedding is larger than the thing it summarises. Sixteen thousand bits out, ten thousand bits in. There is no information-theoretic bottleneck here at all. Whether the waveform is recoverable from the embedding is not limited by capacity — it is limited only by whether the training objective happened to discard it. And "the objective happened to discard it" is not a security property. It is a hope about an optimisation you did not control.

Contrast this with LaBraM's tokenizer output, which we computed in Chapter 2: a single 13-bit code index per patch, from a 6,400-bit patch. That is a bottleneck — a 492× one. The general rule falls out cleanly:

Representation	Bits	vs input	Leakage potential
LaBraM neural code (one patch)	13	492× compression	Low — a cluster identity, nothing more
PaPaGei embedding (10 s PPG)	16,384	1.6× expansion	High — capacity to spare
ECG-FM token sequence (5 s, 12-lead)	156 × 768 × 32 = 3.83M	16× expansion over the raw samples at 8 bits	Very high — this is a re-encoding, not a summary

Notice which direction the arrow points. The more expressive the representation — the more "foundation model" it is — the more it can carry. A model designed for maximal downstream usefulness is, by construction, a model that discarded as little as possible.

The objective argument, which is worse

Capacity says leakage is possible. The training objectives say something stronger.

Go back and read what these losses literally optimise:

Objective	Positive pair definition	What it makes the embedding good at
PaPaGei-P (their baseline)	Two segments from the same participant	Telling participants apart. This is the textbook training objective for a biometric identification system
ECG-FM's CMSC	Two adjacent 5-second halves of the same recording	Telling recordings apart — a weaker version of the same thing, since one recording is one person at one moment
PaPaGei-S	Two segments in the same sVRI morphology bin, regardless of person	Telling vascular states apart. Identity is not rewarded — and may be actively discouraged, since one person's segments can land in different bins
LaBraM	None — masked code prediction, no contrastive term	Predicting local spectral structure. Identity is incidental, not optimised

PaPaGei-S's morphology objective is a privacy property, and the paper does not claim it as one. They chose it for representation quality — identity-based positives waste capacity on a nuisance variable. But the side effect is real: an encoder whose positives are defined by physiology rather than by person has strictly less incentive to encode the person. This is worth naming, because it is a rare case where the accuracy-motivated choice and the privacy-motivated choice point the same way.

An illustrative re-identification calculation

The papers do not report re-identification rates, so we will not invent any. What we can do is take plausible numbers, state them as assumptions, and see what the geometry implies. Treat everything in this section as a model, not a measurement.

Suppose an adversary holds a gallery of 1,000 enrolled people, each represented by the mean embedding of their known segments. They obtain one anonymous embedding and score it against all 1,000 by cosine similarity. Assume:

Quantity	Assumed value
Cosine similarity to the correct person's centroid	0.86
Cosine similarity to a wrong person: mean	0.31
Cosine similarity to a wrong person: standard deviation	0.08

How far above the impostor distribution does the true match sit?

z = (0.86 − 0.31) / 0.08 = 0.55 / 0.08 = 6.875 standard deviations

For a normal tail, P(Z > z) ≈ φ(z)/z. With z²/2 = 23.63, we get e^−23.63 ≈ 5.5 × 10⁻¹¹, so φ(6.875) ≈ 0.399 × 5.5×10⁻¹¹ ≈ 2.2 × 10⁻¹¹, and:

P(one impostor scores above the true match) ≈ 2.2×10⁻¹¹ / 6.875 ≈ 3.2 × 10⁻¹²

Across 999 impostors, the expected number beating the true match is 999 × 3.2×10⁻¹² ≈ 3 × 10⁻⁹. Rank-1 identification succeeds essentially every time.

The instructive part is the sensitivity. Halve the separation to 0.275 and z drops to 3.44; the per-impostor tail becomes about 3 × 10⁻⁴, and across 999 impostors you expect roughly 0.3 of them to beat the true match — identification becomes unreliable. Re-identification risk is exponentially sensitive to how cleanly the embedding separates people, which is precisely the quantity a subject-contrastive objective is trained to maximise.

What the ECG-FM authors did about it

This is the practical bridge, and it is why their data table in Chapter 3 deserves a second look.

They had 622,000 cardiologist-over-read ECGs from their own hospital network — by far their highest-quality data — and they did not pretrain on it. Their stated reason: "so that we may release our pretrained model while respecting patient privacy."

Read that as a threat model, because that is what it is. Released weights are a function of their training data. Nobody publishes the training set, but the weights were fitted to it, and the literature on memorisation in large models makes "the weights contain nothing recoverable" an assumption rather than a guarantee. The authors declined to test the assumption on 211,000 real patients. They paid for that in data quality and got two things back: a releasable model, and a genuinely out-of-distribution benchmark. It is the most quietly grown-up decision in the three papers.

What follows for a system you might build

Belief	Correction
"We only upload embeddings, so no personal data leaves the device"	The embedding may be larger than the waveform it came from and is trained to be person-discriminative. Treat an embedding store as identifiable health data, with the same access controls as the raw signal
"The vector database is just an index"	A vector database of patient embeddings is a biometric gallery. Its retrieval function is a re-identification API, and it will be usable that way whether or not that was the intent
"We anonymised by removing names"	Identifier removal is orthogonal. The waveform is the identifier
"Pretraining on our private corpus is fine because we will not publish the data"	Publishing the weights publishes a function of the data. ECG-FM's choice shows the alternative is affordable
"Subject-wise splits protect privacy"	They protect validity. They do nothing about disclosure — different problem, different fix

The other half of the stakes

Privacy is one failure axis. The other is what happens when a well-calibrated-looking model is wrong in a room with a patient in it.

The three quantities from Chapters 5 and 6 compose into the actual clinical picture. Take AF screening at 2% prevalence, where we computed a PPV of 0.494. Now add ECE 0.095 with overconfidence in the top bin. The alert says "94% likely atrial fibrillation". Slightly more than half of those alerts are wrong, and the stated confidence overstates the truth by roughly seventeen points in that bin. Meanwhile the clinician has automation bias — agreement with a confident machine is the path of least resistance, especially at hour eleven of a shift.

Both error directions have a named cost, and they are not symmetric. A false negative on AF is a missed opportunity to prevent a stroke. A false positive is an anticoagulant with real bleeding risk, or a cascade of confirmatory tests, in someone who was fine. Whichever direction you tune, you are choosing a harm. That choice is a threshold, the threshold requires calibrated probabilities, and calibration is the thing AUROC is mathematically incapable of reporting. That is the whole chain, and it starts with a self-supervised objective on unlabelled data.

A wearable uploads only 512-dimensional PPG embeddings, never raw waveforms. Why is "no personal data leaves the device" a weak claim?

Because the embedding could be intercepted in transit Because 512 dimensions is too many and a smaller embedding would be safe Because embeddings can be reversed exactly back into the original waveform Because 512 float32 values are 16,384 bits, more than the 10,000 bits needed to store the 10-second waveform at 8-bit resolution — there is no capacity bottleneck — and subject-contrastive objectives explicitly train the embedding to separate people. The embedding is a biometric with room to spare, not an anonymised summary

Chapter 8: One Recipe, Three Signals

Three papers, three labs, three continents, no coordination. Lay their pipelines side by side and the shared skeleton is unmistakable — and so is the place where each one had to stop being generic and start being about the body it measures.

The shared skeleton

Stage 1 · canonicalise

Force one sampling rate, one amplitude convention, one filter band. Delete every degree of freedom that describes the recorder rather than the body

↓

Stage 2 · choose a unit

Cut the stream into a unit that exists in every dataset: a 1-second channel patch, a 5-second 12-lead segment, a 10-second single-channel window

↓

Stage 3 · invent supervision

Manufacture a target from structure already in the data: a discrete code to predict, an adjacent segment to match, a morphology bin to share

↓

Stage 4 · encode

Transformer or 1-D ResNet → a fixed-length vector per unit

↓

Stage 5 · cash out

Freeze and probe, or finetune, or retrieve. Labels enter here and nowhere earlier

The full comparison

Decision	LaBraM (EEG)	ECG-FM (ECG)	PaPaGei (PPG)
Unlabelled scale	~2,500 h, ~20 datasets	1.5M 12-lead ECGs	57,641 h, 20.7M segments, 13,517 people
Resample to	200 Hz	500 Hz	125 Hz (the corpus minimum)
Filtering	0.1–75 Hz bandpass + 50 Hz notch	z-score normalisation	4th-order Chebyshev 0.5–12 Hz
Amplitude convention	Unit = 0.1 mV, so values sit near ±1	z-score	z-score
Quality gate	—	None — poor-quality ECGs kept and predicted	Drop segments >25% flatline
Unit of input	1 channel × 1 s patch, ≤256 patches	12 leads × 5 s	1 channel × 10 s
Channel handling	Patches + learned spatial embedding — montage-agnostic	12 leads as input channels + random lead masking	Single channel, no question
Backbone	Transformer, 5.8M / 46M / 369M	wav2vec 2.0: CNN + BERT-Base, 90.9M	1-D ResNet, 18 blocks, ~5M at inference
Discretisation	8,192-code neural codebook, D = 64, ℓ₂-normalised lookup	Two codebooks of 320, product-quantised → 102,400 codes	None on the representation; sVRI discretised into 8 bins
Generative objective	Masked code prediction, cross-entropy, mask ratio 0.5 + symmetric mask	Masked contrastive prediction, ~49% masked in 10-token spans	—
Contrastive objective	—	CMSC: adjacent segments are positives	NT-Xent over shared sVRI bin
Auxiliary objective	Tokenizer regresses Fourier amplitude + phase	Codebook diversity loss	Two mixture-of-experts heads regress IPA and SQI (MAE)
Augmentations	None	Random lead masking only	Cropping 0.25 + Gaussian noise 0.25 — nothing that alters morphology
Embedding used downstream	Pooled patch tokens	768-d contextual vectors, 156 per segment	512-d projected embedding
Evaluation style	Full finetune; balanced accuracy, Cohen's κ	Finetune + linear probe + random-init control, at 1/10/50/100% of labels	Linear probe only, 20 tasks, 10 datasets
Headline	TUEV κ 0.6637 vs BIOT 0.5273	AF AUROC 0.996; LVEF ≤40% AUROC 0.929	+6.3% classification, +2.9% regression, beating models 70× larger

The unified recipe, and where each signal diverges

Three lanes through the same five stages. Solid segments are shared structure; the branch markers are where physiology forced a different answer. Step through the stages to see what each paper decided and why.

Where the three genuinely disagree, and who is right

Disagreement 1: should the representation be discrete?

LaBraM says yes, emphatically, and pays for a whole extra training stage to get it. ECG-FM says yes but only for the target, not for the representation the downstream model sees. PaPaGei says no.

The reconciliation is that discretisation solves two different problems and each paper had a different one. LaBraM discretised because raw EEG targets are unlearnably noisy — Chapter 2's flat-line collapse. ECG-FM discretised because raw ECG targets are too easy to match on artifacts, which is a generalisation problem rather than a convergence problem. PaPaGei never needed either, because their objective never asks the model to reconstruct anything: their supervision is three scalars computed by formula. No reconstruction, no reconstruction target, no need to discretise it.

The transferable rule. Discretise the prediction target when your raw signal contains a large component that is either unpredictable (noise, phase) or predictable-but-irrelevant (device fingerprints, contact artifacts). The codebook is a bandwidth limit on what the pretext task is allowed to be about. If your objective does not involve reconstruction at all, the question does not arise.

Disagreement 2: what makes two segments "the same"?

This is the sharpest disagreement in the three papers, and it maps directly onto the privacy analysis in Chapter 7.

Answer	Who	Assumption it rests on	Failure mode
Same recording, adjacent in time	ECG-FM (CMSC)	Cardiac function is stable across seconds	Fails on arrhythmias that come and go within the recording
Same participant	PaPaGei-P, and most prior work	A person's physiology is a stable class	Learns identity; pushes apart two people who are physiologically identical
Same morphology bin	PaPaGei-S	The chosen index actually captures the physiology that matters	Fails if sVRI is the wrong summary, or if binning is too coarse or too fine
Nothing — no positives at all	LaBraM	Masked prediction alone yields useful structure	Weaker global/semantic structure; more finetuning needed downstream

PaPaGei's own ablation answers this within PPG: the morphology objective beats the participant objective, and beats generic time-series foundation models seventy times its size. Whether the same holds for ECG is an open question with an obvious experiment attached — the ECG analogue of sVRI would be some combination of interval durations and axis, and nobody has run it.

Disagreement 3: finetune or freeze?

LaBraM finetunes everything and reports finetuned numbers. PaPaGei freezes everything and reports only linear probes. ECG-FM does both and shows you the crossover.

ECG-FM is right, and not because both is more work. Reporting only finetuned numbers confounds encoder quality with finetuning-recipe quality; reporting only probe numbers understates what the encoder can support. Running both with a random-initialisation control is the only configuration that isolates what pretraining contributed — which was, in their result, most of the label efficiency.

If you were building the fourth one

Say you have a signal these papers do not cover — EMG, respiratory effort belts, continuous glucose, accelerometry. The recipe transfers, and the three papers tell you which decisions to actually think about:

Decision	How to make it
Sampling rate	The minimum across your corpora. Do not upsample to match the best device; you would be teaching the model a device fingerprint
Filter band	Where the physiology is, stated as a claim you can defend. PaPaGei's 0.5–12 Hz is an assertion about pulsatile content
Segment length	Long enough to contain at least one full cycle of the phenomenon; short enough that a label is plausibly constant across it
The unit	The smallest thing that exists in every dataset you want to use. That is what makes cross-dataset pretraining possible at all
Positive pairs	Prefer structure that already exists (adjacency, shared derived index) over invented augmentations. If you must use identity, know that you are training a biometric
Augmentations	List each one, and name the diagnosis it would destroy. If you cannot, do not use it
Discretise the target?	Only if the raw target is dominated by noise or by device idiosyncrasy
Domain knowledge	Inject it as an objective, not as an input feature. PaPaGei's 70× parameter advantage is what that buys
Evaluation	Grouped splits, at least one unseen site or device, probe + finetune + random-init control, AUPRC alongside AUROC, disaggregated by subgroup

LaBraM discretises its prediction target with an 8,192-code vocabulary; ECG-FM discretises its target with product-quantised codebooks; PaPaGei discretises nothing about its representation. What best explains the difference?

PPG is simpler than EEG or ECG, so it needs less machinery PaPaGei was published later and discretisation had fallen out of favour Discretisation is a bandwidth limit on a reconstruction target. LaBraM needed it because raw EEG targets are dominated by unpredictable noise; ECG-FM needed it because raw ECG targets let the model memorise device artifacts. PaPaGei's objective never reconstructs anything — it regresses three formula-computed scalars — so there is no target to bandlimit Discretisation only works for transformers, and PaPaGei uses a ResNet

Chapter 9: Connections & Cheat Sheet

Ten chapters compressed into what you should be able to reconstruct from memory, followed by where each thread goes next.

The one-page summary

Idea	The compressed form
The problem	Biosignals are recorded automatically and labelled by specialists. The ratio is thousands to one
Why transfer learning did not fix it	Corpora disagree on tensor shape and label space. Every dataset gets its own model. Small models, small ceilings
The escape	Self-supervised pretraining on the unlabelled flood → embeddings → a tiny probe per task
ECG anatomy	P (atria) → PR pause (AV node) → QRS 80–100 ms (ventricles, 1–2 mV) → ST (plateau) → T (repolarisation)
EEG's constraint	~100× weaker, smeared by skull, low SNR, non-stationary. Structure lives in frequency bands, not in waveform shape
PPG's constraint	Optical volume trace. Morphology carries the meaning and motion destroys morphology. Melanin absorbs green light
Why raw reconstruction fails on EEG	Right amplitude, wrong phase scores A²; predicting zero scores A²/2. The optimiser is pushed to the flat line
Vector quantization	Nearest code by cosine after ℓ₂ normalisation. Normalising removes magnitude from the competition and stops codebook collapse
Codebook health	perplexity = e^H(usage). Counts 70/20/7/3 → H = 0.863 → 2.37 effective codes out of 4
LaBraM in one line	Tokenize patches into 8,192 codes by predicting the Fourier spectrum, then BERT the codes with mask ratio 0.5 and a symmetric second view
ECG token arithmetic	5 s × 500 Hz = 2500 → stride 16 → 156 tokens → 32 ms each → QRS ≈ 3 tokens
Masking arithmetic	span-start p = 0.065, span 10 → P(unmasked) = 0.935¹⁰ = 0.511 → 48.9% masked; a span is 320 ms ≈ 40% of a beat
ECG-FM in one line	wav2vec 2.0 on 12 leads, plus CMSC (adjacent segments are positives), plus random lead masking
Contrastive temperature	Same similarities, τ = 0.1 → loss 0.32 and the hardest negative dominates; τ = 1.0 → loss 1.17 and all negatives contribute equally
PaPaGei in one line	Positives are segments sharing an sVRI bin, not a person, plus mixture-of-experts heads regressing IPA and SQI
Morphology indices	sVRI = post/pre-systolic mean ratio (amplitude); IPA = systolic/diastolic area at the notch (width); SQI = skewness (quality, 0.68 for a pulse, 0.00 for symmetric noise)
AUROC, exactly	P(random positive outranks random negative). Count wins over pairs: 9/12 = 0.750. Invariant to any monotone rescaling
The prevalence trap	Sensitivity 0.957, specificity 0.980, prevalence 2% → PPV 0.494. AUROC 0.996 and half the alarms are false
Subject leakage	100 subjects × 60 segments, random 80/20 → 47 same-subject twins per test segment → 97.4% reported vs 68% honest
Calibration	ECE = ∑(n_b/N)\|acc − conf\|. A model can be perfectly ranked and badly calibrated. Temperature scaling fixes calibration without touching AUROC
Aggregation	State labels → mean; event labels → max. PVC AUPRC moves 16.65% on that choice alone
Privacy	512 float32 = 16,384 bits > 10,000 bits of 8-bit waveform. No bottleneck. Subject-contrastive objectives train the biometric on purpose
The unified recipe	Canonicalise → choose a universal unit → invent supervision from existing structure → encode → probe. Labels enter only at the end

Where the threads go next, on this site

Thread from this lesson	Where it continues
PPG in a consumer wearable, end to end	Passive Heart Rate Monitoring — the sensor, the motion problem, and what a watch does with the waveform
Vector quantization and codebook collapse	VAE and VQ-VAE — where LaBraM's tokenizer comes from, derived properly
The DFT, amplitude, and phase	EE269: The DFT, then the STFT — the transform LaBraM's tokenizer reconstructs
Why 13 bits per patch is a design choice	Quantization noise and Lloyd-Max — the classical theory under every codebook
NT-Xent, InfoNCE, temperature	Contrastive Learning — the loss family CMSC and PaPaGei both live in
Masked modeling as a pretext task	Audio-MAE — the same idea on spectrograms, with the reconstruct-vs-tokenize argument in a different domain
Where the wav2vec 2.0 architecture came from	CLAP and Whisper — the speech lineage ECG-FM borrowed wholesale
What an embedding is and how to evaluate one	Vector Embeddings and Embedding Benchmarks
Stationarity, and why EEG statistics move	Stationary Signals

Open questions worth an afternoon

These are not rhetorical. Each is a small experiment somebody could run this week.

Question	Why it is open
Does a morphology-defined positive pair beat a subject-defined one for ECG?	PaPaGei showed it for PPG. The ECG analogue would bin on intervals and axis. Nobody has published the comparison
How much of each encoder's embedding is identity, quantitatively?	Train a probe to predict subject ID from the frozen embedding. High probe accuracy is a direct measurement of the leakage Chapter 7 could only bound
Can one encoder serve all three signals?	All three tokenise into fixed-length units. Nothing structurally forbids a shared transformer with per-modality stems — and the datasets frequently contain two signals simultaneously
Does LaBraM's 500-hour-beats-2500-hour result survive better data curation?	It may say more about mixture composition than about data volume
What is the right aggregation function, learned rather than chosen?	mean vs max is worth 17 AUPRC points and is currently a hand-set hyperparameter

The sentence to leave with. Every one of these three papers is, structurally, the same argument: the labels were never the bottleneck — the representation was. Take the flood of unlabelled signal, invent supervision out of structure that is already sitting in the data, and the teaspoon of labels you have turns out to be enough. What separates a good version from a bad one is not model size. It is whether the invented supervision is about the body or about the recording equipment — and whether your evaluation split is honest enough to tell you which one you got.

Which single sentence best captures why these three papers matter more than their individual benchmark numbers suggest?

They showed that transformers outperform CNNs on physiological signals They demonstrated that scaling to hundreds of millions of parameters is what unlocks clinical accuracy They converted a labelling problem into a representation problem: supervision is manufactured from structure already present in unlabelled recordings, so the scarce labels are needed only for a final linear step — and the quality of the result turns on whether the invented supervision tracks physiology rather than recording equipment They proved that EEG, ECG, and PPG can be processed by identical architectures with no modality-specific design

References

Jiang, W.-B., Zhao, L.-M. & Lu, B.-L. "Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI." ICLR 2024. arXiv:2405.18765
McKeen, K., Masood, S., Toma, A., Rubin, B. & Wang, B. "ECG-FM: An Open Electrocardiogram Foundation Model." 2024, revised 2025. arXiv:2408.05178
Pillai, A., Spathis, D., Kawsar, F. & Malekzadeh, M. "PaPaGei: Open Foundation Models for Optical Physiological Signals." ICLR 2025. arXiv:2410.20542
van den Oord, A., Vinyals, O. & Kavukcuoglu, K. "Neural Discrete Representation Learning" (VQ-VAE). NeurIPS 2017. arXiv:1711.00937
Baevski, A., Zhou, H., Mohamed, A. & Auli, M. "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." NeurIPS 2020. arXiv:2006.11477
Peng, Z. et al. "BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers." 2022. arXiv:2208.06366 — the source of LaBraM's ℓ₂-normalised codebook lookup
Kiyasseh, D., Zhu, T. & Clifton, D. "CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients." ICML 2021. arXiv:2005.13249 — where CMSC comes from
Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR, NT-Xent). ICML 2020. arXiv:2002.05709
Abbaspourazad, S. et al. "Large-scale Training of Foundation Models for Wearable Biosignals." 2023. arXiv:2312.05409 — the participant-level contrastive approach PaPaGei-S argues against
Yang, C., Westover, M. B. & Sun, J. "BIOT: Biosignal Transformer for Cross-data Learning in the Wild." NeurIPS 2023. arXiv:2305.10351 — LaBraM's strongest baseline
Obeid, I. & Picone, J. "The Temple University Hospital EEG Data Corpus." Frontiers in Neuroscience, 2016 — the source of TUAB and TUEV
Lee, H.-C. et al. "VitalDB, a high-fidelity multi-parameter vital signs database in surgical patients." Scientific Data 9:279, 2022
Elgendi, M. "Optimal Signal Quality Index for Photoplethysmogram Signals." Bioengineering 3(4):21, 2016 — the SQI in PaPaGei's loss
Lyu, Y. et al. "Measuring Photoplethysmogram-Based Stress-Induced Vascular Response Index to Assess Cognitive Load and Stress." CHI 2015 — the sVRI in PaPaGei's loss
Goswami, M. et al. "MOMENT: A Family of Open Time-series Foundation Models." ICML 2024. arXiv:2402.03885 — one of the general-purpose baselines PaPaGei outperforms at a fraction of the size

Three Electrical Windows Into the Body