Passive Heart-Rate Monitoring

Chapter 1: Light, Blood, and the Pulse You Can't See

How can a camera possibly see your heartbeat? The answer is photoplethysmography (PPG) — a mouthful that just means "measuring blood volume with light." Each time your heart beats, it pushes a pulse of blood through the vessels just under your skin. More blood means slightly more light absorbed and slightly less reflected back. So with every heartbeat, your skin gets imperceptibly darker, then lighter, then darker again — a rhythmic flicker far too subtle for the eye, but not for a camera averaging over many pixels.

Remote PPG (rPPG) does this at a distance, from ordinary video: track a patch of facial skin across frames, average its color, and you recover a faint waveform that rises and falls with the pulse. The frequency of that waveform is the heart rate. A pulse oximeter on your fingertip does the same thing with a dedicated LED; rPPG does it with the front camera and the ambient light already in the room.

Now the crux of this whole paper. The pulsatile signal is tiny — a fraction of a percent change in brightness — sitting atop a mountain of noise: lighting shifts, head motion, camera sensor noise. The ratio of the real signal to that noise is the signal-to-noise ratio (SNR), and everything downstream depends on it. And here's the equity problem made physical: melanin, the pigment that makes skin darker, absorbs light. More melanin means less of the light ever reaches the blood and bounces back carrying pulse information — so the pulsatile signal is inherently weaker for darker skin, and the SNR is lower. Same heartbeat, fainter trace. That single fact is why naive rPPG methods degrade for darker skin tones, and why making PHRM equitable was the hard part.

Chapter 2: The Pipeline — From Unlock to RHR

Before diving into any single component, let's see the whole machine. PHRM has two modules that chain together: a per-video HR module that turns one 8-second clip into a heart-rate measurement (with a confidence score), and a per-day RHR module that aggregates a day's worth of those measurements into a single resting heart rate.

It all begins with a trigger you perform constantly without thinking: a screen-unlock. That event tells PHRM to quietly capture an 8-second front-camera clip. From there, each clip flows through preprocessing, a neural network, and a confidence gate — and only the clips that survive the gate feed the daily aggregation and a Kalman filter that produces the final RHR.

Two design decisions are worth flagging now, because they recur. First, PHRM throws away most of its inputs — in free-living use only about 43% of clips yield a valid measurement. That sounds wasteful until you remember the trigger is free and constant: better to discard a noisy clip than to report a wrong heart rate. Second, the whole thing is built to run on-device, on a phone, which forces every component to be computationally cheap — the reason the network is tiny and the video gets crushed to 32×32 pixels (Chapter 3).

Chapter 3: Preprocessing — Squeezing Out the Pulse

Raw phone video is a terrible input: shaky, different frame rates per device, mostly irrelevant background, and far too high-resolution to run a network on every unlock. So PHRM applies a five-step preprocessing pipeline that turns a messy clip into a clean, tiny tensor that foregrounds the pulse.

Step 5 is the quiet hero, so let's dwell on it. The face barely changes frame to frame — the pulse is a fraction of a percent of brightness. If you feed raw frames to a network, that tiny signal is buried under the huge constant of "this is a face." But subtract consecutive frames and the constant vanishes, leaving only what moved: a little motion, a little noise, and — rhythmically — the pulse. Frame-differencing is a cheap high-pass filter that throws away the boring constant and keeps the interesting change.

Chapter 4: Heart Rate as a Classification Problem

Now the core network, and the single most important modeling decision in the paper. The preprocessed 120-frame clip goes into a TSCNN — a temporal-shift convolutional network, chosen because it models time dependencies cheaply enough to run on a phone. Its first job is to output a pseudo-PPG: a clean, denoised waveform representing the pulse it extracted from the messy frame-differenced input.

Here's the elegant part. A heart rate is just the frequency of that waveform — so the network applies a Fast Fourier Transform (FFT), converting the time-domain pseudo-PPG into a frequency spectrum. A clean pulse shows up as a sharp peak at one frequency; that frequency, times 60, is the beats per minute. The FFT is the bridge from "wiggle over time" to "which heart rate."

But the network doesn't just read off the peak. It buckets the frequencies into 1-bpm bins across the biologically plausible range of 40–180 bpm, applies a softmax, and produces a full probability distribution over heart rates. The final HR is the weighted sum (the distribution's center of mass). PHRM thus reframes a regression problem ("predict the number 72") as a classification problem ("assign probability to each of the 141 possible bpm values"), trained with focal loss on the bucketed ground truth.

Why on earth turn a number-prediction into a 141-way classification? Because a distribution can express uncertainty, and a single number cannot. When the input is clean, the distribution is a sharp spike — "I'm confident it's 72." When the input is garbage (extreme motion, no face), the distribution goes flat — "I have no idea." A regression model, forced to emit one number, would confidently output a wrong 95 and you'd never know it was guessing. The flat-vs-peaked shape of this distribution is the foundation of everything in the next chapter.

# PHRM-HR head: regression reframed as classification (PyTorch-style)
def hr_head(frames):                       # frames: (120, 32, 32, 3) frame-differenced
    pseudo_ppg = tscnn(frames)                # (120,) extracted pulse waveform
    spectrum   = torch.fft.rfft(pseudo_ppg).abs()       # time -> frequency
    # keep only 40-180 bpm, one bin per bpm (141 classes)
    bins   = bucketize(spectrum, lo=40, hi=180, width=1)   # (141,)
    probs  = torch.softmax(bins, dim=-1)         # distribution over HR
    hr     = (probs * torch.arange(40, 181)).sum()   # weighted-sum HR
    return hr, probs                          # probs shape = the uncertainty signal

# trained with focal loss on the bucketed ground-truth bpm — NOT MSE on a scalar
loss = focal_loss(probs, one_hot(round(true_hr) - 40))

Chapter 5: Confidence Gating — Knowing When to Stay Quiet

We ended Chapter 4 with a distribution that's peaked when the model is sure and flat when it isn't. The confidence gate turns that shape into a yes/no decision: is this measurement trustworthy enough to keep, or should we silently discard it? In the wild — phones at weird angles, faces half-covered, people walking — most clips deserve to be discarded, and the gate is what makes "report nothing" better than "report garbage."

How do you measure "peakiness" with a single number? With entropy. Entropy quantifies how spread-out a probability distribution is: a sharp spike has low entropy (all the probability mass is in one place — little disorder), while a flat distribution has high entropy (mass spread everywhere — maximum disorder). PHRM uses the negative entropy as its confidence score — flipping the sign so that higher = more confident. A peaked, certain distribution scores high negative entropy; a flat, clueless one scores low.

Here p_k is the probability the model assigned to the k-th bpm bin, and the sum runs over all 141 bins. When one bin holds almost all the probability, this number is near zero (high confidence); when probability is smeared across many bins, it's a large negative number (low confidence). Compare it to a learned cut-off threshold: above, keep the measurement; below, discard. (The paper notes negative entropy beat alternatives like the pseudo-PPG SNR or the single max-probability — because it uses the whole distribution's shape, not one point of it.)

And here's where equity gets engineered into a threshold, not just hoped for. The cut-off wasn't chosen to maximize overall accuracy — it was searched under two hard fairness constraints: every skin-tone group's error (MAPE) had to be below 8%, and the gap between any two groups had to be below 3%. A more aggressive gate could have boosted the average while quietly failing dark-skin users; these constraints forbade that. Fairness became a literal inequality the threshold search had to satisfy.

Chapter 6: From Scattered HRs to a Daily Resting Rate

By the end of the day, PHRM has a scatter of valid heart-rate measurements — one per surviving unlock, taken at random moments: while you were calm, stressed, climbing stairs, sipping coffee. The RHR module's job is to distill this messy cloud into a single number that means "your heart rate at rest." The insight is beautifully simple and statistical, not neural.

Think about what those measurements look like as a distribution. Most of the day you're at or near rest, so the bulk of measurements cluster low. The high readings — the stairs, the stress — are the upper tail. Resting heart rate, by definition, is down at the low end of this distribution. So PHRM estimates it by taking roughly the tenth percentile of the day's valid HRs (plus a small constant bias-correction tuned once for everyone), then requires at least 20 valid measurements in a day before it'll commit to an RHR at all.

Why the tenth percentile and not, say, the minimum or the mean? The mean gets dragged up by every stressful moment — it measures your average day, not your resting state. The minimum is the noisiest single point, easily a fluke low reading. The tenth percentile threads the needle: low enough to capture rest, robust enough to ignore both a freak minimum and the entire upper tail. This is the quiet genius of passive sensing — you don't need any single measurement to be taken "at rest." Gather enough measurements and the resting state simply appears as the low percentile of the distribution. Arousal spikes can't move it because they're upper-tail outliers, filtered out by construction.

Chapter 7: The Kalman Filter — Smoothing Across Days

Each day now yields one RHR estimate — but each is itself noisy, drawn from however many measurements happened to survive that day. A day with 200 valid clips gives a tight estimate; a day with 22 gives a rough one. Plot the daily estimates raw and they jitter around the true underlying trend. PHRM's final touch smooths them with a Kalman filter.

The Kalman filter is the classic tool of optimal state estimation: how to track a quantity that changes slowly over time when each of your measurements is noisy. Its logic is a running negotiation. Each day it holds a prediction of your true RHR (essentially "probably close to yesterday's, since RHR drifts slowly"). Then today's noisy measurement arrives, and the filter blends the two — trusting the new measurement more when it's reliable (a day with many clips), trusting yesterday's accumulated estimate more when today is sparse. The output is a refined RHR that's more stable than any single day and tracks the real physiological trend.

The intuition for why this helps: your true resting heart rate doesn't teleport day to day — it drifts. So yesterday's well-supported estimate is genuine evidence about today. The Kalman filter is the principled way to combine "what we believed yesterday" with "what we measured today," weighted by how much we trust each. (We derive this trust-weighting — the Kalman gain — from scratch in the dedicated Kalman Filter lesson.)

Chapter 8: The Results — Accurate, and Equitable

A system like this is only as good as its validation, and this is where the paper earns its place. PHRM was developed on 192,353 videos from 485 participants and validated on 162,546 videos from 211 participants — the largest validation of its kind — spanning controlled lab studies and messy free-living use on participants' own phones (26 different models).

The headline accuracy numbers all clear the bar. Against reference ECG, free-living HR hit a MAPE of 6.09% at the participant level (lab: 5.65%), comfortably under the 10% industry standard (ANSI/CTA-2065). Daily RHR landed within a MAE of ~4.4 bpm of a wearable reference, under the prespecified 5 bpm target, and correlated with the wearable RHR at r = 0.87.

But the number that matters is the equity result. Across the three skin-tone groups, free-living HR MAPE was 5.04% (light), 5.12% (medium), 7.84% (dark) — all under 10%, and the differences between groups met a prespecified non-inferiority target (gaps below 5 percentage points). Of 15 state-of-the-art rPPG models tested on the same data, PHRM was the only one to stay under 10% for every skin tone. That's the whole point of the paper, in one comparison.

Finally, a validity check that the RHR is biologically real, not just internally consistent. In a statistical model, PHRM-derived RHR was independently predicted by two well-known cardiovascular risk factors: higher BMI raised RHR (+1.92 bpm per standard deviation) and better cardiovascular fitness lowered it (−1.90 bpm per SD of VO₂max). The passive smartphone signal recovers the same physiology a clinic would — which is the ultimate evidence that it's measuring something true.

Chapter 9: Connections & Cheat Sheet

PHRM is a masterclass in turning a faint, biased physical signal into a robust, equitable health measurement — not with one breakthrough, but with a chain of careful choices, each handling a specific failure mode. Let's lock it in.

The pipeline, end to end

# PHRM: one screen-unlock to a daily resting heart rate
# --- per-video HR module ---
clip   = capture_8s_video()                 # on screen-unlock, front camera
frames = stabilize(clip)                     # affine-warp to facial landmarks
frames = resample_15fps(frames)              # -> 120 frames
frames = crop_face(frames, margin=0.2)        # isolate skin
frames = resize(frames, 32, 32)              # tiny + on-device
frames = frame_difference(frames)            # cancel static face -> keep the pulse

hr, probs = tscnn_fft_head(frames)           # pseudo-PPG -> FFT -> softmax over 40-180 bpm
conf      = -entropy(probs)                   # peaked = confident
if conf > cutoff:  day_hrs.append(hr)        # GATE: discard noisy clips (~57% in the wild)

# --- per-day RHR module (needs >= 20 valid HRs) ---
raw_rhr   = percentile(day_hrs, 10) + bias    # rest = low tail, robust to arousal spikes
daily_rhr = kalman_update(prev_state, raw_rhr) # blend across days, smooth noise

Cheat sheet — every design choice and the failure it fixes

The key numbers

Where this sits in the field

• Builds on: rPPG / remote photoplethysmography and efficient temporal CNNs (TS-CAN/TSCNN). Its contribution is robustness + equity + scale, plus the uncertainty-aware classification framing and the statistical RHR aggregation.
• The estimation backbone is pure classical signal processing & estimation — FFT, entropy, percentiles, Kalman filtering — wrapped around a small neural net. A reminder that deep learning often shines brightest as one link in a well-engineered chain.
• Impact: passive, equitable cardiovascular monitoring on hardware people already own — and the authors publicly released a pretrained HR model and a large, diverse annotated smartphone video dataset.

Related lessons on Engineermaxxing

• Kalman Filter — derive the trust-weighted blending that produces the final RHR.
• Bayes Filter — the recursive estimation family the Kalman filter belongs to.
• AI Evaluation — MAPE, non-inferiority targets, and validating across subgroups.
• Transcultural ML in Music — a kindred lesson on building ML that's equitable across populations by design.

Press Teach Mode and explain why HR estimation is framed as classification — out loud, from memory. If you can connect the flat distribution to the confidence gate to the equity constraint, you own this system.