Six Numbers From Your Wrist

Chapter 1: The Signal

Every model in this lesson eats the same object: a window. A window is a contiguous slice of the sensor stream, shaped (T, C) — T timesteps by C channels. Before we can talk about architectures we have to be exact about how that object is made, because almost every subtle failure in this field is a windowing failure wearing a different hat.

From a stream to a tensor, with real arithmetic

Take the canonical setting from the UCI Human Activity Recognition dataset, which set the conventions the field still uses: sampling rate 50 Hz, window length 128 samples, overlap 50%.

Window duration is the first quantity. 128 samples at 50 Hz is 128 / 50 = 2.56 seconds. That number is not arbitrary. A comfortable walking cadence is about 110 steps per minute, which is 110 / 60 = 1.83 steps per second, so one full gait cycle (left step plus right step) takes about 1.09 seconds. A 2.56-second window therefore contains between two and three complete gait cycles — enough for the periodicity to be visible, short enough that you do not smear the boundary between "walking" and "stopped."

Overlap of 50% means the hop (or stride, or step size) is 64 samples = 1.28 seconds. Now count the windows in a ten-minute recording:

And the tensor those windows form:

Ten minutes of a person walking around is 1.43 megabytes of model input. Hold on to that, because it explains why the pretraining datasets in this field can reach hundreds of thousands of windows from a few hundred hours of recording.

Different papers pick different points on the same axis. Here are the four you will meet in this lesson, all of which are just choices of (rate, T):

Units, ranges, and why the six channels are not comparable

Before any arithmetic, know what the numbers mean physically. The two triples are different quantities in different units with different dynamic ranges, and treating them as six interchangeable numbers is the most common beginner error in this domain.

Look at the resting column and the difference is stark: the accelerometer's resting state is a large constant, while the gyroscope's is zero. That single asymmetry has consequences everywhere.

Consequence 1: the mean means different things. The mean of an accelerometer axis is essentially the gravity projection, an orientation measurement. The mean of a gyroscope axis is essentially the net rotation rate, which for any activity that returns to its starting pose is near zero. The same statistic, computed identically, is informative for one triple and near-useless for the other.

Consequence 2: naive scaling is wrong in both directions. Divide everything by its global maximum and the gyroscope's ordinary ±5 range gets crushed into a seventh of the axis because of rare ±35 flicks. Normalise per-channel over a window and you destroy the gravity information that distinguishes sitting from standing.

Consequence 3: the fix is per-group normalisation, and both papers do it. LIMU-BERT normalises inside its embedding layer; IMU2CLIP's very first operation is GroupNorm(2, 6) — two groups over six channels, which is exactly accelerometer and gyroscope treated separately. The paper says so explicitly: it normalises "the Accelerometer (3D) and the Gyroscope (3D) signals independently."

Why groups of three rather than six separate channels? Because the three accelerometer axes are not independent — they are components of one vector, and their relative magnitudes encode device orientation. Normalising each axis independently would rescale the components differently and destroy the vector's direction. Grouping preserves the geometry within each triple while equalising the two triples against each other. It is a small design decision that is correct for a reason from physics, not from optimisation.

Nyquist, and why 20 Hz is a real decision

A sampling rate of f_s can faithfully represent frequency content only up to f_s/2, the Nyquist frequency. Anything faster does not simply vanish — it folds back and impersonates a lower frequency, permanently, in a way no downstream model can undo.

Work an example. Suppose a runner's foot strikes produce a sharp impulse whose energy extends to 18 Hz. Sample at 20 Hz. Nyquist is 10 Hz. The 18 Hz component folds to |18 − 20| = 2 Hz. Your model now sees a spurious 2 Hz oscillation sitting right on top of the real cadence band, indistinguishable from it.

So why does LIMU-BERT use 20 Hz? Because the thing it needs to classify — walking, sitting, standing, lying, stairs — lives almost entirely below 10 Hz. Cadence is 1–3 Hz; the first few harmonics reach 6–9 Hz. Sampling at 20 Hz captures all of that, quarters the sequence length relative to 50 Hz, and quarters the compute. It is a deliberate trade: lose impact transients, gain a model that runs on a phone. IMU2CLIP, chasing fine-grained egocentric semantics ("opening a drawer," "wiping a counter"), goes the other way and resamples Ego4D to 200 Hz.

One more number worth having. The frequency resolution of a windowed spectrum is f_s/T. For the UCI window that is 50 / 128 = 0.3906 Hz per bin. So a walker at 1.83 Hz and a walker at 2.0 Hz land in adjacent bins — distinguishable, barely. Halve the window to 64 samples and the bin width doubles to 0.78 Hz and they merge. Window length is spectral resolution; there is no free lunch.

A decision procedure for T and hop

Since every paper picks different values, it helps to have a procedure rather than a table to copy. Four constraints, applied in order.

Constraint 1: the window must contain at least two periods of the slowest thing you care about. For gait at 1.5 Hz, one cycle is 0.67 s, so you need T ≥ 1.33 s. For a breathing-related signal at 0.25 Hz you would need 8 s. This is the floor.

Constraint 2: the window must be shorter than the shortest bout you need to detect. If you want to catch a five-second stair climb, a ten-second window can never contain a pure example of it. This is the ceiling, and Constraint 1 and Constraint 2 together often leave a narrow band.

Constraint 3: the frequency resolution must separate the classes you care about. Bin width is f_s/T samples, or equivalently 1/T_seconds in hertz. To distinguish a 1.8 Hz walker from a 2.0 Hz walker you need bins narrower than 0.2 Hz, so

Which is a striking result: the canonical 2.56-second window cannot resolve those two cadences spectrally. It works because classifiers use amplitude and shape features too, not because the spectrum separates them. If your task genuinely depends on fine cadence discrimination, 2.56 s is the wrong window and no model will fix it.

Constraint 4: the hop sets your latency and your dataset size. Latency to a decision is at least T, and you emit a decision every hop seconds. Halving the hop doubles the number of windows — which sounds like more data and is not, because the extra windows are 50%-correlated with their neighbours. It buys temporal resolution in the output, and buys no new information.

Gravity is not a bug, and separating it costs you

Here is the single most common surprise for people meeting accelerometers. An accelerometer at rest does not read zero. It reads 9.81 m/s², pointing up.

The reason is that an accelerometer measures proper acceleration — the force on its proof mass relative to free fall — not coordinate acceleration. A device sitting on a table is being pushed upward by the table at exactly 1 g, so it reads 1 g upward. A device in free fall reads zero, which is why your phone can detect that you dropped it.

Consequently the measurement is always a sum:

where g_projected is the 9.81 m/s² gravity vector expressed in the device's own rotating frame. Tilt the phone and gravity redistributes across the axes without changing its length. Concretely: lie the phone flat and you get a = (0, 0, 9.81). Tilt it 30 degrees about the y-axis and you get

Check the length: √(8.496² + 4.905²) = √(72.18 + 24.06) = √96.24 = 9.81. The vector rotated; nothing moved.

This is why orientation is confounded with activity. A phone in a trouser pocket versus a jacket pocket produces completely different per-axis signals for identical walking, because the gravity vector lands on different axes. The classical fix is the magnitude ‖a‖ = √(a_x² + a_y² + a_z²), which is rotation-invariant — and which throws away all directional information. The modern fix is to give the model all six channels and enough data to learn the invariance itself.

The low-pass split, worked by hand

The standard way to separate the two terms exploits a timescale gap: gravity's direction changes slowly (you reorient over hundreds of milliseconds), while linear acceleration changes fast. So estimate gravity with a low-pass filter and subtract.

The cheapest low-pass is a first-order exponential moving average:

Pick the cutoff first, then derive α. For a first-order filter the −3 dB cutoff f_c relates to the time constant τ by f_c = 1/(2πτ). Choosing f_c = 0.3 Hz (well below cadence, well above the rate at which you reorient a limb):

Now the part nobody tells you: this filter steals your signal. Suppose the device is at rest with ĝ = 9.81, and then a genuine +2.00 m/s² linear acceleration begins and holds. The measured value jumps to 11.81 and stays there. Track the estimate, sample by sample, at 50 Hz:

After a tenth of a second, 17% of the real acceleration has been absorbed into the "gravity" estimate and subtracted away. After a full second (50 samples) the filter has swallowed almost all of it: any acceleration you sustain for longer than a few τ is, by construction, indistinguishable from a slow reorientation. That is not a bug in the filter, it is the definition of the filter. It is also exactly why every hand-built preprocessing chain has a knob that someone tuned once, on one dataset, and never revisited.

Rotation invariance, derived

The gravity problem generalises. If the device is mounted differently, every accelerometer reading is multiplied by some rotation matrix R:

A classifier that reads a_x, a_y, a_z separately sees three completely different numbers. But the magnitude is untouched, and that follows in one line:

Verify numerically with the tilt example. Flat: a = (0, 0, 9.81), magnitude 9.81. Tilted 30°: a′ = (4.905, 0, 8.496), and

Identical, as promised. So ‖a‖ is a genuinely mount-invariant feature. What does it cost? Three numbers became one, so you have discarded two degrees of freedom — specifically, all directional information. You can no longer distinguish "the device accelerated upward" from "the device accelerated sideways," which means you cannot distinguish walking up stairs from walking down them, since both produce the same magnitude profile with opposite sign on the vertical axis.

Two other rotation-invariant quantities are worth knowing because they preserve more:

The inner product survives rotation for the same reason: ⟨Ra, Rω⟩ = a^TR^TRω = ⟨a, ω⟩. So a feature set built from ‖a‖, ‖ω‖, and ⟨a, ω⟩ is fully invariant to how the device is mounted, at the price of collapsing six channels into three. This is the classical trade, and the learned alternative — keep all six and let the model see enough mounting variety to build its own invariance — is exactly what population-scale pretraining buys.

Where the energy lives: a frequency budget

Windowing decisions become obvious once you know where the signal energy sits. For a 128-sample window at 50 Hz, the bin width is 0.3906 Hz, and here is where the first few bins land for a walker at 1.83 Hz:

Notice the second harmonic row. Human gait is not perfectly symmetric — your left step and your right step differ slightly — and that asymmetry appears as energy at twice the cadence. It is one of the strongest individual signatures in the whole signal, which makes it useful for identifying people and dangerous for the leakage reasons in Chapter 7.

Resampling, and the order of operations that everyone gets wrong

Every real pipeline resamples, because devices report at whatever rate the driver felt like. IMU2CLIP's preprocessing forces all Ego4D IMU to 200 Hz using torchaudio.functional.resample, and the choice of function matters.

Downsampling by simply keeping every N-th sample is decimation without filtering, and it is a bug. Go from 200 Hz to 20 Hz by keeping every tenth sample and every frequency component between 10 Hz and 100 Hz folds down into your 0–10 Hz band, permanently. The correct order is: low-pass filter to below the new Nyquist frequency first, then decimate. A proper resampler does both; a slice operation does not.

python — resampling, right and wrongimport numpy as np
from scipy.signal import decimate, resample_poly

# WRONG — aliases every component above 10 Hz into the band you keep
x20_bad = x200[::10]

# RIGHT — anti-alias filter, then decimate. scipy does both.
x20 = decimate(x200, 10, ftype='fir', zero_phase=True)

# For non-integer rate changes (e.g. 198 Hz device → 200 Hz target)
x200_from_198 = resample_poly(x198, up=100, down=99)

# And ALWAYS resample before windowing, never after: a window boundary
# placed on the old grid does not land on the new grid, so your window
# lengths drift by a sample here and there and your tensor stops stacking.

Two more traps that live in the windowing code

Trap 1: the label of a window is a fiction. A 2.56-second window that straddles the moment you sat down contains 1.4 s of walking and 1.16 s of sitting. What is its label? LIMU-BERT's released preprocessing handles this bluntly and correctly: its merge_dataset function in all mode keeps a window only if every sample inside it carries the same label, and discards the rest. That is honest, and it is why the released UCI dataset yields only 2,088 labelled windows.

Trap 2: overlap manufactures fake data. With 50% overlap, consecutive windows share 64 of their 128 samples. If you then shuffle all windows and split randomly into train and test, you have put half of a test window into the training set. Chapter 7 quantifies exactly how bad this gets; for now, just notice that the bug is created here, in the windowing function, two hundred lines before the model.

Sliding windows versus event windows

Everything above assumes a sliding window: fixed length, fixed hop, running forever. It is not the only option, and the alternative is worth knowing because it is what production systems often do.

An event window is triggered by something in the signal — a threshold crossing, a hardware significant-motion interrupt, a detected heel strike — and extends for a fixed duration or until some end condition. Compare them honestly:

Row three is the underrated one. A sliding window starts wherever the clock says, so the same footstep appears at sample 12 in one window and sample 76 in the next. The model must learn translation invariance from data. An event window anchored to heel strike puts the same physical moment at the same index every time, which removes that burden entirely — and this is precisely why gait-analysis systems in clinical settings use event windows while machine-learning pipelines use sliding ones.

Row six is why every model in this lesson uses sliding windows anyway. Self-supervised pretraining wants an unbiased sample of what the sensor sees, including the boring parts. Trigger on motion and your pretraining corpus contains only motion, so the encoder never learns what stillness looks like — and stillness is most of a day.

The preprocessing chain, in code

python — from stream to model-ready tensorimport numpy as np

FS, T, HOP = 50, 128, 64          # 2.56 s windows, 50% overlap

def windows(stream, labels):
    # stream: (N, 6) float32 — ax, ay, az, gx, gy, gz
    n = (len(stream) - T) // HOP + 1
    X = np.stack([stream[i*HOP : i*HOP+T] for i in range(n)])   # (n, 128, 6)
    L = np.stack([labels[i*HOP : i*HOP+T] for i in range(n)])   # (n, 128)
    # keep only windows with ONE label throughout — no straddlers
    pure = np.array([len(np.unique(l)) == 1 for l in L])
    return X[pure], L[pure, 0]

def split_gravity(win, fs=FS, fc=0.3):
    # win: (T, 3) accelerometer only
    alpha = np.exp(-(1.0/fs) / (1.0 / (2*np.pi*fc)))     # 0.9630 at 50 Hz, 0.3 Hz
    g = win[0].copy()
    out = np.empty_like(win)
    for t in range(len(win)):
        g = alpha * g + (1 - alpha) * win[t]
        out[t] = win[t] - g                          # linear component
    return out, g                                     # (T,3) and final gravity dir

Notice what the second function is: a one-pole IIR filter written as a Python loop, with a hard-coded 0.3, applied identically to every subject, every device, every body position. It is a strong prior about the world baked in before the model sees anything. Every era of this field is partly a story about which of these hard-coded priors we managed to give back to the data.

Chapter 2: Features, Trees, and the Ceiling

For roughly a decade, human activity recognition had a recipe so stable you could write it on an index card. Window the stream. Compute a fixed list of statistics per window per axis. Concatenate into a feature vector. Train a tree ensemble. Ship.

It is worth taking this pipeline seriously, and computing it by hand, for two reasons. First, it works — it is still in production in devices you own. Second, the precise way it fails is the argument for everything that came after.

The feature vector, computed on eight numbers

Take a toy window of 8 samples. In practice T is 128; 8 is small enough to do on paper and large enough for every formula to be real. Here is the vertical accelerometer channel during walking, in m/s²:

Feature 1, the mean. Sum: 9.9 + 12.4 = 22.3; + 8.1 = 30.4; + 6.7 = 37.1; + 10.2 = 47.3; + 13.1 = 60.4; + 7.9 = 68.3; + 6.9 = 75.2. Divide by 8:

Notice that this is essentially gravity. The mean of an accelerometer axis over a window is, to first order, the projection of the gravity vector onto that axis — which is to say the mean is an orientation feature, not a motion feature. That single fact explains why so many classical HAR papers report that "mean" is the most important feature for distinguishing sitting from standing (different orientations) and almost useless for distinguishing walking from running (same orientation).

Feature 2, the standard deviation. Deviations from the mean: 0.5, 3.0, −1.3, −2.7, 0.8, 3.7, −1.5, −2.5. Squares: 0.25, 9.00, 1.69, 7.29, 0.64, 13.69, 2.25, 6.25. Sum = 41.06.

This one is a motion feature. It measures how violently the axis is being shaken, and it is the single most powerful classical feature for separating stationary from ambulatory activities.

Feature 3, the energy. Defined as the mean of the squares. Direct computation: 98.01, 153.76, 65.61, 44.89, 104.04, 171.61, 62.41, 47.61, summing to 747.94.

Sanity-check the relationship E = σ² + μ² = 5.1325 + 88.36 = 93.4925. It agrees, which tells you something uncomfortable: energy is not an independent feature. Given the mean and the variance you already have it. Classical pipelines are full of these near-duplicates, and tree ensembles tolerate them by splitting the importance across correlated columns — which is also why feature-importance plots from these models are so often misleading.

Feature 4, the cross-axis correlation. Let the paired x-axis samples be

Its mean is (1.2 + 2.8 + 0.4 − 0.6 + 1.5 + 3.1 + 0.2 − 0.4)/8 = 8.2/8 = 1.025. Deviations: 0.175, 1.775, −0.625, −1.625, 0.475, 2.075, −0.825, −1.425. Now the covariance, term by term against the a_z deviations:

And σ_x: squared deviations sum to 13.455, so σ_x² = 1.6819 and σ_x = 1.2969. Then

Almost exactly 1. The two axes are moving in lockstep, because the device orientation is fixed and the motion is essentially one-dimensional (up and down along the body's vertical). This is a genuinely informative feature — it drops sharply when the device tumbles in a pocket — and it is also a warning: with r = 0.9985 the x-axis is contributing almost no independent information for this window, so a model that treats the six channels as six independent evidence sources is fooling itself.

Do this for all 6 channels × roughly 8–20 statistics and you get the classical feature vector: typically 100 to 600 dimensions. Feed it to a random forest. Done.

The algorithm in your pocket, worked by hand

Before the feature vector, there was something even simpler, and it is worth doing because it ships in every phone you have owned: the step counter. It is the purest example of the classical philosophy, and its failure modes are the classical failure modes in miniature.

The algorithm is three rules. Compute the accelerometer magnitude. Detect local maxima that exceed a threshold. Reject any peak that arrives too soon after the last one. That is it.

Run it on our eight samples of a_z at 50 Hz, using magnitude as a stand-in:

Sample 1 (12.4). It exceeds 11.0, and it is greater than both neighbours (9.9 and 8.1). Peak. Time = 1/50 = 0.02 s. Steps = 1.

Sample 5 (13.1). Exceeds 11.0, greater than 10.2 and 7.9. Peak candidate at t = 5/50 = 0.10 s. But the refractory check: 0.10 − 0.02 = 0.08 s, which is less than 0.25 s. Rejected. Steps stays at 1.

So the counter reports one step in 0.16 seconds of data. The refractory period exists to stop a single footfall's ringing from counting as several steps — and here it has just discarded a genuine second step, because our toy signal oscillates faster than a real gait. Set the refractory period to 0.25 s and you have declared, in code, that nobody takes more than four steps per second:

That is above sprint cadence, so it is a safe choice. Now look at the threshold. θ = 11.0 sits 1.19 m/s² above resting gravity. Apply subject B's amplitude from earlier — her peaks reach only 11.6 rather than 13.1 — and she still clears it. Attenuate a little further, as a phone in a loose bag rather than a trouser pocket would, and her peaks fall below 11.0 and the step counter reports zero steps while she walks.

How the tree actually picks its threshold

It is worth doing one split by hand, because it makes the failure mode in the next section unavoidable rather than merely plausible.

A decision tree chooses the feature and threshold that maximise information gain: the reduction in label entropy from splitting. Entropy of a set with class proportions p_i is

Take 20 windows: 10 walking, 10 standing. Before any split, the classes are balanced, so

Now try the split σ_z > 1.8. Suppose it sends 11 windows right (9 walking, 2 standing) and 9 left (1 walking, 8 standing). Compute each child's entropy.

Right child, proportions 9/11 = 0.8182 and 2/11 = 0.1818. Using log₂(0.8182) = −0.2895 and log₂(0.1818) = −2.4594:

Left child, proportions 1/9 = 0.1111 and 8/9 = 0.8889. With log₂(0.1111) = −3.1699 and log₂(0.8889) = −0.1699:

Weight the children by size and subtract:

The tree keeps this split because 0.397 bits is a large gain for one comparison. And now look closely at what it has committed to: the literal number 1.8, in the units of the training data, frozen into the model.

Where the ceiling is, and why

Now the failure. It has one dominant cause, and it is not model capacity.

Take the same walking activity, performed by a different person. Subject B is heavier, has a slower cadence, and wears the device slightly differently. Her vertical channel over 8 samples might read:

Mean: 9.7 + 11.2 + 8.9 + 8.1 + 9.8 + 11.6 + 8.7 + 8.2 = 76.2, so μ = 9.525. Deviations: 0.175, 1.675, −0.625, −1.425, 0.275, 2.075, −0.825, −1.325. Squares: 0.0306, 2.8056, 0.3906, 2.0306, 0.0756, 4.3056, 0.6806, 1.7556, summing to 12.075.

Same activity. Same sensor. Same window length. And the most important classical feature has moved from 2.2655 to 1.2286 — a 46% drop. Now suppose the forest learned, from subject A and friends, a split of the form "σ_z > 1.8 → ambulatory, else stationary." Subject B's genuine walking lands on the wrong side of that threshold and is classified as standing.

Look at the scale of the labelled data in the four datasets LIMU-BERT ships preprocessing for. These were the field's standard benchmarks:

Nine subjects. Thirty subjects. Two thousand windows. For comparison, ImageNet has 1.28 million labelled images across a thousand classes, and that was considered small by 2020. A model trained on 30 people cannot learn what is invariant across humans, because it has seen thirty humans.

And note the third column of the released config for the Shoaib dataset: it carries a position_label with five values — left pocket, right pocket, wrist, upper arm, belt. Five mounting positions, each of which rotates the gravity vector differently and attenuates the motion differently. Combine that with 10 subjects and you have 50 distinct signal regimes represented by 10,500 windows, roughly 210 windows per regime.

Deep nets came first, and they were not enough

There is a step between "hand features plus trees" and "self-supervision" that is easy to skip, and skipping it makes the story wrong. Around 2015 to 2019 people replaced the hand-designed features with convolutional networks on raw windows. It worked, it beat the feature pipelines, and it did not break the ceiling. Understanding why is the whole argument for Chapter 3.

The released LIMU-BERT repository ships these as its baselines — a benchmark.py that runs DCNN, DeepSense, and a recurrent GRU model — so we can read the architecture rather than guess it. The convolutional baseline stacks three 2-D convolutions with kernels of shape (5,1), (5,1), and (3,1), each followed by batch normalisation and a (2,1) max-pool, and then flattens into a 400-unit dense layer before the classifier.

Trace a (120, 6) window through it, treating the window as a single-channel image of height 120 and width 6, so the kernels slide along time only and never mix channels:

Look at the last box. The flatten-to-dense step alone carries 576,400 parameters, which is more than eleven times the entire LIMU-BERT encoder we will build in Chapter 3. And here is the part that matters: every one of them is trained with labels.

Now recall the label budget. The released UCI dataset yields 2,088 labelled windows. Training 576,400 parameters on 2,088 examples is a ratio of 276 parameters per example. The network does not need to generalise; it has more than enough capacity to memorise the training set outright, and the only things standing between it and doing exactly that are batch norm, early stopping, and luck.

Reading a confusion matrix, by hand

One more piece of arithmetic before we leave the classical era, because it is the tool you will use to evaluate everything that comes after. Suppose a four-class model produces this confusion matrix on 100 test windows — rows are truth, columns are prediction:

Accuracy is the diagonal over the total: (45 + 8 + 4 + 2)/100 = 59.0%. Respectable-sounding. Now compute per-class precision, recall, and F1, where precision is TP/(TP + FP) reading down a column and recall is TP/(TP + FN) reading across a row.

Head motion: TP = 45. False positives are the rest of column one: 14 + 10 + 6 = 30. False negatives are the rest of row one: 5.

Walking: TP = 8, FP = 3 + 1 + 1 = 5, FN = 17.

Manipulation: TP = 4, FP = 1 + 2 + 1 = 4, FN = 11.

Stationary: TP = 2, FP = 1 + 1 + 0 = 2, FN = 8.

Average the four F1 scores equally and you get macro-F1:

Weight them by support instead and you get weighted-F1: (50 × 0.720 + 25 × 0.4211 + 15 × 0.3478 + 10 × 0.2857)/100 = 54.60/100 = 54.6.

Three numbers for one model: 59.0 accuracy, 54.6 weighted-F1, 44.4 macro-F1. They differ by fifteen points, and which one you report determines what story gets told. Accuracy and weighted-F1 are both dominated by the 50-window majority class, where the model is genuinely good. Macro-F1 gives equal voice to "stationary," where it recovers 2 windows out of 10.

Three fixes people tried, and why each is partial

Fix 1: normalise the features per subject. Z-score each feature using that subject's own mean and standard deviation. This genuinely helps, and it requires a calibration period per user before the model works — which is a product disaster and a cold-start disaster. It also destroys real between-subject signal (a frail user really does shake less).

Fix 2: augment with rotations. Apply random 3D rotations to the accelerometer triple during training so the model cannot rely on absolute axis assignment. Real and useful. But rotation is only one of the nuisance factors; it does nothing about cadence, amplitude, sensor bias, or sampling-clock drift, and it cannot be applied to hand-crafted features after the fact — you have to go back to the raw signal, at which point you may as well learn the features.

Fix 3: collect more labelled data. The honest answer, and the one that does not scale. LIMU-BERT's own framing is exactly this: the whole point of the paper is "unleashing the potential of unlabeled data," because the labelled pile stopped growing while the unlabelled pile grew without bound. Every phone in the world produces IMU data continuously. Almost none of it is annotated, and none of it ever will be.

Click through to "learned embedding" and notice what changes. The axes are no longer mean and standard deviation — they are two coordinates of a representation trained on unlabelled data from both subject groups. The classes have not become more separable by magic; they have become separable in a coordinate system where the subject difference is a direction the encoder chose to compress. That is a claim the next chapter has to earn.

python — the entire classical pipeline, honestlyimport numpy as np
from sklearn.ensemble import RandomForestClassifier

def features(win):
    # win: (128, 6) → a fixed 60-dim vector
    f = []
    for c in range(6):
        x = win[:, c]
        f += [x.mean(), x.std(), (x**2).mean(),          # mean, std, energy
              x.min(), x.max(), np.median(x),
              np.abs(np.diff(np.sign(x - x.mean()))).sum()/2]   # zero crossings
        S = np.abs(np.fft.rfft(x - x.mean()))
        f += [S.argmax() * 50/128, S.max(), (S/S.sum() * np.log(S/S.sum() + 1e-9)).sum()]
    for i, j in [(0,1), (0,2), (1,2)]:
        f.append(np.corrcoef(win[:, i], win[:, j])[0, 1])   # axis correlations
    return np.array(f)

X = np.stack([features(w) for w in train_windows])    # (N, 60)
clf = RandomForestClassifier(n_estimators=300).fit(X, y)

# Everything above the fit() line was written by a human who guessed.
# Every guess is a hyperparameter that never gets retuned when the
# device, the population, or the mounting position changes.

What survived from this era, and should have

It would be a mistake to leave Chapter 2 thinking the classical era was simply wrong. Four of its contributions are load-bearing in every modern system in this lesson.

The windowing convention. Fifty hertz, 128 samples, 50% overlap came from this era, and every architecture since consumes windows because of it. LIMU-BERT's (120, 6) and IMU2CLIP's (6, 1000) are both descendants of a decision made when the models were decision trees.

The magnitude feature. ‖a‖ is still the right first thing to compute, still the basis of every step counter, and still the only exactly mount-invariant quantity available. LSM's per-minute aggregates in Chapter 6 are computed from it.

Subject-wise evaluation. The discipline of holding out people rather than rows was established here, out of hard experience, and Chapter 7 shows it is more necessary now than it was then — a 600,000-parameter network memorises a subject far more eagerly than a shallow tree does.

Reporting F1 alongside accuracy. This convention exists because the classical community kept discovering that their 90%-accurate models could not detect the class anyone cared about. That lesson transferred intact.

Read the last two rows against the five above them. What got replaced was the parts a human chose: which statistics to compute, and which categories exist. What survived was the parts that describe the problem: the shape of the data, the geometry of the sensor, and the discipline of honest evaluation. That is a good heuristic for any field in transition — the methodology usually outlives the method.

Chapter 3: Masked Reconstruction

Here is the idea that broke the labelled-data ceiling, and it is one sentence long. Delete part of the window and make the model put it back.

No labels are required, because the answer is the data you deleted. Every unlabelled window in the world becomes a training example. And to fill in the hole convincingly, the model has no choice but to learn how motion actually behaves: that acceleration is smooth, that gait is periodic, that gravity's direction persists, that the gyroscope and accelerometer are coupled.

LIMU-BERT is the cleanest realisation of this for inertial data. It borrows BERT's masked-language-model recipe, and it changes two things in ways that matter enormously. We will derive both.

The objective, on a window you can hold in your head

Take a 12-sample window at 20 Hz — 0.6 seconds — of a single accelerometer channel during walking at 1.5 Hz cadence. Model it as a clean sinusoid on a gravity offset:

Evaluating for t = 0 … 11, the phase step is 2π(1.5)/20 = 0.47124 radians per sample:

Now mask t = 4 and t = 5. In LIMU-BERT's implementation, masking literally means writing zeros into those rows of the input tensor — there is no learned [MASK] token, because the input is continuous, not a vocabulary index. The model sees ten real values and two zeros, and must output predictions at exactly the masked positions.

The loss is mean squared error over masked positions only:

Predictor 1: the window mean. A model that has learned nothing except "output something plausible" predicts the average of what it can see. The ten visible samples sum to 95.021, so the mean is 9.5021.

Predictor 2: linear interpolation. A model that has learned only "acceleration is continuous" draws a straight line from x₃ = 12.773 to x₆ = 10.737 and reads off the midpoint, 11.755, for both masked slots.

Predictor 3: a model that knows the gait is a 1.5 Hz sinusoid. It outputs 12.663 and 11.931 and scores essentially zero.

Change one: mask spans, not points

Now the first departure from BERT, and it is the important one. Look again at the numbers. Masking two adjacent samples at 20 Hz means hiding 0.1 seconds — and simple linear interpolation already gets the loss down to 0.4277 without understanding anything. Mask single scattered samples, as BERT does with tokens, and interpolation would do even better. The task would be nearly free, the gradients would be nearly zero, and the encoder would learn a smoothing filter.

Text does not have this problem because adjacent words are not numerically close. "The cat sat on the ___" cannot be solved by averaging "the" and nothing. Continuous sensor signals absolutely can.

So LIMU-BERT masks spans. Its released mask configuration is exactly four numbers:

config/mask.json — the whole masking policy{
    "mask_ratio": 0.15,     # fraction of timesteps to predict
    "mask_alpha": 6,
    "max_gram": 10,        # longest contiguous span
    "mask_prob": 0.8       # zero it (else: replace with random values)
}

The span length is drawn from a truncated geometric distribution. In the released span_mask function, span lengths n = 1 … 10 get unnormalised weights p(1 − p)ⁿ⁻¹ with p = 0.2, then normalised. Compute them:

Dividing through gives P(n = 1) = 0.2/0.8926258 = 0.2241, P(2) = 0.1793, P(3) = 0.1434, and so on down to P(10) = 0.0301. The expected span length is the weighted sum:

So the average hole is about 3.8 samples wide. At the 20 Hz rate LIMU-BERT uses, that is 0.19 seconds of motion erased in one contiguous block — a meaningful fraction of a gait cycle, not a pinprick.

Does span masking actually make the task harder? Measure it on our 12-sample window. Mask t = 2 through t = 9 — eight contiguous samples — and let linear interpolation try again, now spanning from x₁ = 11.172 to x₁₀ = 6.810. The per-sample decrement is (6.810 − 11.172)/9 = −0.48467.

The interpolation baseline's loss went from 0.4277 to 3.7703 — an 8.8× increase — purely from making the holes contiguous. That gap is the room the encoder has to earn its keep. Span masking is not a detail; it is what makes the objective non-trivial for a continuous signal.

Change two: masking means zeros, sometimes noise

The mask_prob of 0.8 governs a second detail. Reading the released Preprocess4Mask: with probability 0.8, the selected span is overwritten with zeros; otherwise, it may be overwritten with uniform random values instead.

This is inherited from BERT's 80/10/10 rule and it serves the same purpose. If masking were always zeros, the model would learn a shortcut — "detect exact zeros, that is where I must predict" — and its representations would be useless at inference time, when nothing is zeroed. Occasionally substituting random values forces the encoder to reason about every position's plausibility rather than pattern-matching on the mask marker. It is a small line of code that protects the whole point of the exercise.

The architecture, shape by shape

Now walk a real batch through LIMU-BERT's released base_v1 configuration: feature_num 6, hidden 72, hidden_ff 144, n_layers 4, n_heads 4, seq_len 120.

Step 1, the factorised embedding. Input is (B, 120, 6). A single linear layer lifts 6 → 72. That is 6 × 72 + 72 = 504 parameters. The paper calls this "factorised" because unlike BERT there is no vocabulary table — the embedding is a projection of continuous values, which is why the whole model can be so small.

Step 2, position. A learned embedding table of shape (120, 72) is added: 120 × 72 = 8,640 parameters. Note that this is 17× the size of the input projection. In a model this narrow, telling the network where in the window a sample sits costs more parameters than reading the sample itself — and that is appropriate, because "the fourth sample of the stride" is exactly the kind of thing gait structure depends on.

Step 3, four transformer layers — sharing one set of weights. This is the detail people miss. Read the released Transformer.forward: there is one attention block, one projection, one feed-forward, and the loop applies them n_layers times. It is ALBERT-style cross-layer parameter sharing. Four layers of depth, one layer of parameters.

Count them. Q, K, V projections: 3 × (72×72 + 72) = 15,768. Output projection: 5,256. Feed-forward 72 → 144 → 72: (72×144 + 144) + (144×72 + 72) = 10,512 + 10,440 = 20,952. Two LayerNorms: 288. Total per shared block: 42,264 parameters.

Attention head geometry: 72 hidden / 4 heads = 18 dimensions per head. Each head computes a 120 × 120 attention matrix over timesteps. With four heads and four applications, that is sixteen chances for the model to discover a different temporal relationship — one head can track the half-stride offset, another the gravity baseline, another the gyroscope-accelerometer coupling.

Step 4, the reconstruction head. Gather only the masked positions. With mask ratio 0.15 on 120 timesteps, that is round(120 × 0.15) = 18 positions, so the gathered tensor is (B, 18, 72). Then Linear 72→72 with GELU, LayerNorm, and a final decoder Linear 72→6 producing (B, 18, 6). MSE against the true values at those positions.

Add it up. The encoder — the part that survives pretraining and ships — is 504 + 8,640 + 144 + 42,264 = 51,552 parameters. At float32 that is 206 kB. Quantised to int8 it is about 52 kB. For scale: a single 3×3 convolution layer with 256 input and 256 output channels has 590,000 parameters, eleven times more than this entire model.

Why MSE, and what it silently assumes

Mean squared error is such a default choice that it is worth asking what it commits you to. Start from maximum likelihood. If you model the true value at a masked position as the prediction plus Gaussian noise of fixed variance σ²,

then the negative log-likelihood over the masked set is

which is MSE up to a constant scale. So minimising MSE is maximum-likelihood estimation under a fixed-variance Gaussian. Two assumptions come along for free, and both are false in interesting ways.

Assumption A: the noise variance is the same everywhere. It is not. Reconstructing a masked sample in the middle of quiet standing is easy and the residual is tiny; reconstructing one in the middle of a heel strike is hard and the residual is large. MSE therefore weights the vigorous parts of the signal far more heavily — not because they matter more, but because they are numerically bigger. A model minimising MSE will happily be sloppy about the whole class "sitting" in exchange for being slightly better about "running."

Assumption B: the six channels are commensurable. They are not, and this one is severe. Accelerometer values live around ±20 m/s²; gyroscope values around ±5 rad/s in ordinary motion. Compute the loss contribution if the model is 10% wrong on each:

The same relative error contributes 16 times more loss on the accelerometer. Unnormalised, an MSE objective over raw channels is roughly an accelerometer-only objective with the gyroscope along for the ride. This is exactly why both architectures in this lesson normalise before anything else — LIMU-BERT with LayerNorm inside its embedding, IMU2CLIP with a GroupNorm(2, 6) that treats accelerometer and gyroscope as separate groups. That layer is not hygiene; it is what makes the loss mean what you think it means.

What one attention head can see

Make the attention concrete. Each head works in 18 dimensions (72 hidden / 4 heads) and produces, for every timestep, a distribution over all 120 timesteps. Consider a masked position at t = 60 in a 6-second window of walking at 1.5 Hz, sampled at 20 Hz. One gait cycle is 20/1.5 = 13.33 samples.

The most useful thing a head could possibly do is attend to the same phase of neighbouring cycles — that is, to t = 60 ± 13.33, ±26.67, ±40, and so on. Suppose a head produces the raw scores

The scaled dot-product divides by √d_head = √18 = 4.243 before the softmax, so those scores become 0.754, 0.259, 0.707, 0.495 and the rest zero. Exponentiate: 2.125, 1.296, 2.028, 1.640, and 116 entries at 1.000. The denominator is 2.125 + 1.296 + 2.028 + 1.640 + 116 = 123.089, giving

Two things fall out of that arithmetic. First, the "sharp" attention is only about 2.1 times the background weight — softmax over 120 positions is a much flatter operation than the pictures in most explainers suggest, and getting genuinely peaked attention requires large score gaps. Second, the division by √18 is doing real work: without it the scores would be 3.2 and 3.0 rather than 0.754 and 0.707, the exponentials would be 24.5 and 20.1, and the peaks would carry roughly 30% of the mass between them. The scaling factor is what keeps gradients from saturating early in training, and its size depends on the head width, which depends on the hidden size, which was chosen to be 72 so the model would fit on a phone. Every constant in this architecture is downstream of the power budget.

The cost of sharing weights across layers

Parameter sharing is not free, and it is worth naming the trade. In a standard transformer, layer 1 can learn "smooth the signal" while layer 4 learns "compare gait phases," because they hold different weights. With ALBERT-style sharing, the same function is applied four times, so the model must find one operator whose repeated application is useful — something closer to a fixed-point iteration than a pipeline.

What you buy is a 4× reduction in parameter memory (42,264 shared rather than 169,056 distinct) with no reduction in compute, since all four applications still run. On a wearable that is exactly the right side of the trade: SRAM is the scarce resource and the arithmetic was never the bottleneck. On a server it would be a strange choice.

That last row is not a small consideration. Pretraining on four public datasets totalling roughly 26,000 windows, a model with four independent layers has a great deal of freedom to memorise. Sharing is a regulariser as much as a compression.

Two-phase training, and why the freeze matters

python — the two phases, following the released scripts# Phase 1 — self-supervised. NO labels used anywhere.
#   pretrain.py: 3200 epochs, batch 128, lr 1e-3, Adam with 10% warmup
for batch in unlabeled_windows:                    # (B, 120, 6)
    masked, pos, target = span_mask(batch)        # (B,120,6), (B,18), (B,18,6)
    pred = limu_bert(masked, masked_pos=pos)      # (B, 18, 6)
    loss = F.mse_loss(pred, target)
    loss.backward(); opt.step()

# Phase 2 — supervised, on a FROZEN encoder, with very few labels.
#   embedding.py caches representations; classifier.py trains a small GRU.
H = limu_bert(labelled_windows, output_embed=True)   # (N, 120, 72), no grad
clf = GRUClassifier(input=72, output=n_classes)
for h, y in loader(H, labels):
    loss = F.cross_entropy(clf(h), y)               # only clf gets gradients

The released code goes further than convenience: embedding.py runs the encoder once over the whole dataset and saves the representations to disk as a .npy file. Phase 2 then never touches the transformer at all. That is a deliberate engineering choice with a real consequence — you can sweep classifier architectures, label rates, and target tasks in seconds, because the expensive part is a cached tensor.

It also enforces an honest evaluation. If the classifier is a small GRU on frozen features, then improvements are attributable to the representation, not to the classifier quietly relearning the features. Note the parameter in the released classifier.py: label_rate, the fraction of labelled data used. The entire claim of the paper is a curve of accuracy against that number.

What else you could hide

Masking timesteps is one choice out of several, and the alternatives are instructive because each teaches the encoder something different.

Mask whole channels. Zero out all of the gyroscope for a window and ask the model to reconstruct it from the accelerometer alone. This forces the encoder to learn the coupling between the two sensors — that a rotation about the vertical axis appears as a particular phase relationship in the horizontal accelerometer axes. It also produces a model that degrades gracefully when a sensor fails, which is a real deployment property.

Mask the future. Hide everything after timestep 100 and predict forward. This is extrapolation rather than imputation, it is strictly harder because there is no right-hand context, and it is one of the three tasks LSM measures its scaling laws on in Chapter 6.

Mask across modalities. Hide the accelerometer entirely and reconstruct it from heart rate and skin temperature. This sounds absurd until you realise it is exactly what a multimodal wearable model must learn if it is going to impute a sensor that a cheaper device does not have. LSM explicitly measures imputation "across time and sensor modalities."

Mask in the frequency domain. Remove a band and reconstruct it. Rarely used, and worth knowing why: it changes the loss geometry — an error in one frequency bin spreads across every timestep — so the objective stops corresponding to any per-sample likelihood and becomes hard to interpret.

Read the last row as the pivot of this entire lesson. Every other row keeps the supervision inside the sensor: the target is always some other part of the same signal, so the encoder can only ever learn what the signal is like. The last row goes outside, and that is the only way meaning gets in.

Exactly what you keep, and what shape it is

A point that trips people up. When LIMU-BERT is used as a feature extractor, the thing you keep is not a single vector per window. Read the released forward pass: with output_embed=True it returns h_masked straight from the transformer, before any gathering. The shape is

Not (B, 72). There is no pooling, no CLS token, no mean over time. That is a deliberate and consequential choice, and it explains the second phase's architecture.

Because the representation is still a sequence, the downstream classifier must consume a sequence — which is why the released classifier.py defaults to a GRU and offers LSTM, CNN, and attention variants. Walk the shapes for the default:

Contrast this with IMU2CLIP, which does pool: its GRU keeps only the final hidden state, giving (B, 512). One vector per window, full stop.

The difference is forced by the objective. IMU2CLIP must produce a single point in CLIP's space, because that is what it is being compared against — a caption is one vector, so a window must be one vector. LIMU-BERT has no such constraint; its objective is per-timestep reconstruction, so a per-timestep representation is the natural output, and preserving it keeps temporal information the downstream classifier might want.

One practical consequence worth flagging. Storing (120, 72) float32 per window is 34,560 bytes; storing a pooled (72,) is 288 bytes. For a corpus of 500,000 windows that is 17 GB against 144 MB. If you plan to cache representations for a retrieval index, you pool. If you plan to fine-tune a temporal head, you do not. That decision is made when you choose the pretraining objective, not later.

What the representation actually contains

One detail in the released dataset config deserves attention, because it foreshadows Chapter 7. Every dataset ships multiple label columns. HHAR has activity_label (6 classes), user_label (9 subjects), and model_label (3 phone models). UCI has 30 users. Shoaib has 5 body positions.

Those extra columns exist so you can probe what else the representation encodes. And it does encode them: a representation trained to reconstruct masked motion necessarily captures gait idiosyncrasy, because your gait is the structure it is predicting. This is simultaneously the source of the model's power and a genuine privacy property to reason about. An embedding good enough to fill in your missing footsteps is good enough to recognise you.

Chapter 4: Binding Motion to CLIP

Masked reconstruction gave us an encoder that understands the physics of motion. It still knows nothing about the meaning of motion. Ask it "which of these windows is someone chopping vegetables" and it has no way to represent the question, because its output space is 72 anonymous dimensions with no connection to language.

IMU2CLIP closes that gap with an observation about hardware. On a head-mounted device — smart glasses, an Aria research headset, a GoPro strapped to a forehead — the IMU is never alone. There is a camera on the same rigid body, recording the same instant. In datasets like Ego4D there is often also a human narrator describing, with timestamps, what the wearer is doing.

Those are free labels with an unbounded vocabulary. Nobody had to sit down and enumerate a taxonomy. The narration says "#C C picks up a knife from the counter," and that sentence is already aligned in time with a slice of accelerometer and gyroscope readings.

Why this transfers meaning it never saw

The magic here is transitivity, and it is worth being precise about why it works.

CLIP was trained on hundreds of millions of image-text pairs, so within CLIP's space, the vector for a photo of someone cycling already sits near the vector for the sentence "a person riding a bicycle." That relationship is baked in and, because the CLIP encoders are frozen, it cannot be damaged.

Now train IMU → the video vector. The IMU encoder pulls "vigorous rhythmic pedalling motion" toward "photo of cycling." But "photo of cycling" is already near "a person riding a bicycle." So IMU inherits the text relationship without ever being trained against text.

The paper measures this directly, and the number is one of the most persuasive in the field. Train IMU2CLIP only on IMU↔Video — the narration is never shown to the model — and then evaluate Text→IMU retrieval on Ego4D. It reaches R@1 = 4.17, R@10 = 15.62, R@50 = 43.06, MRR = 0.084. That is not chance. The model retrieves motion from sentences it was never trained to read.

Train with narration too and it improves to R@1 = 7.29, R@10 = 28.82, MRR = 0.143. Direct supervision beats transitive supervision, as you would expect — but transitive supervision alone already works, which is the surprising part.

Three other ways to transfer, and why contrastive won

"Make the IMU vector match the video vector" can be implemented in several ways, and the choice is not obvious in advance. Four candidates, in order of how naive they look.

Candidate 1: regression. Minimise ‖i − v‖² directly. Predict the CLIP vector from the IMU window, as a plain 512-dimensional regression. This is the most literal reading of "match," and it fails in a specific way: with a mean-squared objective on a stochastic target, the optimal prediction is the conditional mean. Many different scenes produce similar head motion, so the model learns to output the average of all of them — a blurry vector near the centre of the distribution that is close to everything and discriminative of nothing. The loss goes down; retrieval does not work.

Candidate 2: distillation with a fixed vocabulary. Run CLIP zero-shot on the video against a list of classes, take its output distribution, and train the IMU encoder to match that distribution with a cross-entropy loss. This works, and it is roughly what Wav2CLIP did for audio. The cost is that you have reintroduced a fixed vocabulary — the model can only ever express what is on the list, which is the exact prison Chapter 2 was about.

Candidate 3: contrastive. Only require that the true pair scores higher than the others in the batch. Note how much weaker a requirement that is: it says nothing about the absolute position of any vector, only about relative ordering. That weakness is the strength. The model is free to place a scene's vector anywhere, as long as the IMU that co-occurred with it lands nearby and the IMU from a different scene does not.

Candidate 4: captioning. Train the IMU encoder to generate the narration text autoregressively. Powerful, and expensive: you need a decoder, the objective is much harder, and you get a generative model when what you wanted was a retrieval index.

The IMU tower, shape by shape

The released encoder is called MW2StackRNNPooling, and it is a stack of three dilated 1D-convolution blocks followed by a GRU. Trace a 5-second Ego4D window, resampled to 200 Hz, so 1000 timesteps.

Input: (B, 6, 1000). Channels first, because the first operation is a convolution.

GroupNorm(2, 6). Two groups over six channels: group one is the three accelerometer axes, group two is the three gyroscope axes. This is exactly the paper's statement that it normalises "the Accelerometer (3D) and the Gyroscope (3D) signals independently." It matters because the two sensors have completely different units and dynamic ranges — roughly ±20 m/s² against roughly ±35 rad/s. Normalising them jointly would let whichever has the larger numeric spread dominate; normalising per group puts them on equal footing while preserving the internal geometry of each triple.

Block 1: Conv1d(6 → 32, kernel 10, dilation 2), then MaxPool(3). A dilated kernel of size 10 with dilation 2 spans 2(10 − 1) + 1 = 19 timesteps — 95 milliseconds at 200 Hz — while using only 10 weights per channel pair. Output length: 1000 − 19 + 1 = 982. After max-pooling by 3: ⌊982/3⌋ = 327. Tensor: (B, 32, 327).

Block 2: Conv1d(32 → 32, kernel 5, dilation 2), MaxPool(3). Span 2(4) + 1 = 9. Length 327 − 9 + 1 = 319, pooled to ⌊319/3⌋ = 106. Tensor: (B, 32, 106).

Block 3: identical. 106 − 9 + 1 = 98, pooled to ⌊98/3⌋ = 32. Tensor: (B, 32, 32).

GroupNorm(4, 32), then GRU(hidden = 512), keep the final hidden state. Output: (B, 512) — the CLIP embedding dimension for ViT-B/32.

A thousand timesteps compressed to thirty-two, then pooled to a single vector. That 31× temporal reduction before the recurrence is what makes the GRU affordable — a GRU over 1000 steps would be both slow and prone to forgetting the beginning of the window.

Parameter count, so you can compare it honestly to what it is imitating:

The student is a hundred times smaller than the teacher it is copying. That asymmetry is the entire product argument: at inference, on the device, only the small one runs.

The pair-construction problem, which is the actual hard part

Everything above assumes you can hand the model matched (IMU, video, text) triples. Producing them from egocentric recordings is where most of the engineering lives, and the released preprocessing tells the story.

Problem 1: the modalities have different clocks. Ego4D IMU arrives at varying rates and must be resampled — the released preprocessing script uses torchaudio.functional.resample to force everything to 200 Hz, and the video is separately re-encoded to 224×224 at 10 fps. Two independent resamplings that must agree on a common timeline to within a fraction of a second, or every training pair is slightly wrong in a way no loss can detect.

Problem 2: sensor data goes bad, silently. The IMU2CLIP repository ships two files called bad_imu_windows_2.5.json and bad_imu_windows_5.0.json — explicit deny-lists of windows, one per window length, that are excluded from training. Sensors drop out, saturate, or record garbage during a mount adjustment, and a contrastive objective is actively harmed by garbage: a nonsense IMU window paired with a valid video vector teaches the encoder to map noise to meaning.

Problem 3: which frame is "the" frame? A 5-second window contains 50 video frames at 10 fps. The released config names its video encoder clip_1frame. One frame is selected to represent the whole window. That is a real approximation with a real cost: if the wearer picks up a knife in second 4 of a 5-second window and you embedded the frame from second 1, your target vector describes an empty counter while the IMU describes a reach.

Problem 4: narration timestamps are approximate. Ego4D narrations are free-text sentences with a single timestamp, written by annotators watching the video. "C picks up a knife" is attached to a moment, but the motion begins before and ends after. The paper's solution is scale — 528,000 training windows on Ego4D — where the misalignment averages out rather than being fixed.

One pair, traced end to end on a timeline

Make the alignment problem concrete by building a single training triple from raw recordings, with real timestamps.

A recording begins at wall-clock time 0.000 s. The narrator's annotation reads "#C C picks up a knife from the counter" at t = 47.300 s. You want the training window centred there, 5 seconds long, so you need samples from 44.800 s to 49.800 s.

The IMU side. After resampling to 200 Hz, sample index i corresponds to time i/200. So

The video side. After re-encoding to 10 fps, frame index j corresponds to time j/10. The window spans frames 448 through 498 — fifty frames. But the released config uses clip_1frame, so exactly one is selected. Choose the centre:

The text side. The narration string is passed to CLIP's tokeniser, which pads or truncates to 77 tokens, producing a (77,) integer tensor.

So the triple is: (6, 1000) float, (3, 224, 224) float, (77,) int. Three tensors of wildly different shapes, three encoders, one 512-dimensional meeting point.

Now the failure modes, quantified. Suppose the two resamplers disagree about the origin by a mere 200 milliseconds — a plausible discrepancy if IMU timestamps are device-clock and video timestamps are container-relative. At 10 fps that is 2 frames; at 200 Hz it is 40 samples. The IMU window you extracted is shifted 0.2 s relative to the frame you paired it with. Over a 5-second window that is a 4% misalignment, which sounds harmless.

It is not harmless for short events. A knife pickup takes perhaps 0.6 seconds. A 0.2-second offset is a third of the event, and if the offset is systematic across the whole dataset the model learns a consistently lagged correspondence — which then fails on any deployment where the lag is different or absent.

The loss, derived and then computed by hand

Both towers produce vectors on the unit hypersphere. Because they are unit-normalised, similarity is just an inner product — the paper writes exactly this:

Within a batch of B pairs, define a retrieval distribution over the batch using a temperature γ:

The loss is the negative log-probability of the true partner, averaged both directions:

Symmetry is not decoration. L_i2t asks "given this motion, is the right sentence ranked first?" and L_t2i asks "given this sentence, is the right motion ranked first?" Those are different questions with different failure modes, and optimising only one produces a space that is good in one direction and lopsided in the other.

Now compute it. Take B = 3 and put the embeddings on a 2-D unit circle so every dot product is a cosine of an angle difference. IMU vectors at 20°, 95°, 112°; text vectors at 35°, 100°, 125°. The similarity matrix, with rows indexed by IMU and columns by text:

Look at row 3 before doing any arithmetic. The true pair scores 0.97437. The impostor t₂ scores 0.97815. The wrong answer is winning, by 0.00378. That is what the third window of a real batch looks like when two narrations describe similar motions.

Set the temperature so that 1/γ = 10 (the released InfoNCE class defaults to temperature 0.1, and makes it learnable). Row by row:

Row 1. Logits: [9.6593, 0.8716, −2.5882]. Subtract the max for stability: [0, −8.7877, −12.2475]. Exponentials: 1, 0.00015261, 0.0000048. Sum = 1.00015741. So P = 0.99984, and

Row 2. Logits: [5.0, 9.9619, 8.6603] → [−4.9619, 0, −1.3016]. Exponentials: 0.0069996, 1, 0.272096. Sum = 1.2790956. P = 1/1.2790956 = 0.781803.

Row 3, the hard one. Logits: [2.2495, 9.7815, 9.7437] → [−7.532, 0, −0.0378]. Exponentials: 0.00053565, 1, 0.962905. Sum = 1.96344. The probability assigned to the true partner is 0.962905/1.96344 = 0.490423, while the impostor gets 0.509311.

Now the other direction, reading down the columns. Column 1 gives loss 0.010028; column 2 gives 0.60705; column 3 gives 0.29151.

Sit with column 3 for a moment. Reading down, the true IMU window i₃ gets probability 0.747 — from the text's point of view the match is comfortable. Reading across, from the IMU's point of view, the same pair is losing at 0.490. The relationship is not symmetric even though the similarity matrix is not asymmetric — the asymmetry comes entirely from what else is in the batch. That is precisely why you need both terms.

What temperature does to a mistake

Recompute row 3 alone at three temperatures. The gap between the impostor and the truth is 0.00378 in cosine, so the gap in logits is 0.00378 × s.

The loss is not monotone in the scale, and understanding why is understanding temperature. At s = 1 everything is nearly flat, so the distribution is close to uniform and the loss sits near its ceiling of ln 3 = 1.0986 regardless of what the model knows. At s = 1000 the single closest impostor swamps everything and the loss becomes essentially the gap times the scale — enormous gradients aimed at one hard negative. In between, at s = 10, the far negative t₁ has already been suppressed (its exponential is 0.0005) while the near negative has not yet been amplified. That is the minimum.

Where the gradient actually points

The loss value tells you how wrong the model is. The gradient tells you what it will do about it, and for InfoNCE it has a beautifully interpretable closed form. Derive it once and you will never have to guess again.

Write the row-a loss with logits z_b = s · ⟨i_a, t_b⟩:

Differentiate with respect to a single logit. The first term contributes −1 only when b = a; the log-sum-exp contributes the softmax probability p_b:

Now chain through z_b = s ⟨i_a, t_b⟩, whose derivative with respect to i_a is simply s t_b:

Read that in English. The gradient is the difference between the probability-weighted average of every text vector in the batch and the true one. A gradient-descent step moves the IMU embedding toward its true partner and away from the impostors, with each impostor's pull weighted by exactly the probability the model currently assigns it. A negative the model already dismisses contributes nothing. A negative the model finds tempting dominates.

Compute it on row 3 of our batch, where the model was getting the answer wrong. The probabilities were p = (0.000273, 0.509311, 0.490423) for t₁, t₂, t₃. In 2-D the text vectors are

The weighted average, term by term:

Subtract the true partner t₃ and multiply by s = 10:

Now check what a descent step does. The embedding i₃ sits at 112°, so its counterclockwise tangent direction is (−sin112°, cos112°) = (−0.92718, −0.37461). Project the gradient onto it:

The projection is negative, so descent (which moves against the gradient) pushes i₃ in the positive tangent direction — counterclockwise, from 112° toward larger angles. That is toward t₃ at 125° and away from t₂ at 100°. The arithmetic does exactly what the story promised, and you can verify every digit.

Why both towers must be normalised

One more detail with real consequences. The similarity is defined as an inner product only because the vectors are constrained to the unit sphere. Drop that constraint and everything breaks in a specific way.

Suppose an IMU embedding has norm 3 while its true text partner has norm 1, and an impostor also has norm 1. The unnormalised inner products are 3× larger for whichever direction the long vector points, so a single example can dominate the batch's logits purely by being long. The model then discovers the cheapest available strategy: make the vectors for easy examples longer. Length becomes a confidence channel that the loss never asked for and cannot control.

Normalising kills that degree of freedom. After L2 normalisation the inner product equals the cosine, which lies in [−1, 1] regardless of anything the encoder does, so the only way to increase a similarity is to rotate. The paper states the constraint directly: the embeddings of each modality lie on the unit hypersphere S^D, and "since the embeddings are unit-normalized, the similarity can be simply calculated as their inner product."

The geometric consequence is worth holding on to. With similarities bounded in [−1, 1] and a scale of 10, the maximum possible logit gap between the best and worst pair in a batch is 10 × 2 = 20. That bound is precisely why the temperature has to be tuned or learned: it sets the entire dynamic range of the objective.

Batch size is the difficulty knob, and it was 16

Notice what B controls. With B = 3, the model has to pick the right partner out of three — random guessing scores 1/3 and the loss ceiling is ln 3 = 1.099. With B = 256 the ceiling is ln 256 = 5.545 and the task is far harder, so the gradient signal is far richer. This is why CLIP itself trained with a batch of 32,768.

IMU2CLIP trained with batch size 16, across 8 GPUs, for 15 epochs. The paper explains why plainly: "to accommodate the memory constraints, we pool the negative samples within the same batch (randomly shuffled), reducing the load on each GPU." The optimiser is Adagrad with learning rate 0.01, epsilon 10⁻⁸, decay 0.1.

Sixteen is small. It means each training step asks a fairly easy question, and it is a real limitation you should read into the results in the next chapter. It is also a straightforwardly improvable one — the standard fixes (gathering negatives across all GPUs, or a momentum queue) were well known by 2022 and would raise the effective B by orders of magnitude without extra memory per device.

python — the symmetric objective, written outimport torch, torch.nn.functional as F

def symmetric_infonce(z_imu, z_other, temperature):
    # z_imu, z_other: (B, 512), both L2-normalised onto the unit sphere
    z_imu   = F.normalize(z_imu,   dim=-1)
    z_other = F.normalize(z_other, dim=-1)

    logits = z_imu @ z_other.T / temperature      # (B, B) — cosine / γ
    target = torch.arange(len(z_imu), device=z_imu.device)   # diagonal is truth

    loss_i2t = F.cross_entropy(logits,   target)   # rows: motion → text
    loss_t2i = F.cross_entropy(logits.T, target)   # cols: text → motion
    return 0.5 * (loss_i2t + loss_t2i)

# Training step. Note which encoders receive gradients.
z_imu   = imu_encoder(batch["imu"])                          # (B,6,1000) → (B,512)  TRAINED
with torch.no_grad():
    z_video = clip.encode_image(batch["frame"])              # (B,3,224,224) → (B,512)  FROZEN
    z_text  = clip.encode_text(batch["narration"])          # (B,77) → (B,512)  FROZEN

loss = symmetric_infonce(z_imu, z_video, tau) \
     + symmetric_infonce(z_imu, z_text,  tau)          # the i↔v↔t variant

Read the last three lines carefully, because they encode the paper's three training regimes. Use only the first term and you get IMU↔Video. Use only the second and you get IMU↔Text. Use both and you get IMU↔Video↔Text. The paper reports all three, and — as we will see — the best model for each task is the bi-modal one trained on that task's modality, not the tri-modal one.

Chapter 5: What an Embedding Actually Buys

An embedding is not a result. It is a change in what questions are expressible. This chapter is about the four capabilities that appear the moment your IMU encoder lives in a shared space, and the honest numbers behind each.

The datasets, stated plainly

IMU2CLIP evaluates on two egocentric corpora, split by video ID so that no recording appears on both sides:

Look at the two rows for Ego4D and let the ratio land: 528,000 self-supervised windows against 1,552 labelled ones. That is a factor of 340. Every one of those 528,000 came for free from data that already existed; every one of the 1,552 required a human decision. The entire argument for cross-modal pretraining is visible in that single comparison, before any accuracy number.

Reading retrieval metrics without fooling yourself

Two metrics carry every number in this chapter, and both are worth computing once by hand.

Recall at k (R@k) is the fraction of queries whose correct answer appears somewhere in the top k results. It is a hit-or-miss measure: rank 2 and rank 10 are equally good at k = 10, and equally bad at k = 1.

Mean reciprocal rank (MRR) is the average of 1/rank across queries. Suppose five queries place the correct answer at ranks 1, 3, 10, 50, and 100:

Notice the shape of that computation. The rank-1 result supplied 68% of the total; the rank-100 result supplied 0.7%. MRR is dominated almost entirely by how often you are right at the very top, which is exactly the property you want for a search box and exactly the wrong property if you care about the tail.

Now use it to read the paper's headline. IMU2CLIP's best Text→IMU MRR is 0.143. If every query behaved identically, that would correspond to a rank of 1/0.143 = 7.0. Interpreting it that way is a slight abuse — the reciprocal of a mean is not the mean of reciprocals — but as a rule of thumb it says: the right motion window is typically somewhere in the top ten. For a query typed in free-form English against a pool of egocentric IMU windows, with no class list anywhere, that is a genuinely useful system.

Capability 1: search your motion history with a sentence

Type "jumping." Embed it with CLIP's frozen text tower. Compute the dot product against every cached IMU embedding. Sort. You have just searched a motion archive with natural language, and at no point did the system need to have a "jumping" class.

Here is the full retrieval table from the paper, on Ego4D, with the CLIP video-text reference row at the bottom:

Three readings, in order of importance.

First, the calibration. R@1 of 7.29 sounds low in isolation. It is meaningless without a reference, and the paper supplies the right one: run CLIP itself, a near-state-of-the-art video-text model in 2022, on the same test set and the same protocol. Text→Video gives MRR 0.168. Text→IMU gives MRR 0.143. The paper's own summary of that comparison is the honest one: "the narrow margin in the performances shows that the IMU encoder could serve as a power-efficient alternative for a video encoder in many applications." An 0.85-million-parameter model reading six numbers gets within 85% of the MRR of an 88-million-parameter model looking at pixels.

Second, IMU↔Video is easier than IMU↔Text. Look at the first row: IMU→Video reaches R@10 of 43.13, against 18.75 for IMU→Text. The paper attributes this to "higher compatibility", and the mechanism is intuitive. Video and IMU are recorded by the same rigid body, so head rotation appears in both as a literal geometric fact. Text is a summary written by a person, filtered through their choice of words, and two different narrators will describe the same motion differently.

Third, tri-modal training is a compromise, not a win. The bottom trained row is never the best in any column. Train on video and you win the video columns; train on text and you win the text columns; train on both and you land in between. There is a single 512-dimensional output vector and two objectives pulling on it, and they do not want the same thing.

The pattern replicates on Aria, which has video but no narrations: IMU→Video reaches MRR 0.182 with R@{1, 10, 50} of {8.48, 38.83, 77.67}, and Video→IMU reaches MRR 0.190 with R@{1, 10, 50} of {8.48, 44.19, 78.57}.

Flip to "unaligned" and the ranking becomes noise. That is the control experiment the paper runs as its zero-shot baseline, and it matters: retrieval only works because training put the IMU vectors somewhere specific relative to the text vectors. Nothing about the geometry of a 512-dimensional sphere makes this happen for free.

Capability 2: retrieve video without touching the camera

Reverse the query. Take an IMU window and retrieve the video segments whose CLIP embeddings are nearest. Video→IMU on Ego4D reaches R@1 of 12.19 and R@10 of 45.31 — the strongest numbers in the whole table.

The product framing is what makes this interesting. A device can run the IMU tower continuously at negligible cost, index every window, and then answer "show me the clip where I was doing this" against an archive — without ever having run a vision model on-device. The paper names this precisely: "a new and power-efficient way of performing media retrieval or online action detection."

Capability 3: activity recognition that needs almost no labels

Here is the result that turns the representation into a resource. Same IMU architecture in every row; only the initialisation differs.

The headline. On Ego4D, identical architecture: 49.92% accuracy from scratch, 65.87% after cross-modal pretraining. That is about 16 points absolute, and the paper flags exactly this comparison. On F1 the jump is larger in relative terms: 23.23 to 45.15, nearly double.

The zero-shot row is where the honesty is. Zero-shot here means: encode each activity's name with CLIP's text tower, then classify each IMU window by nearest neighbour in the shared space, using no labelled examples at all. On Ego4D with the i↔t model it reaches 31.89 F1 — well above chance, and genuinely remarkable given zero supervision. But on Aria, zero-shot scores 18.46 F1 against a random-init baseline of 56.35. It is worse than training from scratch.

Why? Two compounding reasons. Aria has no narrations, so the i↔v model reaches text only transitively through CLIP. And the class names for Aria are physical-activity words — hiking, running, biking — whose CLIP text embeddings were shaped by photographs of those activities, not by their kinematics. The word "hiking" evokes a mountain in CLIP's space. The IMU sees a gait.

Zero-shot, computed on one window

Zero-shot classification deserves to be worked rather than described, because the mechanism is so simple that people assume it must be more than it is.

Take a single IMU window of vigorous pedalling. Encode it: one 512-dimensional unit vector. Now encode five candidate strings with CLIP's frozen text tower and take the cosine with each. Realistic CLIP cross-modal cosines cluster in a narrow band around 0.15 to 0.35 — the two towers do not land on top of each other even when they agree — so suppose you measure:

Check the arithmetic. CLIP's learned logit scale is close to 100, so multiply each cosine by 100 and subtract the maximum, 26.4, giving 0, −2.3, −5.2, −6.6, −8.1. Exponentiate: 1, 0.100259, 0.005517, 0.001360, 0.000304. The sum is 1.107440, so

Three observations, each of which is a practical lever.

First, the raw cosine spread is tiny. The winner and the loser differ by 0.081 in cosine. Without the ×100 scale the softmax would be almost uniform. The temperature is not a detail here; it is the only reason zero-shot produces a decision at all.

Second, the sentence beats the bare word. "a person riding a bicycle" scores 0.264 while "biking" scores 0.212, and after the scale that 0.052 difference becomes a factor of 180 in probability. CLIP's text tower was trained on captions, so it represents caption-shaped strings better than bare nouns. This is the prompt template effect, and it is free accuracy: wrapping every class name in "a person {name}" costs one text-encoder pass per class, once, ever.

Third, the failure mode is visible in the table. "a photo of a mountain trail" — a scene description with no motion content — still scores 0.183, only 0.081 below the winner. CLIP's text space is organised by what things look like, and the IMU tower inherited that organisation. This is precisely the mechanism behind Aria's poor zero-shot result: class names like "hiking" carry scenery in CLIP's geometry, and scenery is the one thing an accelerometer cannot see.

python — zero-shot with prompt ensemblingTEMPLATES = ["a person {}", "a video of someone {}",
             "first-person footage of {}", "{}"]

def class_embeddings(names):
    # One vector per class, averaged over wordings, then renormalised.
    E = []
    for n in names:
        v = np.stack([clip.encode_text(t.format(n)) for t in TEMPLATES])
        v /= np.linalg.norm(v, axis=1, keepdims=True)   # each on the sphere
        m = v.mean(0)
        E.append(m / np.linalg.norm(m))                   # back ONTO the sphere
    return np.stack(E)                                    # (C, 512)

# The renormalisation is not cosmetic. The mean of four unit vectors is
# SHORTER than one, and it is shortest exactly for the classes whose
# wordings disagreed most — so without it, the ambiguous classes are
# silently penalised in every comparison.
logits = 100.0 * (z_imu @ class_embeddings(names).T)      # (N, C)
pred = logits.argmax(axis=1)

The other honest row is Aria's random-init baseline: 76.11% accuracy but only 56.35 F1. A twenty-point gap between accuracy and macro-F1 is the signature of class imbalance. The model is doing well on the majority class and poorly elsewhere; accuracy is flattering it. Note that even the best pretrained Aria model gains only about 7 points of accuracy (76.11 → 83.54) — a real improvement, but nothing like the Ego4D story, because Aria's baseline was already strong on an easier, more balanced-in-practice five-way problem.

How much of that 16 points should you believe?

Look back at the dataset table before celebrating. The Ego4D activity-recognition test set contains 241 windows. That is the entire basis for the 65.87% figure.

Compute the sampling uncertainty. For a proportion p estimated from n independent samples, the standard error is √(p(1−p)/n). At p = 0.6587 and n = 241:

Do the same for the from-scratch baseline at p = 0.4992: SE = √(0.25/241) = 0.03221, so ±6.3 points. Both numbers carry an interval of roughly six points, and a single misclassified window moves the score by 1/241 = 0.41 points.

So is the 16-point gap real? Almost certainly yes — it is well outside either interval, and the same ordering reproduces across F1, across probing versus fine-tuning, and across a second dataset. But you should hold "16 points" loosely and "the pretrained model is clearly better" firmly. Those are different claims with very different error bars, and the honest way to compare two models on the same 241 items is a paired test on their per-item agreements, not two independent intervals.

The same arithmetic reframes the retrieval table. R@1 differences of half a point between rows, on a test set of this size, are not differences. The pattern — video-trained models win the video columns, text-trained models win the text columns, tri-modal lands between — is the finding, because it is consistent across eight columns and two datasets. Individual cells are noise.

Capability 4: sensor logs as text for a language model

The paper's final experiment is a sketch rather than a benchmark, and it is the one that aged best. Run IMU2CLIP as a zero-shot tagger — score each IMU window against a bank of candidate descriptions and keep the top one — and you get a textual log of motion. Do the same for the audio channel. Now hand that log to a large language model as context and ask questions in English: "What can you tell me about the user activity using the motion logs?" or "What time did I start biking?"

The model answers from the log. The paper's own comparison is with Socratic Models, and the distinguishing claim is about the power budget: this approach "does not rely on the video signals at all — which would incur much higher power consumption — thus operating better under real-world constraints."

What is really being described is a translation layer. The IMU is a modality no language model has ever been trained on. Give it a text surface and every downstream capability of language models — summarising, question answering, tool use — becomes available to it without a single new training run.

python — the four capabilities, all from one cached matriximport numpy as np

# Encode the archive ONCE. This is the only expensive step, and on a
# wearable it is also the only step that runs on-device.
Z = np.stack([imu_encoder(w) for w in archive])        # (N, 512)
Z /= np.linalg.norm(Z, axis=1, keepdims=True)

def search(query: str, k=10):                            # capability 1
    q = clip.encode_text(query); q /= np.linalg.norm(q)
    return np.argsort(-(Z @ q))[:k]

def similar_moments(idx, k=10):                        # capability 2
    return np.argsort(-(Z @ Z[idx]))[1:k+1]

def zero_shot(class_names):                            # capability 3
    E = np.stack([clip.encode_text(f"a person {c}") for c in class_names])
    E /= np.linalg.norm(E, axis=1, keepdims=True)
    return (Z @ E.T).argmax(axis=1)

def to_log(vocab):                                    # capability 4
    tags = zero_shot(vocab)
    return "\n".join(f"{i*5}s: {vocab[t]}" for i, t in enumerate(tags))
    # → feed this string to an LLM as context and ask questions in English

Every function after the first is a dot product and a sort. That is the payoff of an embedding: capabilities that used to be separate models become linear algebra over one cached matrix.

What would make these numbers substantially better

IMU2CLIP's results are a 2022 snapshot, and it is worth being precise about which of its limitations are fundamental and which are simply engineering that nobody had done yet. Five levers, roughly in order of expected effect.

Lever 1: batch size. Training used batch 16, with the paper explicitly citing memory constraints and in-batch negative pooling. Random-guess loss at B = 16 is ln 16 = 2.77; at B = 4,096 it is 8.32. Gathering negatives across all 8 GPUs, or keeping a momentum queue, raises the effective batch by orders of magnitude at no extra memory per device, and this was standard practice by 2022. Of all five levers this is the cheapest and probably the largest.

Lever 2: a temporal video encoder. The released config uses clip_1frame — one frame per window. A window's target therefore describes a static scene, so the IMU tower is trained to predict scenery rather than motion. Any encoder that consumes several frames would supply targets that distinguish "picking up" from "putting down," which single frames structurally cannot.

Lever 3: better pair construction. Chapter 4 traced the alignment problem. Narration timestamps are approximate, the frame is chosen by a fixed rule, and bad windows are excluded by a hand-curated deny-list. Every improvement here raises the ceiling on every downstream number, and none of it requires a new idea — only careful work.

Lever 4: a stronger frozen teacher. The method inherits whatever geometry the teacher has. CLIP ViT-B/32 was a reasonable 2022 choice; a larger or better-trained image-text model would give a better space to imitate, at zero inference cost, because the teacher never ships.

Lever 5: more, and more diverse, IMU. 540 hours of Ego4D training data is small next to the 1.43 million hours LIMU-BERT-X later used. Combining a masked-reconstruction pretraining stage on a huge unlabelled corpus with a cross-modal alignment stage on a smaller paired one is the obvious synthesis, and it is exactly what the two halves of this lesson suggest.

Capability 5: noticing that something changed

One capability the paper does not name, but which falls out of the same cached matrix and is arguably the most valuable in a health context: change detection without a label for what changed.

The construction is three lines. Take a user's windows for a reference period — last month, say. Compute the centroid of their embeddings. Then, for each new window, measure the cosine to that centroid. A drift in that number says this person is moving differently than they used to, and it says it without anyone ever having defined a class called "moving differently."

Work an example. Suppose the reference-period cosines to the centroid have mean 0.82 and standard deviation 0.06. A new week comes in at mean 0.71. In standard deviations:

Not dramatic on its own — roughly a one-in-fifteen event if nothing changed. But run it over 40 windows and take the mean, and the standard error of that mean is 0.06/√40 = 0.0095, so the same shift is

This is the shape of nearly every deployed wearable health signal: a per-window measurement that is far too noisy to act on, aggregated over enough windows to become certain. And it works on embeddings specifically because the embedding space has geometry — a distance in it means something — which is the property Chapter 0 defined and everything since has been earning.

What it takes to actually ship the search box

Retrieval as a research result is a dot product. Retrieval as a product is an index, and the arithmetic decides the architecture.

Take a year of a user's motion, windowed at 5 seconds with no overlap. That is

Store each as a 512-dimensional float32 embedding and you need 6.3M × 512 × 4 = 12.9 GB. That is not going on a watch, and it is a lot even for a phone. Three compressions, each with a known cost:

And the search cost. Brute-force cosine over 6.3M vectors of 512 dimensions is

Three billion multiply-accumulates for one search. On a phone that is perhaps a second of full-power compute — noticeable, and terrible for battery if the user types more than one query. Approximate nearest-neighbour structures (HNSW, IVF) bring that down by two to three orders of magnitude at a few percent recall cost, which is the standard trade and the reason every production embedding system has one.

Notice what does not appear in any of these numbers: the text encoder. Encoding the query is a single forward pass through CLIP's text tower, and in a product you would run it on a server or ship a distilled version. The asymmetry that started in Chapter 4 — small tower on device, large tower elsewhere — persists all the way into the deployment diagram.

Chapter 6: What Changes at Population Scale

Everything so far has been about a few hundred hours of recording from a few hundred people. Between 2023 and 2025 the numbers changed by four orders of magnitude, and when they did, three things stopped being true.

The four datasets that redefined "large" for this modality

Read the second column top to bottom. Thirty subjects, then 141,000, then 165,000, then 60,000 across 1,100 different phone models, then 162,000 people contributing 2.5 billion hours. The field went from "we have a benchmark" to "we have a population" in four years.

Change one: raw sampling stops being affordable, and that is a modelling decision

Do the arithmetic that forces LSM's design. Forty million hours of continuous 6-channel data sampled at 50 Hz:

That is not a training run, that is a data-centre project. So LSM does not use raw samples. Its stated unit is per-minute data across heart rate, heart-rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter. Redo the count:

Three thousand times smaller, and now it fits. But look at what was traded away. Per-minute aggregates cannot represent cadence, foot-strike sharpness, or tremor — everything Chapters 1 through 3 were about. What survives is the shape of a day: the slow envelope of activity, sleep, stress, and recovery.

The token budget, and why a day does not fit

There is a second forcing function behind the per-minute choice, and it is the sequence length. Self-attention costs O(T²) in both time and memory. Ask what T would be for a context of one day at various resolutions:

Compare the third row to LIMU-BERT's T = 120 and its 14,400-entry attention matrix. A per-minute day is 1,440 steps — only twelve times longer — so the same architectural family works without modification. That is not a coincidence; it is the resolution chosen so that a day fits the same machinery a gait cycle does.

And notice what the two regimes have in common: both pick T ≈ 10² to 10³ and then choose the sampling rate to make the window they care about fit. The window length is the design goal; the rate is the consequence.

python — the aggregation that makes population scale possibleimport numpy as np

def per_minute(stream, fs=50):
    """(N, 6) raw at 50 Hz  →  (minutes, K) summary features.
    3000 raw values per channel per minute become a handful."""
    n_min = len(stream) // (fs * 60)
    B = stream[:n_min * fs * 60].reshape(n_min, fs * 60, 6)
    mag = np.linalg.norm(B[:, :, :3], axis=2)          # rotation-invariant
    return np.stack([
        mag.mean(1),                                   # activity level
        mag.std(1),                                    # variability
        (mag > 10.5).mean(1),                          # fraction in motion
        np.abs(np.diff(mag, axis=1)).mean(1),          # jerkiness
        np.linalg.norm(B[:, :, 3:], axis=2).mean(1),   # rotation rate
    ], axis=1)                                        # (n_min, 5)

# 3000 values per channel per minute → 5 numbers. A 600× reduction,
# and the discarded 2995 contained every cadence and impact detail.
# That is the trade, stated as code rather than as prose.

Change two: "what is a positive pair" changes meaning

At small scale, a positive pair is two things that happened at the same instant — the IMU window and the video frame. At population scale, that framing runs out, because for most wearables there is no camera and no narrator.

Apple's PPG and ECG foundation models answer this with participant-level positive pair selection. Two segments recorded from the same participant, potentially days apart, are treated as a positive pair. The augmentation module perturbs them, and a regularised contrastive loss with momentum training pulls them together.

Think about what that objective is asking for. It says: everything that is stable about this person is signal; everything that varies within them is nuisance. The representation it produces is therefore a representation of the individual. And the paper reports exactly the consequence you would predict — that the pretrained models "readily encode information regarding participants' demographics and health conditions."

That is a genuine capability and a genuine hazard, and both come from the same line of code. An embedding trained to be invariant within a person and discriminative between people is, definitionally, a biometric.

Change three: heterogeneity becomes the dominant variable

The most instructive single number in this chapter is LIMU-BERT-X's 1,100 phone models. Not 1,100 devices — 1,100 distinct models, each with its own IMU part, its own factory calibration, its own sampling-clock accuracy, its own driver-level filtering, its own thermal behaviour.

At 30 subjects on 3 phone models — the HHAR configuration — you can plausibly normalise these differences away by hand. At 1,100 you cannot enumerate them, let alone characterise them. The nuisance factors have to become something the model learns to be invariant to, which means they have to be represented in the training distribution, which is the whole reason the data collection is worth 1.43 million hours.

This is a different argument for scale than the language-model one. In language, scale buys you knowledge. Here, scale mostly buys you invariance. The physics of walking has not become more complex; the number of ways it can be measured has.

Distribution shift, computed on two devices

"Distribution shift" is a phrase that hides its own mechanism. Make it concrete with the same eight samples from Chapter 2.

Device A reports the true acceleration. Device B has a 2% scale-factor error and a +0.15 m/s² bias on the z-axis — both entirely ordinary for consumer parts, both within datasheet tolerance. So device B reports

Recompute the two headline features. The mean transforms affinely:

The standard deviation ignores the bias entirely and scales with the gain:

Two different features, two completely different sensitivities to the same hardware defect. An additive bias attacks mean-like features and leaves scale-like features untouched; a multiplicative gain error does the reverse. A tree that split on the mean at 9.5 now flips its decision for every window from device B; a tree that split on the standard deviation at 1.8 does not notice.

And here is the fix, which you have already seen twice without recognising it. Standardise within the window: z = (x − μ)/σ. Ask what happens under an arbitrary affine device transform x′ = ax + b, with a > 0:

Exactly invariant. Every scale-factor error and every additive bias vanishes, for free, in one line.

What normalisation cannot fix is anything non-affine: a device whose firmware applies its own low-pass filter, a sensor that saturates at a different threshold, a part whose noise floor is ten times higher. Those distort the shape of the signal, not just its offset and scale, and no closed-form transform undoes them. The only remedy is to have seen enough of them during training — which is the entire justification for collecting 1.43 million hours across 1,100 phone models rather than 26,000 windows across three.

Scaling laws, and what they were measured on

LSM's central contribution is a set of scaling laws — how performance moves with compute, data, and model size — and it is worth being precise about the tasks those laws were measured on, because they are not classification tasks.

The three are imputation (fill in a gap inside an observed span), interpolation (fill between two observed spans), and extrapolation (predict forward past the last observation), measured both across time and across sensor modalities. That last phrase deserves emphasis: predicting one sensor's channel from the others is a legitimate task in this framework, so a model can be evaluated on inferring what the missing heart-rate channel would have said given accelerometer and skin temperature.

Notice that these are the same family of objectives as Chapter 3's masked reconstruction, generalised. Imputation is masking in the middle. Extrapolation is masking the end. What changed with scale is not the idea; it is that at 40 million hours you can measure how the idea scales, and the paper additionally reports that the resulting model enables sample-efficient downstream learning for exercise and activity recognition — the same downstream claim LIMU-BERT made at 1/10,000th the data.

What a scaling law actually promises

"Scaling laws" is a phrase that sounds like a guarantee of progress. Compute one and the mood changes.

Empirical scaling laws are power laws in the resource: loss L falls as

where C is compute (or data, or parameters) and α is a small positive exponent, typically in the range 0.03 to 0.1 for the regimes people report. Take α = 0.05 and ask what a tenfold increase in compute buys:

Now ask what it takes to halve the loss:

A million times the compute to halve the loss. That is what a power law with a small exponent means, and it is simultaneously the encouraging news and the brutal news. Encouraging, because the relationship is smooth and predictable — you can run three small experiments, fit two parameters, and forecast the result of a run you have not done. Brutal, because the returns are relentlessly diminishing.

This is why LSM's contribution is framed as establishing the laws rather than as reporting a number. Knowing the exponent tells you whether the next order of magnitude of data collection is worth the consent process, the storage, and the compute — and for wearables, where data acquisition means enrolling more humans, that forecast is worth more than any single benchmark result.

Why imputation is the task that matters most

LSM's headline tasks — imputation, interpolation, extrapolation — sound like academic exercises. They are the most operationally important capability in wearable sensing, and the reason is mundane: the data has holes in it.

Count them. A watch that is worn 20 hours a day, charging for the other 4, produces

And the gaps are not random. They occur at the same time each day, they correlate with showering and sleeping and exercise, and they differ systematically between users who charge at night and users who charge at their desk. A model that requires complete sequences must discard any day containing a gap, which is nearly every day. A model trained to impute simply fills them in and proceeds.

Now the second kind of hole, which is more interesting. Different devices carry different sensors. A premium watch has heart rate, HRV, electrodermal activity, skin temperature, accelerometer, and altimeter; a cheaper band has heart rate and accelerometer only. Under a conventional supervised framing these are two different input spaces requiring two different models.

Under a cross-modality imputation framing they are the same model with different masks. LSM's evaluation covers imputation "both across time and sensor modalities", which means the missing skin-temperature channel is handled by exactly the machinery that handles the missing four hours — you mark it as absent and the model infers what it would have said, from the channels that are present.

One objective, three deployment problems. And notice that this is precisely the generalisation of Chapter 3: LIMU-BERT masks contiguous timesteps in a 6-second window, LSM masks hours or whole channels in a multi-day window, and the loss is the same loss.

Three things that stay hard no matter the scale

Hard thing 1: rare events are still rare. Scaling the unlabelled pool does nothing for a condition with a base rate of one in ten thousand. With 165,000 participants you might have 16 positive cases, and no self-supervised objective conjures more. This is why the Apple behavioural work evaluates across 57 tasks rather than one: breadth is the available response to depth being unattainable.

Hard thing 2: people change. Every model in this lesson assumes the mapping from motion to meaning is stationary. Over three years of longitudinal data — the span of the Apple Heart and Movement Study cohort — participants age, get injured, recover, change jobs, change devices. A representation trained on year one is being applied to a slightly different person in year three. This is distribution shift within a subject, and holding out subjects does not test for it. Holding out time does.

Hard thing 3: withdrawal is a hard technical constraint. Consent-based studies grant participants the right to leave and have their data deleted. Deleting a row from a table is easy; removing a participant's contribution from a model already trained on 2.5 billion hours is not. The practical answer today is periodic retraining on the current consented cohort, which turns a research artefact into a maintained system with an ongoing cost. That is a genuine engineering consequence of an ethical commitment, and it belongs in the architecture discussion rather than an appendix.

Evaluating a foundation model when there is no single task

At small scale, evaluation is easy to state: one dataset, one label set, one accuracy number. At population scale that framing dissolves, and the Apple behavioural work makes the replacement explicit — it reports across 57 health-related tasks.

Why 57 rather than one? Because a foundation model's claim is not "I am good at task X," it is "I am a better starting point for whatever task you bring." That claim can only be tested by bringing many tasks, and it changes what a result looks like:

The Apple result has a second layer worth extracting. It compares foundation models built on behavioural signals — sleep, activity, and other derived quantities aligned to physiologically relevant timescales — against models built on low-level sensor data, and reports that the behavioural model excels on behaviour-driven tasks like sleep prediction and improves further when combined with representations of raw sensor data.

That last clause is the finding. The two representations are complementary rather than competing, which is exactly what Chapter 6's timescale argument predicts: one encodes what happened in the last few seconds, the other what has been happening for weeks, and a health question usually needs both.

Federation, and why the training data may never move

There is one more thing that changes at population scale, and it is a constraint rather than a capability. Continuous physiological and motion data from 165,000 people is among the most sensitive data any company holds. The consent structures are explicit — Apple's models are trained on data collected under informed consent through the Apple Heart and Movement Study — and the engineering follows from the constraint, not the other way around.

Two architectural consequences follow directly from what you have already derived:

Small encoders are a privacy affordance, not just a power one. A 51,552-parameter LIMU-BERT encoder or a 0.85-million-parameter IMU tower can run entirely on the device. If the embedding is computed locally, the raw signal never has to leave. What leaves, if anything, is a 512-dimensional vector.

But an embedding is not anonymisation. This is the point most product discussions get wrong. A representation trained to reconstruct your masked footsteps, or trained with a participant-level contrastive objective, encodes gait identity by construction. LIMU-BERT's released datasets ship a user_label column with 9, 24, 30, and 10 distinct values precisely so that this can be probed. Transmitting embeddings instead of raw signals reduces bandwidth and reduces the legibility of what you send. It does not, on its own, reduce identifiability.

Cold start, and the case for personalisation

Population-scale pretraining solves one problem and creates another. A model trained on 165,000 people is good at the average person, and every individual user is not the average person.

Quantify the gap with what Chapter 2 already gave us. Subject A's walking has σ_z = 2.2655; subject B's is 1.2286. The population mean of that feature might be 1.75 with a between-subject standard deviation of 0.5. So subject A sits at (2.2655 − 1.75)/0.5 = +1.03σ and subject B at (1.2286 − 1.75)/0.5 = −1.04σ. Both are ordinary people, two standard deviations apart.

A population model must be right for both, which means it must have learned an invariance that discards the difference. That is achievable — it is exactly what scale buys — but discarding the difference is not always what you want. A health application often cares precisely about this person's deviation from their own baseline, which is information the invariance threw away.

Three responses, with their costs:

Notice that the fourth row is the only one that needs neither labels nor a long wait for supervision — it needs only enough of the user's own unlabelled data to form a centroid, which a wearable accumulates passively. This is why so many shipped wearable health features are framed as "your baseline has changed" rather than "you have condition X": the first is achievable from embeddings and passive data, and the second is a labelling problem nobody has solved at scale.

The federation arithmetic

If raw data must not leave the device, the training has to come to the data. Federated learning is the standard answer, and its cost is communication rather than compute. Put numbers on it for the models in this lesson.

In federated averaging, each round every participating device downloads the current model, trains locally, and uploads a model-sized update. So the per-device traffic per round is roughly twice the model size:

Now scale it. A thousand devices participating in each of a hundred rounds, with the LIMU-BERT-sized model:

Entirely feasible. Repeat with the IMU2CLIP tower and it is 680 GB, which is expensive but conceivable. Repeat with the CLIP image tower and it is 70 terabytes over metered mobile connections, which ends the conversation.

So the same asymmetry that made the deployment work makes the training story work too. The frozen teacher lives on a server and never federates; the small student is the only thing small enough to travel. If you were designing this system from the privacy constraint backwards, you would arrive at exactly IMU2CLIP's architecture — not because of accuracy, but because 412 kB fits on a phone's data plan and 352 MB does not.

Chapter 7: The Leakage Traps

This chapter will save you more real-world pain than any other in the lesson. Human activity recognition has a reputation for published numbers that do not survive contact with a new user, and the cause is almost never the model. It is the split.

Trap 1: overlapping windows, split at random

Return to Chapter 1's arithmetic. Ten minutes of recording, 50 Hz, 128-sample windows, 50% overlap, giving 467 windows. Consecutive windows share 64 of their 128 samples.

Now shuffle all 467 and split 80/20 at random. Take any test window. Its left neighbour and its right neighbour each share half their samples with it. What is the probability that both neighbours also landed in the test set, leaving it uncontaminated?

Ninety-six percent of your test windows have a training window that contains half of their exact samples. Not "similar" samples. The same floating-point numbers, from the same instant, of the same footstep.

Any model with enough capacity can exploit this. It does not need to learn what walking is; it needs to learn to recognise this particular sequence of 64 numbers, which it has already memorised. The reported accuracy is then a measurement of memorisation, and it will be high, and it will be worthless.

The fix costs one line: split by time, not by window. Cut the recording into contiguous blocks first, assign whole blocks to train or test, and window within blocks. Or simply drop the overlap for evaluation.

Trap 2: the same subject on both sides

This one is subtler and more damaging, because fixing Trap 1 does not fix it.

Suppose you split by time: the first eight minutes of each recording is training, the last two is test. No window overlaps a boundary. And the number is still inflated, because the same person appears on both sides. Your gait has an individual signature — the exact asymmetry between left and right steps, the precise cadence, the way you swing an arm. A model can learn "this is subject 7, and subject 7 walks like this" and score beautifully, while having learned nothing that transfers to subject 31.

The correct protocol is a subject-wise split, often called leave-one-subject-out (LOSO): every window from a given person goes entirely to train or entirely to test. UCI HAR's official protocol does exactly this, partitioning its 30 volunteers into 21 for training and 9 for testing, and this is why the released LIMU-BERT dataset configs carry a user_label_index at all — the subject identity is preserved through preprocessing so it can be used to split.

Two supporting observations make the danger concrete:

The representation demonstrably encodes identity. The datasets ship user-identification as a target task with 9, 24, 30, and 10 classes. That column exists because subject identity is recoverable from these embeddings. If it is recoverable, a classifier can and will use it as a shortcut.

IMU2CLIP splits by video ID, not by window. Its dataset table separates 1,444 training, 161 validation, and 688 test media files, and derives windows within those partitions. That is the right structure, and it is why their numbers are quotable.

Drag the subject-signature strength to zero and the three protocols converge, because there is no identity to memorise. Drag it up and they diverge. That slider is the whole phenomenon: leakage is only harmful in proportion to how much identity-specific structure exists in the signal — and in gait, there is a great deal.

The cross-validation trap, which is worse than it looks

The instinct when a test set is small is to run k-fold cross-validation. On overlapping windows, this makes leakage worse, and the arithmetic is short enough to be startling.

With random k-fold, a given neighbour window lands in the same fold as your test window with probability 1/k, and therefore in a training fold with probability 1 − 1/k. Each test window has two neighbours, so

More folds means a larger training set, which means a higher chance that any particular neighbour is in it. Leave-one-out cross-validation — the protocol people reach for when data is scarcest — is the most contaminated of all: every single test window has both its neighbours in training, guaranteed.

The remedy is grouped cross-validation, where the fold assignment is made over subjects (or recordings, or contiguous blocks) rather than over rows. Scikit-learn ships GroupKFold and LeaveOneGroupOut for exactly this, and using them is a one-word change that typically costs you several points of headline accuracy and buys you a number that means something.

What an honest report looks like

Numbers in this field are only comparable if the protocol is stated. Here is the minimum set of facts a result needs to carry, using the papers in this lesson as the worked examples.

Trap 3: accuracy on an imbalanced problem

We already met this one in the data. IMU2CLIP's Aria random-init baseline reports 76.11% accuracy and 56.35 F1. On Ego4D the same model reports 49.92% accuracy and 23.23 F1.

Work out what a 49.92%/23.23 pair implies for a four-class problem. Macro-F1 averages the per-class F1 scores equally. If a model predicted the majority class always, and that class held about half the data, accuracy would be near 50% while three of four per-class F1 scores would be zero and the fourth around 0.67, giving macro-F1 near 16.7. The observed 23.23 is above that but not far above it — the model is doing a little better than the degenerate solution, and accuracy alone would have hidden it completely.

Report both, always. Accuracy tells you about the population; macro-F1 tells you whether the minority classes exist for the model at all. In health and safety applications the minority class is usually the one you care about — falls, arrhythmias, seizures — and it is by definition rare.

How much does overlap leakage actually buy a cheater?

The probability calculation says 96% of test windows are contaminated. It does not say how much that is worth. Bound it with a thought experiment you can compute.

Consider the dumbest possible cheating model: a 1-nearest-neighbour classifier over raw windows, using Euclidean distance. It has learned nothing; it simply memorises the training set and copies the label of the closest stored window.

Take a test window w_k. Its neighbour w_k−1 is in the training set with probability 0.8, and the two share samples 64 through 127 exactly. Write the squared distance between them:

Half the sum is identically zero, by construction. Now compare against a genuinely different window from another subject performing the same activity: none of its 128 samples match, so every term contributes. If the per-sample squared difference averages δ in both cases, then

The overlapping neighbour is 29% closer before accounting for the fact that it is also the same person doing the same thing at the same moment, which shrinks the remaining 64 terms too. In practice the overlapping neighbour wins the nearest-neighbour contest essentially always, so a memorising model scores close to

Ninety-six percent accuracy from a model with no parameters and no understanding. That is the ceiling a random window split hands to anyone who wants it, and a deep network with sufficient capacity will find it just as reliably as a nearest-neighbour classifier will — it just will not tell you that it did.

Small test sets, and what to report instead of a point estimate

Chapter 5 computed that 241 test windows give a ±6-point interval. When the test set is that small, a single number is close to useless on its own. Three cheap remedies:

Bootstrap the interval. Resample the test set with replacement a few thousand times, recompute the metric each time, and report the 2.5th and 97.5th percentiles. This costs milliseconds and handles macro-F1 correctly, where the closed-form binomial interval does not apply because F1 is not a mean of independent Bernoulli trials.

Compare models on the same items, not on their summaries. Two models evaluated on identical windows have correlated errors, so their difference has a much tighter interval than either score does. Build the 2×2 table of agreements and disagreements and test that; McNemar's test is the standard tool and takes one line.

Report the whole leave-one-subject-out distribution. With subject-wise splits you get one score per held-out subject. Report all of them, or at least the median and range, because the spread across subjects is the number that predicts what a new user will experience — and it is routinely far wider than the confidence interval on the mean.

python — the three diagnostics, togetherimport numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score

# 1 — the leakage canary. If this is high, your split is broken.
flat = train_X.reshape(len(train_X), -1)                    # (N, 128*6)
nn = KNeighborsClassifier(1).fit(flat, train_y)
print("1-NN on raw windows:", nn.score(test_X.reshape(len(test_X), -1), test_y))

# 2 — bootstrap interval on macro-F1 (works for any metric)
rng = np.random.default_rng(0)
boots = []
for _ in range(2000):
    idx = rng.integers(0, len(test_y), len(test_y))
    boots.append(f1_score(test_y[idx], pred[idx], average="macro"))
print("macro-F1 95%% CI:", np.percentile(boots, [2.5, 97.5]))

# 3 — the per-subject spread, which is what a new user will feel
per_subj = [f1_score(test_y[test_subj == s], pred[test_subj == s], average="macro")
            for s in np.unique(test_subj)]
print("per-subject F1: min %.3f  median %.3f  max %.3f"
      % (min(per_subj), np.median(per_subj), max(per_subj)))

Trap 4: device and placement leakage

A close cousin of subject leakage. HHAR ships a model_label with 3 values; Shoaib ships a position_label with 5. If phone model A appears in both train and test, the network can key on that device's particular noise floor and calibration offset. If body position "wrist" appears in both, it can key on the orientation prior rather than the motion.

The honest protocol for a product claim is to hold out the axis you intend to generalise over. If you will ship on new phones, hold out phone models. If you will ship to new users, hold out users. If both, hold out both, and expect the number to fall.

The label protocol is itself a modelling choice

One decision hides upstream of every metric, and papers rarely state it: what label does a window get when the activity changes inside it?

Three protocols are in common use, and they produce different datasets from identical recordings.

Protocol A: discard straddlers. Keep a window only if every sample carries the same label. This is what LIMU-BERT's released merge_dataset does in all mode. It is the cleanest, and it is why the UCI set yields only 2,088 windows.

Protocol B: majority label. Assign whichever label covers the most samples. Keeps more data, and creates windows whose content is 51% one thing and 49% another, labelled unambiguously. Those windows are effectively noise, and at a 2.56-second window length with typical activity durations they can be several percent of the set.

Protocol C: label the centre. Take the label at the window's midpoint. Keeps everything and is at least self-consistent, but a window whose activity changed at 60% through gets a label describing only its first part.

Quantify the difference. Suppose a recording alternates between activities with a mean bout length of 12 seconds, and you use 2.56-second windows with a 1.28-second hop. A transition falls inside a window whenever the window's start lies within 2.56 seconds before it. Transitions occur every 12 seconds on average, so the fraction of windows containing a transition is roughly

One window in five is impure. Protocol A throws all of them away, which is a 21% reduction in dataset size and a systematic bias — you have specifically removed all the transitions, so your evaluation never tests the moments that are hardest and most operationally important. Protocol B keeps them and mislabels roughly half. Neither is wrong; both change what the number means.

Why windows are not independent samples

One more statistical point that underlies all of the above. Every confidence interval in this lesson assumed independent samples. Windows from a continuous recording are emphatically not independent, and the dependence has a measurable timescale.

Consider the standard deviation feature across consecutive windows during steady walking. It changes slowly — the person's cadence and vigour are roughly constant over tens of seconds — so the autocorrelation between window k and window k + 1 is high, perhaps 0.9, and decays over several windows.

The consequence is the effective sample size. For a first-order autocorrelated series with correlation ρ between adjacent samples, the effective number of independent samples is approximately

At ρ = 0.9 and n = 467 windows from ten minutes of recording:

Four hundred and sixty-seven windows are worth about twenty-five independent observations. Which means confidence intervals computed as if the windows were independent are too narrow by a factor of √(467/24.6) = 4.4.

This is the deep reason subject-wise splitting is the right unit. Windows are correlated within a recording; recordings are correlated within a subject; subjects are approximately independent. The unit of independence is the person, so the person is what you split on and the person is what you count when you ask how much evidence you have. A study with 30 subjects has roughly 30 independent observations, no matter how many windows it slices them into.

Trap 5: the pretraining set touching the test set

The trap unique to the self-supervised era, and the easiest to create by accident.

The whole appeal of Chapter 3 is that pretraining uses unlabelled data, of which you have an unlimited supply. So the temptation is to pretrain on everything, then split the labelled subset for evaluation. But if a test subject's windows were in the unlabelled pretraining pool, the encoder has already seen — and been optimised to reconstruct — that person's gait. The label was withheld; the signal was not.

LIMU-BERT's released scripts contain a small guard against this that is easy to miss. Both pretrain.py and classifier.py take a training_rate parameter, and the README states plainly that "this value must be equal to the training_rate in pretrain.py." The same partition is used in both phases, so the classifier's evaluation data was excluded from pretraining too. One shared constant, enforcing a whole methodology.

python — an evaluation harness that does not lie to youimport numpy as np

def honest_split(subjects, devices, seed=0, test_frac=0.3):
    """Hold out whole SUBJECTS (and optionally whole DEVICES).
    Returns boolean masks over windows — never over samples."""
    rng = np.random.default_rng(seed)
    uniq = np.unique(subjects)
    n_test = max(1, int(round(len(uniq) * test_frac)))
    held = rng.choice(uniq, size=n_test, replace=False)
    test = np.isin(subjects, held)
    return ~test, test

train_m, test_m = honest_split(subject_ids, device_ids)

# CRITICAL: pretraining sees ONLY the training partition.
# Reusing the same mask in both phases is the whole discipline.
encoder = pretrain_masked(unlabeled_windows[train_m])
H_train = encoder(labelled[train_m], output_embed=True)
H_test  = encoder(labelled[test_m],  output_embed=True)

from sklearn.metrics import f1_score, accuracy_score
pred = clf.fit(H_train, y[train_m]).predict(H_test)
print("acc", accuracy_score(y[test_m], pred),
      "macro-F1", f1_score(y[test_m], pred, average="macro"))
# Report BOTH. Report the subject IDs you held out. Report the seed.

All five traps, as one pre-flight function

Every trap in this chapter can be checked before you train anything. Here is the whole chapter as code you can paste into a project and run once.

python — run this before you trust any HAR numberimport numpy as np
from sklearn.neighbors import KNeighborsClassifier

def preflight(X, y, subject, device, position, train_m, test_m, hop, T):
    """X: (N, T, C) windows. train_m / test_m: boolean masks."""
    ok = True

    # Trap 1 — overlap. Any window index adjacent across the split?
    if hop < T:
        idx = np.arange(len(X))
        adj = np.intersect1d(idx[train_m], np.concatenate([idx[test_m]-1, idx[test_m]+1]))
        if len(adj):
            print("FAIL trap 1: %d train windows overlap a test window" % len(adj)); ok = False

    # Trap 2 — subject leakage.
    shared = np.intersect1d(np.unique(subject[train_m]), np.unique(subject[test_m]))
    if len(shared):
        print("FAIL trap 2: %d subjects appear on BOTH sides" % len(shared)); ok = False

    # Trap 3 — imbalance. Report the majority-class rate as the real floor.
    _, counts = np.unique(y[test_m], return_counts=True)
    print("majority-class accuracy on test: %.3f" % (counts.max() / counts.sum()))

    # Trap 4 — device / position leakage (warn, do not fail).
    for nm, arr in (("device", device), ("position", position)):
        if arr is not None and len(np.intersect1d(np.unique(arr[train_m]), np.unique(arr[test_m]))):
            print("WARN trap 4: %s values shared across the split" % nm)

    # Trap 5 canary — a parameterless memoriser. Should be near chance.
    flat = X.reshape(len(X), -1)
    nn = KNeighborsClassifier(1).fit(flat[train_m], y[train_m])
    canary = nn.score(flat[test_m], y[test_m])
    print("1-NN canary: %.3f  (near chance = clean; near 1.0 = leaking)" % canary)
    if canary > 0.85:
        print("FAIL trap 5: raw-window memorisation solves your test set"); ok = False

    # Effective sample size, so you know how much evidence you actually have.
    print("independent units (subjects) in test: %d" % len(np.unique(subject[test_m])))
    return ok

Three of these checks fail loudly, two warn, and the last simply prints the number that determines your confidence interval. Running it costs seconds. Not running it has cost this field a decade of results that did not transfer.

A reproduction checklist, applied to a real number

Take IMU2CLIP's 65.87% Ego4D fine-tuned accuracy and ask what you would need to reproduce it. Working through this makes every abstraction in this chapter concrete.

Nine of eleven are fully specified, which by the standards of this field is excellent — and note why it is excellent: the specification lives in released code and configuration files rather than in prose. A YAML file cannot be vague about a batch size.

The two soft spots are instructive. The label derivation ("soft annotations for Ego4D via text matching of the narrations") is a procedure whose details would change the class boundaries, and the seed is unreported on a test set small enough that seed variation is not negligible. Neither undermines the paper's conclusion, because the conclusion is a large, consistent, multi-column pattern rather than one cell. But if you set out to reproduce exactly 65.87, those two are where you will end up.

Chapter 8: On the Device

Every architectural choice in this lesson was made under a power budget. This chapter makes the budget explicit, because once you can compute it, the designs stop looking arbitrary and start looking inevitable.

Counting the arithmetic in LIMU-BERT

Take the released config: sequence length 120, hidden 72, feed-forward 144, 4 heads, 4 layer applications of one shared block. Count multiply-accumulate operations (MACs) for a single window; a MAC is conventionally two FLOPs.

So one 6-second window costs about 28.25 million MACs, or roughly 56.5 MFLOPs. Now the rate. With 50% overlap you emit a window every 3 seconds:

That is a small number by any modern standard. A mid-range phone application processor does that in a fraction of a millisecond. Even a microcontroller-class DSP with vector extensions, running at a hundred megahertz and retiring a couple of MACs per cycle, clears it with an enormous margin — which means the inference can happen in a short burst and the core can go back to sleep.

Memory: 51,552 parameters at float32 is 206 kB; quantised to int8 it is about 52 kB. Activations for one window at (120, 72) float32 are 34.6 kB, or 8.6 kB at int8. The whole working set fits comfortably in on-chip SRAM, which is the thing that actually matters — a DRAM access can cost two orders of magnitude more energy than the multiply it feeds.

Counting the arithmetic in IMU2CLIP's tower — and its teacher

Now the other model, on a 5-second 200 Hz window (1000 timesteps), using the shapes we derived in Chapter 4.

The GRU is 87% of the arithmetic, which is worth pausing on. The convolutions do the temporal work — they are what turn 1000 timesteps into 32 — and they cost almost nothing. The expense is entirely in the 512-wide recurrent state, and the only reason it is 512 wide is that CLIP's embedding dimension is 512. The output dimension of a frozen external model is dictating 87% of your on-device compute budget. If you wanted this cheaper, the first move is not a better IMU architecture; it is a smaller shared space.

Now compare against the teacher it is imitating. CLIP ViT-B/32 costs on the order of several GFLOPs per 224×224 frame. Even in IMU2CLIP's most generous configuration — the released clip_1frame video encoder, one single frame per window — the image tower is roughly 70× the arithmetic of the entire IMU tower. Give the video branch the temporal treatment it deserves, say 10 frames per 5-second window, and the ratio passes 700×. And none of that counts the image sensor, the ISP, or the memory traffic for the frames themselves.

That asymmetry is the deployment story in one diagram. Text embeddings for a vocabulary of candidate activities are computed once on a server and shipped as a fixed matrix. Adding a new searchable phrase costs one text-encoder forward pass at build time and zero device compute, because the device only ever computes dot products against a cached table.

Where the power actually goes

An important correction to intuition: at these model sizes, inference is usually not the dominant cost. Four other things compete.

The sensor itself. Running a 6-axis IMU continuously at 50 Hz costs a fraction of a milliwatt to a few milliwatts depending on the part and mode. It is small, but it is always on, and always-on beats occasionally-expensive on a daily energy budget.

Waking the host processor. This is the one that kills naive designs. If the application processor wakes for every single sample, it wakes 50 times per second, and the wake-plus-sleep transition costs far more than the work done. The fix is hardware FIFO batching: the sensor buffers samples internally and interrupts only when the buffer is full. At 50 Hz with a 128-sample window, the FIFO fills in 2.56 seconds, so the host sleeps for essentially all of that interval and wakes once. One wake every 2.56 s instead of 128 wakes is a 128× reduction in transitions.

Radio. Sending data off-device is expensive per byte, and the ratio here is decisive. One window is 128 × 6 × 4 = 3,072 bytes raw. Its embedding is 512 × 4 = 2,048 bytes at float32, or 512 bytes at int8 — a 6× reduction against the raw window, and against a 6-second LIMU-BERT window the compression is larger still. Computing locally and transmitting the vector is usually cheaper in total energy than transmitting the signal, and this only becomes more true as radios stay expensive while compute gets cheaper.

Duty cycling. The cheapest inference is the one you skip. Almost every IMU part offers a hardware significant-motion or any-motion interrupt that fires only when the device actually moves. Gate the encoder behind it and a stationary device runs the model zero times per hour instead of 1,200.

Where the energy really goes: memory, not multiplies

One number reframes every on-device optimisation you will ever do. Using the standard published energy tables for a 45 nm process as an order-of-magnitude guide:

Fetching a number from DRAM costs roughly 130 times what multiplying it costs. Which means the real question about any on-device model is not "how many MACs" but "how many times does a weight cross the chip boundary."

Run that number for LIMU-BERT. Its encoder is 51,552 parameters, which at int8 is about 52 kB. Modern microcontroller-class SRAM ranges from a few hundred kilobytes to a couple of megabytes, so the entire model fits on-chip and is loaded once. Every one of the 28.2 million MACs then reads its weight from SRAM at ~5 pJ rather than DRAM at ~640 pJ.

Compare with a model that does not fit. Suppose the weights had to be streamed from DRAM once per window:

A factor of 128 in energy, from a decision about where the weights live rather than what they compute. And the parameter sharing from Chapter 3 now looks different: four independent layers would be 169,056 parameters, about 170 kB at int8, which pushes past the SRAM budget of many small parts. Sharing was not just a regulariser; it was the thing that kept the model on-chip.

Quantisation error, worked

Since int8 is doing so much work in this story, it is worth knowing what it costs numerically. Symmetric per-tensor quantisation picks a scale from the largest magnitude in the tensor:

Take a weight tensor whose largest entry is 0.42. Then s = 0.42/127 = 0.0033071. Quantise the weight w = 0.1234:

Rounding error is uniform on [−s/2, +s/2], so its RMS is s/√12 = 0.0033071/3.4641 = 0.000955. Now push that through a 72-term dot product with activations of RMS 1 and weights of RMS 0.15. Errors add in quadrature:

Under one percent per layer, compounding modestly across four layer applications. Classification margins are almost always much larger than that, which is why int8 post-training quantisation usually costs a fraction of a point of accuracy on models like this.

The place it does break is instructive: per-tensor scaling is set by the single largest weight, so if one channel has an outlier ten times the rest, every other channel gets a scale ten times too coarse and loses 3.3 bits of resolution. The fix is per-channel scales, which every modern toolkit supports and which costs one float per output channel. If your quantised model loses more than a point, look for the outlier channel before you look at anything else.

Turning MACs into milliwatt-hours

Arithmetic counts only become decisions once they are energy. Use a rough but standard figure: an int8 multiply-accumulate on a modern mobile accelerator costs on the order of a few picojoules, including its share of the memory traffic. Take 3 pJ per MAC and run the numbers for a device doing continuous LIMU-BERT inference.

Twenty-eight microwatts. Over a full day:

Now put that against a smartwatch battery of roughly 300 mAh at 3.8 V, which is 1,140 mWh. Continuous inference consumes 0.06% of the battery per day. It is, for practical purposes, free.

Which means the model was never the thing to optimise. Compare the other terms on the same daily basis:

A profiling protocol that produces trustworthy numbers

Every figure in this chapter is an estimate from published architecture configs and standard energy tables. Before shipping anything, measure. Here is a protocol that avoids the four ways these measurements usually go wrong.

Measure at the battery, not in software. On-device power APIs report coarse, smoothed estimates. A bench supply or an inline current meter sampling at a kilohertz shows you the actual current spikes, and the spikes are where the energy is. If you cannot instrument the hardware, at minimum measure over hours, not seconds.

Measure the delta, not the absolute. Run the device for an hour with the pipeline off, then an hour with it on, under identical conditions — same screen state, same radio activity, same temperature. The difference is your model's cost. Absolute readings are dominated by everything else on the device.

Measure with the real duty cycle. A benchmark that runs inference in a tight loop measures throughput, which is not what you deploy. Chapter 8's cascade reaches the encoder on 12% of windows; a tight loop reaches it on 100%, overstating cost by eight times and, worse, hiding the wake-transition cost that only appears when the processor is allowed to sleep in between.

Measure the whole pipeline, including the parts you did not write. Sensor configuration, FIFO handling, the driver, the resampler, the feature extraction, the model, and the consumer of its output. It is common for the resampling code — a Python-style loop that someone never optimised because "it is just preprocessing" — to cost more than the neural network.

That last row is worth internalising alongside Chapter 8's arithmetic. We computed that LIMU-BERT costs 28.2 million MACs per window. A naive Python-style loop implementing the one-pole gravity filter over a 120-sample, 6-channel window costs 720 iterations with per-element overhead — nominally trivial arithmetic that can nonetheless run slower than the entire transformer if it is not vectorised. Profile the pipeline, not the tensor.

The latency budget, and why it is not what you think

A second quantity people optimise prematurely. Suppose your model runs in 2 ms. What is the end-to-end latency from a movement happening to a classification appearing?

The inference is 0.03% of the latency. Everything else is waiting for enough signal to exist. You cannot classify a gait cycle before a gait cycle has happened, and you cannot batch the sensor FIFO without adding the batch interval to the delay.

The only real lever is the window length, and shortening it costs you accuracy for exactly the reason Chapter 1 gave: a shorter window contains fewer gait cycles and has coarser spectral resolution. IMU2CLIP's released config uses 2.5-second windows where the paper's tables use 5-second ones — a direct latency-versus-context trade, made explicit in a YAML file.

The cascade, which is how this is actually deployed

Nothing in production runs one model all the time. The pattern is a cascade: a very cheap gate decides whether the expensive model runs at all. Put numbers on a three-stage version.

Stage 0 — hardware motion interrupt. The IMU itself decides whether anything moved, in silicon, costing effectively nothing above the sensor's own draw. A typical desk-worker's device is stationary perhaps 70% of the day, so this alone removes 70% of the work.

Stage 1 — a three-feature gate. On the remaining 30%, compute the magnitude's mean, standard deviation, and zero-crossing rate — a few hundred operations — and pass through only windows that look like structured activity rather than a pocket jostle. Suppose that is 40% of what reaches it.

Stage 2 — the full encoder. Runs on whatever survives both gates.

An 8.3× reduction, from two decisions that cost almost nothing to make. And because Chapter 8 already showed that inference is 0.06% of the battery, the honest reading is that the cascade is not primarily about energy — it is about thermal headroom, about leaving the application processor asleep, and about the fact that on many wearables the sensor hub can run stages 0 and 1 without ever waking the main core.

Quantising an encoder without breaking it

python — post-training quantisation, and the one thing to watchimport torch

encoder.eval()
q = torch.ao.quantization.quantize_dynamic(
        encoder, {torch.nn.Linear, torch.nn.GRU}, dtype=torch.qint8)

# 51,552 params: 206 kB float32 → ~52 kB int8.
# Dynamic quantisation handles weights; activations stay float.

# The thing that actually breaks: INPUT SCALE.
# Accelerometer values live near ±20 m/s², gyroscope near ±35 rad/s.
# A single int8 scale for all six channels wastes most of the range
# on whichever channel is quiet. This is exactly why the released
# IMU2CLIP encoder opens with GroupNorm(2, 6) — accel and gyro are
# normalised as separate groups BEFORE anything else touches them.
def prepare(win):                # win: (6, T)
    acc, gyr = win[:3], win[3:]
    acc = (acc - acc.mean()) / (acc.std() + 1e-6)
    gyr = (gyr - gyr.mean()) / (gyr.std() + 1e-6)
    return torch.cat([acc, gyr], dim=0)   # now both groups are O(1)

That comment block is the practical heart of the chapter. A design decision made for statistical reasons in Chapter 4 — normalise accelerometer and gyroscope independently so neither dominates the loss — turns out to be exactly the decision that makes int8 quantisation viable. Good architectural choices tend to pay twice.

The memory layout of a window

One last piece of concreteness, because it is where a surprising amount of on-device performance actually goes.

A window is stored as a contiguous block of numbers, and there are two ways to lay it out. Time-major, (T, C), puts the six channels of timestep 0 adjacent, then the six of timestep 1. Channel-major, (C, T), puts all 120 values of channel 0 adjacent, then all 120 of channel 1.

These are transposes of each other and they are not interchangeable in performance. A 1-D convolution slides along time within one channel, so it wants channel-major: its inner loop then walks consecutive addresses and every cache line it fetches is fully used. Give it time-major data and each successive value it needs is 6 floats away, so it touches six times as many cache lines to read the same numbers.

Meanwhile a transformer's first operation is a linear map across the 6 channels at each timestep, which wants time-major for exactly the mirrored reason.

Which is precisely why the two architectures in this lesson use opposite conventions — LIMU-BERT ships (N, W, F) and IMU2CLIP ships (B, 6, window) — and why an unnecessary transpose in a preprocessing loop can quietly cost more than the layer it feeds.

Note the last row. The hardware hands you time-major data, because that is the order it is produced in. Every channel-major pipeline therefore pays for one transpose somewhere, and the only question is whether it happens once per window in optimised code or repeatedly inside a loop that nobody profiled.

Chapter 9: Lineage and Cheat Sheet

Step back and look at what actually moved. In 2015 the question "which of my six activities is this" required a labelled dataset per question. In 2025 the question "find the moment I picked something up off the floor" is a dot product against a cached matrix. Three ideas did that, and none of them was a bigger model.

Idea 1: the label was never the point

Classical HAR treated the class label as the supervision. Masked reconstruction showed that the signal itself is the supervision, and there is unboundedly more of it. LIMU-BERT made this concrete for inertial data with a 51,552-parameter encoder and a config file containing four masking constants; LSM scaled the identical family of objectives — imputation, interpolation, extrapolation — to 40 million hours across six sensor modalities.

Idea 2: another modality can donate its semantics

IMU2CLIP's contribution is not an architecture, it is a wiring diagram. Freeze a model that understands meaning. Train a cheap model to reproduce its outputs from a cheap signal. Inherit, for free, every relationship the frozen model already encoded — which is why an encoder trained only against video can retrieve motion from English text it never saw.

ImageBind generalised this the following year, binding six modalities — images, text, audio, depth, thermal, and IMU — into one space, and demonstrating that only image-paired data is necessary: every modality binds to images, and thereby to each other. IMU is one of its six. The pattern that started as "IMU imitates CLIP" became "everything imitates images."

The same three moves, in four other modalities

Nothing in this story is specific to accelerometers. The pattern — hand features, then self-supervision, then binding to a language-aligned space — has replayed in every modality that had more raw data than labels. Seeing it laid out makes the shape obvious, and makes it predictable where it will happen next.

Two things are worth extracting from that table.

The lag is real and it has a cause. BERT is 2018; LIMU-BERT is 2021. CLIP is 2021; IMU2CLIP is 2022. IMU trailed by roughly three years, and not because the ideas were hard to transfer — LIMU-BERT is explicitly BERT's recipe with one change. It trailed because the paired data did not exist until head-mounted devices with simultaneous video and IMU became common enough to build Ego4D from. The bottleneck was hardware deployment, not algorithms.

Every modality needed its own version of the "one change." BERT masks single tokens; LIMU-BERT had to mask spans, because continuous signals are interpolable and text is not. CLIP pairs images with captions; IMU2CLIP had to pair through video, because nobody captions accelerometer traces. In each case the transfer was 90% mechanical and 10% a domain insight, and the 10% is what the paper is about.

Idea 3: scale buys invariance, not knowledge

The wearable foundation models are not learning more about walking. They are learning that walking looks different on 1,100 phone models, on 165,000 bodies, at five mounting positions, under a hundred firmware filters. LIMU-BERT-X's 1.43 million hours from 60,000 subjects on 1,100 phone models is a statement about nuisance variation, not about physics.

Six things people believe that this lesson should have dislodged

"An accelerometer measures motion." It measures proper acceleration, which at rest is 9.81 m/s² upward. Roughly the entire signal, most of the time, is gravity in a rotating frame. Chapter 1.

"More labelled data is the answer." The benchmarks have 9 to 30 subjects and 2,000 to 10,000 windows, and they have had that for a decade. The unlabelled pile grew by six orders of magnitude in the same period. Every advance in this lesson came from learning to use the second pile. Chapters 3 and 6.

"Self-supervision means no leakage." Pretraining on a test subject's recordings fits that subject's gait, with or without labels. The label was withheld; the signal was not. Chapter 7.

"Masked modelling works the same everywhere." On a smooth continuous signal, BERT-style scattered masking is solved by linear interpolation — the loss falls to 0.43 on our toy window without the model learning anything. Contiguous spans raise the trivial baseline to 3.77 and create the gap a real encoder must fill. Chapter 3.

"Zero-shot works because the model understands the classes." It works because the class names, wrapped in caption-shaped templates, land near the right region of CLIP's text space. When the class names denote scenery rather than kinematics — "hiking" — zero-shot falls below a randomly initialised model. Chapter 5.

"On-device cost means FLOPs." LIMU-BERT inference costs about 0.06% of a smartwatch battery per day. Waking the host processor for every sample costs tens of percent, and streaming raw data over the radio costs about 9%. The model was never the bottleneck. Chapter 8.

What is still open

Four problems that anyone working here in the next few years will run into.

Temporal direction. IMU2CLIP's targets are single video frames, so "picking up" and "putting down" produce nearly identical supervision. Any system built on frame-level alignment inherits that blindness. Fixing it needs a temporal video encoder as the teacher, which multiplies the training cost of the branch that was supposed to be free.

The two timescales. Chapter 6 showed that the second-scale models and the day-scale models cannot share a context window. Nothing in the literature yet cleanly bridges "what movement is happening now" and "how has this person been over the last month," and the obvious answer — hierarchical models where second-scale embeddings become day-scale tokens — has not been demonstrated at population scale.

Identity leakage as a first-class property. Every objective in this lesson produces representations that encode who you are, because individual gait is exactly the structure the objectives reward learning. Nobody has shown how to build an embedding that is discriminative for activity and provably uninformative about identity, and the fact that these datasets ship user-identification as a benchmark task is a standing reminder of the gap.

Evaluation that predicts deployment. Subject-wise splits are necessary and not sufficient. The axes that matter in production — new device models, new mounting positions, new populations, the same person a year later — are rarely all held out at once, and no benchmark in this field currently does so.

The cheat sheet

Every equation in this lesson, on one page

If you retain nothing else, retain these. Each one appeared with a worked numeric example, and the chapter is named so you can go back.

The window count (Ch 1). How many windows a recording yields:

The Nyquist limit (Ch 1). Content above f_s/2 folds down and cannot be recovered:

The gravity filter (Ch 1). One pole, one constant, derived from the cutoff you choose:

Rotation invariance (Ch 1). Why magnitude survives an arbitrary mount:

Affine invariance of standardisation (Ch 6). Why LayerNorm cancels device bias and gain:

Information gain (Ch 2). How a tree picks the threshold it will later be broken by:

The masked reconstruction loss (Ch 3). Scored only where you hid the signal:

Span-length distribution (Ch 3). Truncated geometric, expectation 3.797 at p = 0.2 and max 10:

Symmetric InfoNCE (Ch 4). The objective that binds two modalities:

Its gradient (Ch 4). The single most useful line in the lesson:

Mean reciprocal rank (Ch 5). Dominated by the top of the list:

The binomial interval (Ch 5). Ten seconds that discards a lot of overclaiming:

Overlap contamination (Ch 7). Why k-fold makes it worse:

Effective sample size (Ch 7). Why the person, not the window, is the unit:

Dead-reckoning error (Ch 0). Why nobody integrates:

A power-law scaling law (Ch 6). Smooth, predictable, and unforgiving:

The six questions to ask of any IMU model you meet

1. What is the window, and what is the rate? Every other number is conditional on these two. A result at 200 Hz over 5 seconds is answering a different question than one at 20 Hz over 6 seconds.

2. How was it split? If the answer is not "by subject" — or better, "by subject and by device" — the headline number is not a generalisation estimate.

3. Was the pretraining pool disjoint from the test set? Self-supervision without labels is still supervision by signal.

4. Accuracy or macro-F1? If only accuracy is reported on an imbalanced problem, you have been told about the majority class.

5. What is the trivial baseline? For a masking objective, what does linear interpolation score? For a classifier, what does the majority class score? For retrieval, what does a strong model in an adjacent modality score on the same pool? IMU2CLIP's inclusion of the CLIP video-text reference row is the standard to hold others to.

6. What runs on the device? In every design in this lesson, the answer is "the small tower only." If a proposal requires the frozen teacher at inference, it has missed the point.

If you wanted to build this yourself

A concrete order of operations, using only what has been derived here. Each step names the constants and the check that tells you it worked.

Step 1 — get a stream and window it. 50 Hz, 6 channels, 2.56-second windows, 50% overlap, contiguous-block splits from the very beginning. The check: a 1-nearest-neighbour classifier on raw windows should score near chance on your test split. If it scores well, your split leaks and everything after this is wasted.

Step 2 — establish the classical baseline. Sixty hand features, a random forest, subject-wise evaluation, accuracy and macro-F1 both reported with a bootstrap interval. This number is your floor, and it is usually higher than people expect. If a learned model cannot beat it, the learned model has a bug.

Step 3 — pretrain with span masking. Hidden 72, feed-forward 144, four shared layer applications, four heads, mask ratio 0.15, spans up to 10, zeros 80% of the time and random values otherwise. Normalise accelerometer and gyroscope as separate groups before anything else. The check: masked-position MSE must fall well below what linear interpolation achieves on the same masks — compute that baseline explicitly, because it is the only honest reference.

Step 4 — freeze and probe. Cache the (N, 120, 72) representations to disk. Train a small GRU on top with 1%, 5%, 20%, and 100% of your labels and plot accuracy against label fraction. That curve, not a single number, is the result: the claim of self-supervision is that it bends the low-label end upward.

Step 5 — if you have a paired modality, align to it. Freeze a pretrained encoder for that modality. Train your IMU tower with the symmetric InfoNCE from Chapter 4, largest batch you can afford, temperature learned from 0.1. The check: retrieval must beat a randomly initialised tower by a wide margin, and you should include an adjacent-modality reference row so your absolute numbers mean something.

Step 6 — measure the deployment, not the model. Quantise to int8 with per-channel scales, confirm the working set fits in SRAM, wire the sensor FIFO, and measure current draw as a delta against an idle baseline over hours. Expect the preprocessing to cost more than the network until you vectorise it.

Where to go next

Five things to remember when everything else has faded

One. An accelerometer at rest reads 9.81, not zero, and separating gravity from motion costs you 17% of a real acceleration within a tenth of a second. Every preprocessing chain in this field encodes a guess about that trade, and the guess was made once, on one dataset.

Two. The labelled data stopped growing a decade ago and the unlabelled data never stopped. Every genuine advance since has been a different way of extracting supervision from data nobody annotated — first from the signal itself, then from whatever happened to be recording alongside it.

Three. On a smooth continuous signal, a masking objective is only as useful as the gap between what a smoothing filter achieves and what is achievable. Compute that baseline. On our toy window it was 0.43 for scattered masks and 3.77 for spans, and that ratio is the whole justification for span masking.

Four. The InfoNCE gradient is s(∑_b p_bt_b − t_a). Everything about contrastive training — why batch size matters, why hard negatives dominate, why one mislabelled pair is so destructive, why the temperature is not a detail — is visible in that one line.

Five. Split by subject. If you take nothing else operational from this lesson, take that. Ninety-six percent of your test windows are contaminated under a random split, a parameterless nearest-neighbour classifier will score in the nineties on that contamination, and no amount of modelling care downstream will detect it.

References

1. Xu, H., Zhou, P., Tan, R., Li, M., Shen, G. "LIMU-BERT: Unleashing the Potential of Unlabeled Data for IMU Sensing Applications." Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems (SenSys '21), pp. 220–233, 2021. ACM DL · code
2. Moon, S., Madotto, A., Lin, Z., Dirafzoon, A., Saraf, A., Bearman, A., Damavandi, B. "IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text Narrations." arXiv preprint, 2022. arXiv:2210.14395 · code
3. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I. "ImageBind: One Embedding Space To Bind Them All." CVPR, 2023. arXiv:2305.05665
4. Narayanswamy, G., Liu, X., Ayush, K., Yang, Y., Xu, X., Liao, S., et al. "Scaling Wearable Foundation Models." arXiv preprint, 2024. arXiv:2410.13638
5. Abbaspourazad, S., Elachqar, O., Miller, A.C., Emrani, S., Nallasamy, U., Shapiro, I. "Large-scale Training of Foundation Models for Wearable Biosignals." ICLR, 2024. arXiv:2312.05409
6. Erturk, E., Kamran, F., Abbaspourazad, S., Jewell, S., Sharma, H., Li, Y., Williamson, S., Foti, N.J., Futoma, J. "Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions." ICML, 2025. arXiv:2507.00191
7. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al. "Learning Transferable Visual Models From Natural Language Supervision." ICML, 2021. arXiv:2103.00020
8. Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., et al. "Ego4D: Around the World in 3,000 Hours of Egocentric Video." CVPR, 2022. arXiv:2110.07058
9. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL, 2019. arXiv:1810.04805
10. Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L. "A Public Domain Dataset for Human Activity Recognition Using Smartphones." ESANN, 2013. UCI repository