Time-Series Foundation Models

Chapter 1: Why Series Resisted

Chapter 0 asserted that time series lacks a vocabulary. This chapter makes that concrete, because each missing property maps to a specific engineering decision in each of the three papers — and if you understand the four obstacles you can practically re-derive the models.

Obstacle 1: the number 37.2 does not mean anything

Start with what a language model does on its very first layer. The token cat has an integer id, say 4826, and that id indexes a row of an embedding matrix E of shape (V, d) — a lookup, nothing more. The row was learned. Crucially, every time cat appears anywhere in the corpus it retrieves the same row, so the model accumulates evidence about that atom across billions of occurrences.

Now try it on a series. What is the id of 37.2? There isn't one; the reals are uncountable. You could round — and hold that thought, because that is exactly what Chronos does — but round to what precision, and relative to what? 37.2 in a body-temperature series is routine; 37.2 in a series whose values are usually 0.001 is a catastrophic spike.

So the first layer cannot be a lookup. It has to be a projection: a learned linear map from a vector of raw values into the model dimension. Which immediately raises the question of what goes into the vector.

The pointwise row is the design that early time-series Transformers — Informer, Autoformer, FEDformer — used, and it is the one PatchTST attacks. A linear map from R¹ into R¹²⁸ is a rank-one operation: every token embedding is the same vector scaled by the value. Whatever structure attention then discovers, it had to discover from scaled copies of a single direction. PatchTST's ablation is stark: strip patching out, keep everything else, and several configurations do not merely get worse — they run "out of GPU memory (NVIDIA A40 48GB) even with batch size 1."

Obstacle 2: nine orders of magnitude in one batch

Suppose your pretraining corpus contains hourly electricity load in kilowatt-hours (values around 3,000), daily website conversions (values around 12), and a sensor's drift in volts (values around 0.0004). All three enter the same batch, hit the same first linear layer, and their gradients are summed into the same weights.

Written out, the first layer computes W_px_patch. If x_patch has entries near 3,000 the pre-activation is near 3,000·‖W‖; if near 0.0004 it is seven orders of magnitude smaller. One of the two produces gradients that dominate; the other produces gradients that are numerically invisible. No amount of training fixes this, because the problem is in the inputs, not the parameters.

The fix everyone uses is instance normalisation: normalise each series (or each context window) by its own statistics before it enters the model, then invert the transform on the output. There are three flavours in these papers and the differences matter.

Chronos's justification for mean scaling is one line and it is a genuinely good piece of domain thinking: "An attractive feature of mean scaling is that it preserves zero values in the time series, which are often semantically meaningful, such as zero sales for a product or zero solar energy generation at night." Subtracting a mean turns "we sold nothing" into "we sold 1.4 below average", which is a different fact.

Work one instance by hand. Take a context of eight weekly demand values:

Mean scaling sets m = 0 and s = the mean absolute value: (10 + 12 + 8 + 14 + 11 + 15 + 9 + 13) / 8 = 92 / 8 = 11.5. Dividing through:

Now multiply every original value by 1,000 — a different unit, a different product, a different domain. The mean absolute value becomes 11,500, and x̃ comes out identical to four decimal places. That is the whole point: the model never sees the unit. It sees shape. Keep this exact vector; Chapter 4 turns it into tokens.

Drag the sparsity slider up and watch series 3 fill with zeros. Under mean scaling the zeros stay pinned to the zero line no matter how sparse the series gets. Under standardisation they lift off it, and they lift further the sparser the series becomes — the mean of a mostly-zero series is small, but the standard deviation is smaller still, so the zeros drift to a large negative z-score. Chapter 4 shows the exact failure this causes once you then quantize.

Obstacle 3: "next step" is not one thing

A language model's notion of "next" is fixed by the tokenizer. A forecaster's is fixed by whoever installed the sensor. Within one pretraining corpus you have 5-minute traffic counts, hourly electricity, daily page views, weekly sales, monthly tourism, and yearly macroeconomic indicators — and the seasonal period, the thing a forecaster most needs to detect, is a different integer in each: 288, 24, 7, 52, 12, and none.

TimesFM's context-length choices show how sharp the constraint is in practice. It trains "with a maximum context length of 512 whenever the length of the time-series allows that. For weekly granularity we do not have sufficiently long time-series; therefore a maximum context length of 256 is used. For the same reason, a maximum context length of 64 is used while training on ≥ monthly granularity data." A 64-step context on monthly data is five years and four months — barely five annual cycles. There is no way around it: the data simply does not exist.

Obstacle 4: how many channels is a time series?

A multivariate series has M simultaneous measurements per timestep: the Weather dataset has 21, ETTh1 has 7, Traffic has 862. The obvious instinct is to let the model see all channels at once so it can exploit cross-channel structure — if road sensor 41 spikes, sensor 42 spikes eight minutes later.

The obvious instinct is mostly wrong, and PatchTST is where that became consensus. Its second contribution is channel independence: split the M-variate series into M univariate series, run each through the same shared Transformer independently, and never let them attend to each other.

Why does throwing away cross-channel information help? Three reasons that stack.

One — the parameter count of channel mixing scales with M, and M is dataset-specific. A channel-mixing first layer maps R^M into R^D, so its shape depends on the dataset. A channel-independent one maps R^P into R^D, and P is your hyperparameter, identical everywhere. This alone is why every foundation model in the table adopted it: you cannot pretrain across datasets with different M unless the weights are M-agnostic.

Two — it multiplies the effective dataset size by M. ETTh1 with 7 channels is one 7-channel training sequence per window under mixing, but seven univariate sequences per window under independence. PatchTST's Figure 7 shows the consequence directly: channel mixing "quickly overfits the data", while channel independence "contributes to a quicker convergence as more training data is available."

Three — the cross-channel signal is often weaker than the within-channel signal. For long-horizon forecasting, the dominant structure is each channel's own seasonality and trend. Spending capacity on cross-channel attention buys a small gain at a large overfitting risk.

The four obstacles, and what each paper did

Look at the bottom row. Two of the three foundation models simply declined the multivariate problem. Chronos's discussion says so plainly: "we have focused on univariate forecasting of uniformly-spaced time series since it constitutes the most common of real-world time series use-cases", and lists covariates and multivariate forecasting as open. TimesFM's limitations section says "the model is not pretrained with covariates as one of the key challenges is finding large volumes of pretrained data with meaningful covariates." When you deploy one of these and it ignores your promotion calendar, that is not a bug. It is the scope.

Chapter 2: PatchTST — A Patch Is a Token

The title of the PatchTST paper is "A Time Series is Worth 64 Words." That is not a flourish; it is the arithmetic result of the paper's central choice, and by the end of this chapter you will be able to produce the number 64 yourself from the look-back window and two hyperparameters.

The setup. You are given a multivariate series with look-back window L — L past timesteps — and asked to forecast T future steps. Each timestep x_t is a vector of dimension M. Formally: given (x₁, …, x_L), predict (x_L+1, …, x_L+T).

Step 1: split the channels

Before anything else, the M-variate input is split into M univariate series, each of shape (1, L). Each goes through the backbone independently, and the backbone's weights are shared across all of them. In tensor terms this is a reshape that folds the channel axis into the batch axis:

That single line is the whole of channel independence as an implementation. A batch of 32 samples from ETTh1, which has M = 7, becomes 224 univariate sequences. The Transformer never learns that there were seven of anything.

Step 2: normalise, then patch

Each univariate window is instance-normalised — zero mean, unit standard deviation, with the statistics stashed so they can be added back to the output — and then cut into patches. Two hyperparameters govern the cut:

• P, the patch length: how many consecutive values go into one token.
• S, the stride: how far the window advances between consecutive patches. The paper defines it as "the non overlapping region between two consecutive patches."

If S < P the patches overlap; if S = P they tile exactly; if S > P you skip data. PatchTST's default is P = 16, S = 8 — fifty percent overlap.

The number of patches is given in the paper as

and the "+ 2" deserves a full explanation, because it looks like a fudge and is not. Count carefully. Patches start at index 1, 1 + S, 1 + 2S, … The last patch that fits entirely inside a window of length L starts at 1 + kS where kS + P ≤ L, so k = ⌊(L − P)/S⌋, giving k + 1 patches. The extra one is the padded tail: PatchTST appends S copies of the final value x_L before patching, so one more patch fits, and it is anchored on the most recent observation. That is the "+ 2" — (k + 1) real patches plus 1 padded patch.

The worked patch arithmetic

Do the two configurations the paper actually ships. PatchTST/42 uses the default long-horizon look-back L = 336 — fourteen days at hourly resolution:

PatchTST/42 — the default configuration, by handL = 336, P = 16, S = 8

L - P            = 336 - 16 = 320
(L - P) / S      = 320 / 8  = 40        # exact; no flooring needed here
floor(...)       = 40
N                = 40 + 2  = 42         # hence the name PatchTST/42

# and the larger variant:
L = 512, P = 16, S = 8
(512 - 16) / 8   = 496 / 8 = 62
N                = 62 + 2  = 64         # "A Time Series is Worth 64 Words"

There is the title. Sixty-four words is L = 512 with P = 16 and S = 8. Now the payoff, which is why anyone cares:

what patching buys — attention cost# self-attention builds an N x N matrix. Cost is quadratic in the TOKEN count.

pointwise, L = 512 :  512 * 512  = 262,144 attention cells
patched,   N =  64 :   64 *  64  =   4,096 attention cells
                                    ---------------------
                                    64x fewer

# the reduction factor is (L/N)^2 ~ S^2 for large L, since N ~ L/S.
# S = 8  ->  64x.    S = 16 -> 256x.    S = 32 -> 1024x.

The paper measures the wall-clock consequence rather than only the FLOP count: "By setting patch length P = 16 and stride S = 8 with L = 336, the training time is significantly reduced as much as 22 time on large datasets." Twenty-two times faster, measured, on the Traffic dataset with its 862 channels.

Every tensor shape, in order

This is the part that separates reading the paper from being able to write it. Track one univariate series through the whole forward pass with PatchTST/42 numbers and D = 128:

Two details in that table are worth stopping on.

The projection is 2,048 parameters. That is the entire mechanism by which raw numbers become tokens: one 128×16 matrix. Compare a language model's embedding table at 50,000 × 4,096 = 205 million. The tokenizer here is five orders of magnitude smaller than the thing it replaces, because it does not have to memorise atoms — it has to compute them.

BatchNorm, not LayerNorm. The paper's footnote cites Zerveas et al.: "BatchNorm outperforms LayerNorm in time series Transformer." This is a genuine departure from NLP practice and it is a consequence of the domain. LayerNorm normalises across the feature dimension of a single token — but a patch's features are 16 consecutive time values, and normalising across them would destroy the level and slope information that makes the patch meaningful in the first place. BatchNorm normalises each feature across the batch, leaving the within-patch shape intact.

Why a longer look-back only helps once you patch

Here is the experiment that motivates the whole design, and it is small and sharp. Everyone knew that longer look-back windows should help — more history, more information. Everyone also observed that pointwise Transformers got worse as the look-back grew, because the token count grew with it and the model overfitted.

PatchTST ran the control that separates the two effects. Take L = 380, but subsample it — keep every 4th point, plus the last one — so the token count is 96, the same as a pointwise model with L = 96. Same number of tokens, four times the span. Result: MSE 0.447 with the long-span subsampled input against 0.518 with the recent-96 input.

With that established, the look-back curve inverts. PatchTST sweeps L ∈ {24, 48, 96, 192, 336, 720} and reports that "our model gains performance improvement with increasing look-back window", while the Transformer baselines flatten or degrade. The default of 336 and the large variant at 512 are chosen on that curve.

The results, and the ablation that isolates the cause

On the eight standard long-horizon benchmarks — Weather, Traffic, Electricity, ILI and the four ETT datasets — against FEDformer, Autoformer, Informer, Pyraformer, LogTrans and DLinear:

A twenty percent MSE reduction is a large number in this literature. But an aggregate improvement never tells you which idea earned it, so the paper runs the 2×2 ablation: patching on or off, channel independence on or off.

The dashes in that table are the most eloquent entry. Without patching, a Transformer over Traffic's 862 channels and a 336-step window does not merely underperform. It does not run.

And one more control that is easy to miss: the paper trains PatchTST with and without instance normalisation and finds it "improves the forecasting performance slightly", concluding that "the improvement mainly comes from patching and channel-independence." That is a rare and honest sentence — the authors explicitly deny credit to one of their own components.

Chapter 3: Representations vs Heads

Everything in Chapter 2 was supervised: patches in, horizon out, MSE against the truth. That is a forecaster, not a foundation model. The difference is one sentence long — a foundation model learns a representation before it learns a task — and the second half of PatchTST is where that transition happens for time series.

This chapter is the pivot of the lesson. It is also where the tensor shapes get interesting, because the choice of head turns out to constrain the choice of objective far more tightly than anyone expected.

The forecasting head is bigger than the model

Go back to the shape table. Three Transformer layers at D = 128 come to roughly 600,000 parameters. The flatten-and-project head, for a single horizon T:

head parameter count — flatten (N x D), then Linear to TN = 42, D = 128   ->   flattened width = 42 * 128 = 5,376

T =  96 :  5,376 *  96 =   516,096 params   (0.52 M)
T = 192 :  5,376 * 192 = 1,032,192 params   (1.03 M)
T = 336 :  5,376 * 336 = 1,806,336 params   (1.81 M)
T = 720 :  5,376 * 720 = 3,870,720 params   (3.87 M)

At T = 720 the head is roughly six times the size of the backbone. Worse, it is horizon-specific: the supervised PatchTST trains a separate model for every prediction length, because the output width is baked into a weight matrix.

Now recall the alternative the paper argues against — the pointwise masked-autoencoder design of Zerveas et al., where every timestep is a token. Its head has to map L·D features to M·T outputs:

the head PatchTST is arguing againstW has shape (L * D) x (M * T)
L = 512, D = 128, M = 7, T = 96
  -> (512 * 128) x (7 * 96)
  =  65,536 x 672
  =  44,040,192 params  (44.0 M)   # 85x the patched head, for the same task

The paper's phrasing is dry — "This matrix can be particularly oversized if either one or all of these four values are large. This may cause overfitting when the number of downstream training samples is scarce" — but the factor is 85×. Patching shrinks the head as a side effect of shrinking the sequence, and that is what makes a reusable backbone practical.

The masked-patch objective

Swap the objective. Remove the forecasting head. Attach instead a linear layer of shape D × P — one that maps each token's representation back to the P raw values it came from. Then:

1. Cut the series into non-overlapping patches. Overlap would leak: a masked patch's values would still be visible inside its neighbours.
2. Choose a random subset of patch indices — PatchTST masks 40% of them — and set those patches to zero.
3. Run the encoder. Reconstruct all patches through the D×P head. Compute MSE on the masked positions only.

Their representation-learning configuration is L = 512 with patch size 12, non-overlapping. That gives 512 / 12 = 42.67, so 42 complete patches (the paper reports 42; the trailing 8 values do not form a full patch). Head size: 128 × 12 = 1,536 parameters — four orders of magnitude smaller than the 44M pointwise head, and about a thousandth of the T = 336 forecasting head.

Why masking patches and not points

This is the sharpest argument in the paper and it generalises far beyond forecasting. Mask a single timestep in a smooth series and ask a model to recover it. Its neighbours are right there. Linear interpolation between x_t−1 and x_t+1 recovers x_t to within the local curvature — no understanding required.

Make it numeric. Let the series be a smooth sinusoid sampled at 24 points per period, so consecutive samples differ in phase by 2π/24 = 0.2618 radians. Mask x_t. The midpoint of its neighbours is

The interpolation is off by 3.4% of the amplitude, and that is the entire difficulty of the task. The paper says it exactly: masked values "can be easily inferred by interpolating with the immediate proceeding or succeeding time values without high level understanding of the entire sequence, which deviates from our goal of learning important abstract representation."

Now mask a whole patch of 12 consecutive points. Interpolation across a 12-step gap in a period-24 sinusoid spans half a cycle — a straight line between the endpoints misses the entire hump, and the error is on the order of the amplitude itself, not 3.4% of it. To fill it you must recognise the periodicity from the unmasked patches and extrapolate the phase. That is a representation.

Does the representation actually transfer? The three-way comparison

A pretraining objective is only worth anything if the frozen representation is useful. PatchTST measures this three ways, and the ordering is the point:

Linear probing matching from-scratch supervision is the load-bearing result. It says the frozen features already contain what the forecasting task needs; the head is only reading them out. That is precisely the property that makes a backbone worth shipping as a checkpoint.

The two-stage schedule — probe first, then unfreeze — is not incidental either. The paper cites Kumar et al.: "a two-step strategy with linear probing followed by fine-tuning can outperform only doing fine-tuning directly." The reason is that a randomly initialised head produces large, meaningless gradients on the first steps, and those gradients flow straight into the pretrained backbone and damage it. Ten epochs of probing lets the head become sane before it is allowed to speak to the backbone.

Transfer across datasets — the first real foundation-model signal

The experiment that points directly at TimesFM and Chronos: pretrain on Electricity, fine-tune on the others. The result is honest and modest — "overall the fine-tuning MSE is lightly worse than pre-training and fine-tuning on the same dataset, which is reasonable", and sometimes worse than supervised training — but "the forecasting performance is still better than other models", and at a fraction of the compute, since only the head or a few epochs are retrained.

And the structural observation buried in that section is the one that made everything after it possible:

Reconstruct or predict? The objective fork

Two objectives are now on the table, and the three foundation models split between them. This fork determines everything downstream, so lay it out plainly.

The second row decides most arguments. In a causal model every one of the N tokens produces a training signal on every sequence; in a masked model only the masked fraction does. For the same number of tokens processed, causal training extracts roughly 2.5× more supervision at a 40% mask rate. That efficiency is exactly why language models converged on causal decoding, and it is why both models that set out to be forecasters chose it.

But look at row three. If you want the checkpoint to be useful for classification, imputation and anomaly detection — which is MOMENT's entire thesis — bidirectional reconstruction is the better objective, because those tasks are all instances of "fill in what is missing given everything else". Chapter 7 returns to this with numbers.

Supervision density, counted

"Roughly 2.5× more supervision" was a hand-wave. Count it properly, because the real number is much larger than that and it explains a design choice in Chapter 5 that otherwise looks arbitrary.

Take a 512-point training sequence and ask, for each objective, how many numbers the loss is computed on.

supervised values per 512-point sequence# A. masked reconstruction, PatchTST SSL config: P = 12, disjoint, 40% maskedpatches            = 512 / 12          = 42
masked patches     = 0.40 * 42          = 17
supervised values  = 17 * 12            = 204      # 0.40x the observations

# B. causal next-patch, output patch equal to input patch: p = h = 32tokens             = 512 / 32          = 16
# token j predicts steps 32j+1 .. 32j+32, and the last one falls off the endsupervised values  = 15 * 32            = 480      # 0.94x the observations

# C. causal next-patch, TimesFM's actual config: p = 32, h = 128tokens             = 16
# token j predicts steps 32j+1 .. 32j+128 — targets OVERLAPtokens 1..12 : 128 valid targets each  = 1,536
token 13     :  96   token 14 : 64
token 15     :  32   token 16 :  0     =   192
supervised values                       = 1,728    # 3.38x the observations

# ratio C : A  =  1,728 / 204  =  8.5x more supervision from the same data

And notice what this does to the data problem. Time-series corpora are small compared with text: Chronos trains on a corpus assembled from 28 public datasets, and MOMENT had to build the Time Series Pile from four archives because nothing suitable existed. When data is the binding constraint, an objective that extracts 3.4 targets per observation instead of 0.4 is worth more than an extra billion parameters.

The counter-argument, for balance: those four lessons about step 300 are highly correlated, so eight and a half times the targets is not eight and a half times the information. It is closer to a curriculum — the same target, at four difficulties — and curricula have diminishing returns. But the gradient signal is real, and it is free.

Chapter 4: Chronos — A Value Is a Word

Every paper so far has answered "what is a token?" by building something — a projection, a residual block, an architecture. Chronos answers it by refusing to build anything at all.

The proposal, in one sentence: round the numbers to a fixed grid, call each grid cell a word, and hand the resulting sentence to an unmodified language model. The paper's own summary is almost defiantly plain: Chronos "tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss."

Not a new architecture. Not even a new loss. T5, exactly as published, with one change: the embedding table is resized. The paper is explicit that "No modifications are required to the language model architecture, except adjusting the vocabulary size to |V_ts|… Concretely, adjusting the vocabulary size entails truncating (or extending) the input and output embedding layers of the language model."

Step 1: scaling (already done in Chapter 1)

Recall the mean-scaling transform. Given a context x_1:C, set m = 0 and

and keep our eight weekly demand values from Chapter 1:

These are still real numbers. A language model cannot read them. That is what step two is for.

Step 2: quantization, derived

Pick B bin centers c₁ < c₂ < … < c_B on the real line, with B − 1 edges b_i sitting between consecutive centers. The quantization function q maps a real to an integer, and the dequantization function d maps it back:

Two decisions remain: where to put the centers, and how many.

Where. You could space them by quantiles of the training data — more resolution where values are dense. Chronos deliberately does not: "Since the distribution of values for unseen downstream datasets can differ significantly from the training distribution, we opt for uniform binning." This is a foundation-model decision, not a compression decision. Quantile bins are optimal for the corpus you have and mis-shaped for the corpus you have not seen. Uniform bins are never optimal and never catastrophic.

So the centers are uniformly spaced on [c₁, c_B] = [−15, +15], with edges exactly midway: b_i = (c_i + c_i+1) / 2.

How many. The vocabulary size is 4096, "including the special tokens (PAD and EOS)". Subtract those two and you get B = 4094 numeric bins — the number the paper uses in its own error analysis.

Now the spacing falls out. With B centers spanning a range of 30, there are B − 1 gaps:

and the i-th center (1-indexed) is c_i = −15 + (i − 1)Δ.

The full hand-worked tokenisation

To turn a scaled value v into an index, invert that formula and round to the nearest center:

Do the first one completely, digit by digit. v = 0.8696:

quantizing one value, every step shownv + 15            = 0.8696 + 15 = 15.8696
(v + 15) / delta  = 15.8696 / 0.00732959 = 2164.87
round(...)        = 2165
index i           = 2165 + 1 = 2166

# check by going back: the center of bin 2166 is
c_2166            = -15 + 2165 * 0.00732959 = -15 + 15.868563 = 0.868563
# and un-scaling by s = 11.5:
c_2166 * s        = 0.868563 * 11.5 = 9.9884      # true value was 10
error             = -0.0116                        # 0.12% of the value

The whole context, tokenised:

So the sentence a T5 encoder sees is literally

and the worst error in the table is 0.0385 out of a value of 15 — 0.26%. The theoretical bound is half a bin width: Δ/2 = 0.003665 in scaled units, which is 0.003665 × 11.5 = 0.0421 in original units, and every row respects it.

Step 3: the model, which is not a step at all

Token ids in, T5 out. The only architectural surgery is the embedding table, and it is worth seeing what that does to the parameter counts, because it explains a footnote that otherwise reads as a typo.

The context length is 512 — "the default for T5 models" — and the prediction length is 64, "a value greater than the prediction lengths of all tasks we consider in our evaluation." Note that these are timepoints, not patches: Chronos spends one token per observation, which is the price of using a language model verbatim. Chapter 1's high-frequency squeeze is exactly this constraint.

The loss: regression by classification

The output distribution is a categorical over the 4096 vocabulary entries, and the loss is ordinary cross-entropy. The paper names the paradigm: "Chronos uses a categorical distribution to model the observations, performing regression via classification."

Work an instance. Suppose at some position the model's logits over five neighbouring bins are (0.4, 2.1, 3.6, 1.9, 0.2), and the true next value quantizes to the middle bin.

cross-entropy at one position, by handlogits        = [ 0.4 , 2.1 , 3.6 , 1.9 , 0.2 ]
subtract max  = [-3.2 ,-1.5 , 0.0 ,-1.7 ,-3.4 ]   # numerically safe softmax
exp           = [0.0408, 0.2231, 1.0000, 0.1827, 0.0334]
sum Z         = 1.4799
probabilities = [0.0275, 0.1508, 0.6757, 0.1234, 0.0226]

truth = middle bin  ->  loss = -ln(0.6757) = 0.3920
# and here is the point:
truth = next bin over ->  loss = -ln(0.1234) = 2.0920
truth = four bins away -> loss = -ln(0.0275) = 3.5920

# the guessing baseline for a 4096-way softmax is ln(4096) = 8.3178

Notice the middle line. Being one bin off — an error of 0.0073 in scaled units, utterly negligible — costs 2.09 nats, while being four bins off costs 3.59. The ratio is not 4:1, or 16:1, or anything proportional to the distance. It is whatever the softmax happens to produce, because cross-entropy has no notion of distance between classes. A conventional forecasting loss like MSE punishes errors quadratically by construction; this one does not punish by magnitude at all.

And the paper is candid that this is unfinished business: "An in-depth theoretical and empirical analysis of the regression-via-classification paradigm in the context of time series forecasting would constitute interesting future research."

What you get for free: a predictive distribution

Because the output is a categorical distribution and generation is autoregressive sampling, a probabilistic forecast requires no extra machinery at all. Draw a token, dequantize it, append it to the context, draw the next, and repeat to the horizon. Do that 20 times — the paper's setting — and you have 20 sample paths. Read quantiles off them.

This is a real advantage over TimesFM, whose base model emits point forecasts and would need extra quantile heads (its paper notes this as a straightforward extension). It also costs something: a point forecast from Chronos requires aggregating over samples, which the discussion section lists as a target for future speedups.

The data: two augmentations that carry the whole corpus

Chronos's public pretraining data is smaller than TimesFM's by orders of magnitude — 28 public datasets, not 100 billion timepoints from Google's logs. Two augmentation schemes make up the difference.

TSMixup. Sample k ~ U{1, K} series of a common length from different datasets, scale each, and take a convex combination:

with K = 3 in the main runs. Because k can be 1, "original time series are adequately represented since they are included in the TSMixup augmentations with probability 1/3." The purpose is stated as pattern diversity: a mixture of a traffic series and a retail series is a series with two simultaneous seasonalities, which is a pattern the corpus does not otherwise contain.

KernelSynth. Build a bank of Gaussian-process kernels — linear for trend, RBF for smooth local variation, periodic for seasonality — then sample j ~ U{1, J} of them, combine with random + or × operations, and draw a sample from the resulting GP prior. The paper credits the Automatic Statistician for the idea and notes that it is running it backwards: that system searches over compositional kernels to explain a series, and "We use the inverse of this process — randomly compose GP kernels to generate new time series."

The two ways this tokenizer breaks

Chronos has an honest limitations section, and both failure modes are direct consequences of the arithmetic above. They are worth knowing before you deploy it, because both are silent.

Failure 1 — overflow on sparse series. The representable range in original units is [−15s, +15s]. If s is tiny relative to the peaks, the peaks fall off the end of the vocabulary. The paper's own example: unit spikes every n observations, so s = 1/n.

the intermittent-demand failure, by handspikes of height 1.0, one every n observations
s = mean|x| = 1/n
maximum representable original value = 15 * s = 15/n

n = 10 :  15/10 = 1.50   ->  spike of 1.0 fits   (OK)
n = 20 :  15/20 = 0.75   ->  spike of 1.0 CLIPS  (top bin)
n = 50 :  15/50 = 0.30   ->  spike of 1.0 CLIPS  badly

# the paper: "When 1 > 15/n then the model cannot possibly capture
# the spikes appropriately... since their value is not represented
# accurately by tokens."

Intermittent demand — the spare part that sells four units a year — is one of the most common real forecasting problems there is, and it is precisely where mean scaling collapses. This is the single most practically important sentence in the Chronos paper.

Failure 2 — precision loss on offset series. The opposite regime. If s is large relative to the variance, the interesting variation is smaller than one bin. Token spacing in original units is 30s / (B − 1):

A signal riding on a large DC offset — a temperature in Kelvin, a stock index, a cumulative counter — loses almost all its structure. The fix the paper offers is an inference-time heuristic, not a model change: "preprocess the time series using an alternative normalization scheme, such as standardization, for time series with large scale and small variance." In other words, centre it yourself before you hand it over.

One ablation that is more interesting than it looks

The paper sweeps the vocabulary size on Chronos-T5 (Small) and reports something that is not a clean win: "modest improvements in the point forecasting metric (MASE) as the vocabulary size increases. In contrast, the WQL initially improves but deteriorates for larger vocabulary sizes."

Their explanation is a metric argument and it is worth internalising, because it is a trap you can fall into on any project. MASE is scale-invariant, and so is the training loss (which operates entirely on scaled values), so the two move together: more bins means finer resolution means better MASE. WQL is scale-dependent, so it does not track the training loss and "behaves less predictably as precision increases." The paper's conclusion is deliberately modest: selecting a vocabulary size "would pose a trade-off", between resolution and the number of examples the model sees for each token.

Chapter 5: TimesFM — Decoder-Only Rollout

Chronos borrowed a language model whole. TimesFM borrowed only the shape of one — a causal decoder that predicts the next thing from all previous things — and then broke the analogy in exactly one place, which turns out to be the most interesting design decision in the paper.

The claim is a modest-sounding one for a 200M-parameter model: zero-shot performance that "comes close to the accuracy of fully-supervised forecasting models on a diverse set of time-series data." And the authors are careful to put it in context against the alternative that was popular at the time: "unlike recent work that recommends Large Language Models such as GPT-3 and LLama-2 as out-of-the-box zero-shot forecasters, foundation models trained from scratch exclusively on time-series data can obtain much better zero-shot performance at a tiny fraction of its costs."

The three decisions, in order of increasing interest

Decision one: patch. Inherited from PatchTST, and justified in the same terms — "A patch of a time-series is a natural analogue for a token in language models" — with the added inference argument that "the number of tokens being fed into the transformer is reduced by a factor of the patch length."

Decision two: go causal. "A key difference between our architecture and PatchTST is that our model is trained in decoder-only mode. In other words, given a sequence of input patches, the model is optimized to predict the next patch as a function of all past patches. Similar to LLMs this can be done in parallel over the entire context window, and automatically enables the model to predict the future after having seen varying number of input patches."

That last clause is the whole reason. A bidirectional encoder with a forecasting head is trained for one context length and one horizon. A causal decoder over N patches produces N supervised predictions per sequence, each conditioned on a different amount of history — so it is simultaneously being trained to forecast from 1 patch, 2 patches, …, N−1 patches. Variable context length is not a feature you add; it is a side effect of the objective.

Decision three: make the output patch longer than the input patch. This is the novel one, and it deserves its own section.

Why the output patch is 128 when the input patch is 32

Start with the tension the paper identifies. In long-horizon forecasting "it has been observed that directly predicting the full horizon yields better accuracy than multi-step auto-regressive decoding" — a direct multi-step head beats rolling a one-step model forward, because rolled-forward errors compound. But you cannot bake the full horizon into the output width when you do not know the horizon in advance, "as in the case of zero-shot forecasting which is our primary goal."

TimesFM's answer: "We propose a middle ground by allowing our output patches for prediction to be longer than the input patches." Input patch p = 32, output patch h = 128.

The paper's own training illustration, spelled out: "suppose the input patch length is 32 and output patch length is 128. During training, the model is simultaneously trained to use the first 32 time-points to forecast the next 128 time-steps, the first 64 time-points to forecast time-steps 65 to 192, the first 96 time-points to forecast time-steps 97 to 224 and so on."

And the inference payoff, also the paper's example: "During inference, suppose the model is given a new time-series of length 256 and tasked with forecasting the next 256 time-steps into the future. The model will first generate the future predictions for time-steps 257 to 384, then condition on the initial 256 length input plus the generated output to generate time-steps 385 to 512."

rollout step count — the arithmetic that makes h mattersteps = ceil(H / h)          # H = requested horizon, h = output patch

H = 256, h = 128  ->  ceil(2.00) = 2 steps      # TimesFM as shipped
H = 256, h =  32  ->  ceil(8.00) = 8 steps      # if h were tied to p
H = 512, h = 128  ->  ceil(4.00) = 4 steps
H =  96, h = 128  ->  ceil(0.75) = 1 step       # most benchmarks: ONE forward pass

# each extra step feeds the model its own output, so errors compound.
# 2 steps instead of 8 is a 4x reduction in compounding opportunities,
# and a 4x reduction in sequential latency.

Look at the last line of that block. For the two most-reported long-horizon settings, T = 96 and T = 192, an output patch of 128 means the entire forecast comes out of one forward pass for T = 96 and two for T = 192. The model is behaving like a direct multi-step forecaster on exactly the benchmarks where direct multi-step is known to win, while remaining able to roll out arbitrarily far when asked.

The architecture, with every shape

Track a series of length L = 512 through the model, with the 200M configuration: p = 32, h = 128, model_dim = 1280, 20 layers, 16 heads, FFN hidden size equal to model_dim.

Two structural notes. First, attention cost: 16 tokens is 256 attention cells, against 512² = 262,144 for a pointwise model at the same context — a factor of 1024×. Second, the mask is an input, not a preprocessing step. It goes into the residual block alongside the values, so the model learns to interpret "this part of the patch is padding" rather than being handed silently corrupted numbers.

Where 200M comes from

Derive the parameter count from the four hyperparameters. It lands almost exactly on the headline number, which is a good sanity check that you have understood the architecture.

TimesFM 200M parameter count, derivedd = model_dim = 1280,  layers = 20,  FFN hidden = d (stated in the paper)

per transformer layer:
  attention  W_Q, W_K, W_V, W_O   = 4 * d * d = 4 * 1,638,400 =  6,553,600
  FFN        up + down             = 2 * d * d = 2 * 1,638,400 =  3,276,800
                                                                ----------
                                                                 9,830,400
x 20 layers                                                  = 196,608,000

input residual block  (patch 32 + mask 32 = 64) -> d -> d       ~ 1.80 M
output residual block  d -> d -> 128                            ~ 1.80 M
                                                                ----------
total                                                        ~ 200.2 M   # paper: "200M parameters"

Ninety-eight percent of the model is the transformer stack. The tokenizer and the head together are under 2%. Compare Chapter 3, where PatchTST's forecasting head was six times the backbone — the decoder-only design, with its fixed-width output patch, is what moves the parameters back into the part that generalises.

The masking trick that makes every context length reachable

Here is a subtle bug you would ship if you were not careful, and the two-line fix.

Patches are non-overlapping and aligned to the start of the sequence. So the amounts of history the model is ever asked to condition on are 32, 64, 96, 128, … — multiples of p. The paper states the risk directly: "If we use patches naively, the model might only learn to predict well for context lengths that are multiples of the input patch length." Hand it a context of 100 points at inference and it is in unfamiliar territory.

The fix: "For each time-series in the batch, we sample a random number r between 0 and p − 1. Then we set m_1:r = 1 and the rest as zero, i.e. we mask out a fraction of the first input patch."

Follow the paper's worked example. Max context 512, p = 32, and suppose r = 4:

why masking r points of the FIRST patch covers every context lengthr = 4  ->  the first 4 values are masked out.

token o_1 sees 32 - 4               = 28 real points  ->  supervised at context 28
token o_2 sees 28 + 32              = 60 real points  ->  supervised at context 60
token o_3 sees 60 + 32              = 92 real points  ->  supervised at context 92
...                                                        (always 28 mod 32)

# now vary r over 0..31 across the batch and across training:
r = 0  covers contexts 32, 64,  96, ...   (0 mod 32)
r = 1  covers contexts 31, 63,  95, ...   (31 mod 32)
r = 2  covers contexts 30, 62,  94, ...   (30 mod 32)
...
r = 31 covers contexts  1, 33,  65, ...   (1 mod 32)

# union over r = 0..31: EVERY integer from 1 to 512. The paper:
# "the model has seen all possible context lengths till 512."

One random integer per sequence, and the model becomes context-length agnostic. That is the sort of two-line trick that separates a research prototype from something you can hand to a stranger with an arbitrary series.

The corpus, and what it is made of

TimesFM's pretraining data is the part of the paper that is hardest for anyone else to reproduce, and the authors are open about the sources.

Total: "O(100B) timepoints", against 200M parameters. The ablation confirms the synthetic share is load-bearing — the paper compares the 200M model with and without synthetic data on both Monash and ETT.

Two ablations worth memorising

Input patch length. Sweeping p from 8 to 128 on a 70M model, "p = 16, 32 marks the best performance, with the error increasing towards either end." Too small and you are back to nearly-pointwise tokens with a huge sequence; too large and, as the paper says, "that makes the model shift from decoder only training more towards encoder-decoder style training" — in the limit p equals the context and there is only one token, so there is nothing causal left. They chose 32 over 16 because it is "almost twice as fast to train" for equal accuracy.

Scale. Three sizes — 17M, 70M, 200M — trained to 1.5M iterations at a global batch size of 4096 on TPUv5e, with checkpoints plotted against FLOPs. "It can be clearly seen that the errors decrease monotonically with the number of FLOPS (in log scale)." A clean log-linear scaling curve, on 100 billion timepoints, with the largest model costing "16 core TPUv5e for 2 days". That is a strikingly small budget for a foundation model, and it is the strongest evidence in the paper that the recipe is sound: the gains are coming from the design, not from brute force.

Chapter 6: Zero-Shot, Honestly

Every paper in this lesson reports that its model does well. All of them are telling the truth. This chapter is about reading those claims precisely enough to know what you are buying — because the honest summary is not "foundation models beat statistical baselines", and it is also not "the baselines still win". It is more specific and more useful than either.

First, the metrics — computed by hand

You cannot audit a claim in a unit you cannot compute. There are two units in this literature.

MASE — Mean Absolute Scaled Error. Take the mean absolute error of your forecast, and divide it by the mean absolute error a naive one-step model would have made in sample. The division is what makes it comparable across series with different scales.

Work one completely. Quarterly data, so the seasonal period m = 4. In-sample history and the four actuals we must forecast:

MASE, all of it, by handhistory y = [10, 20, 15, 5, 12, 22, 17, 7]     # 8 points, m = 4
actual  a = [11, 21, 16, 6]                    # what really happened

# denominator: mean absolute FIRST DIFFERENCE of the history
diffs = |20-10|,|15-20|,|5-15|,|12-5|,|22-12|,|17-22|,|7-17|
      =   10  ,   5   ,  10  ,   7  ,  10   ,   5   ,  10
sum   = 57 ,  count = 7
denom = 57 / 7 = 8.142857

# now score four forecasters:
seasonal naive  [12,22,17,7]     errs 1,1,1,1        MAE 1.000  MASE 0.1228
naive (last=7)  [ 7, 7, 7,7]     errs 4,14,9,1       MAE 7.000  MASE 0.8596
mean (13.5)     [13.5 x4    ]    errs 2.5,7.5,2.5,7.5 MAE 5.000 MASE 0.6140
over-smoothed   [11.5,19,15.5,8] errs 0.5,2,0.5,2    MAE 1.250  MASE 0.1535
a good model    [11.4,20.6,16.3,6.2] errs .4,.4,.3,.2 MAE 0.325 MASE 0.0399

Two lessons from that block, and they are the two that people get wrong.

MASE = 1 is not "good". It means you matched a naive one-step predictor's in-sample error. On a strongly seasonal series, seasonal naive already scores 0.1228 — eight times better than the denominator — so a model at MASE 0.9 is not "close to naive", it is seven times worse than copying last quarter.

A smooth, plausible-looking forecast can lose to a crude one. The "over-smoothed" row is what an underconfident model produces: right shape, shrunken amplitude. It scores 0.1535 against seasonal naive's 0.1228. It looks better on a chart. It is worse. Foundation models trained with an MSE-style objective are systematically prone to this, because shrinking toward the mean is the risk-minimising response to uncertainty.

WQL — Weighted Quantile Loss. Used for probabilistic forecasts, computed by Chronos "on 9 uniformly-spaced quantile levels {0.1, 0.2, …, 0.9}", with the quantiles estimated from 20 sample paths for methods that sample. The per-quantile ingredient is the pinball loss, which is asymmetric on purpose:

pinball loss — why the asymmetry is the whole ideatrue value y = 11, and we are scoring the q = 0.9 quantile forecast.

y_hat = 10  (too low)  ->  y >= y_hat  ->  0.9 * (11 - 10) = 0.90
y_hat = 13  (too high) ->  y <  y_hat  ->  0.1 * (13 - 11) = 0.20

# at q = 0.9 the loss punishes being too LOW 9x harder than being too high,
# which is exactly what forces the 0.9 quantile to sit above the data 90%
# of the time. Sum over the 9 levels, divide by sum|y|, and that is WQL.

Second, the aggregation — where a lot of intuition goes to die

Chronos reports "aggregated relative WQL" and "aggregated relative MASE". The recipe: score every model on every task, divide each score by the Seasonal Naive score on that task, and take a geometric mean across tasks.

Three consequences you must hold in your head when reading those figures.

And a rule that quietly matters: "For models that failed or could not finish evaluation within the allotted time on certain datasets, we used a relative score of 1", i.e. the baseline's score. A model that cannot run on a dataset is scored as if it had tied with Seasonal Naive there, rather than being penalised.

The results, stated exactly

Chronos, Benchmark I — 15 datasets that were in its training corpus. "The bigger Chronos-T5 models (Base and Large) significantly outperform baseline models… not only… better than local models (e.g., AutoETS and AutoARIMA), but they also perform better than task-specific deep learning models trained or fine-tuned for each dataset (e.g., PatchTST and DeepAR)."

That is a strong result, and it is an in-domain result — the correct comparison for "should I use one model instead of forty thousand", not for "will this work on my new data". Note also the observation the authors chose to publish: "the Seasonal Naive baseline performs competitively against other local models on this benchmark, suggesting that the datasets in this benchmark exhibit strong seasonal patterns." Half the standard benchmark suite is nearly solved by copying last week.

Chronos, Benchmark II — 27 datasets never seen in training. This is the real test, and the paper opens with the caveat rather than the headline: "This benchmark is clearly more challenging than Benchmark I, as the best models tend to offer lower improvements relative to the baseline."

"Second to fourth, on par with the best task-specific models" is a genuinely impressive result for a model that took no gradient steps on those datasets. It is not "beats everything".

TimesFM, across three benchmark families. The pattern is different and more revealing.

When they win and when they do not

Assemble the pattern from the evidence above and from the tokenizer failures in Chapters 4 and 5. This table is the practical takeaway of the whole lesson.

Play with the last two archetypes. On pure noise the retrieval stand-in loses to the mean forecast, because there is no motif to find and it copies a random one anyway — the same reason a foundation model's confident wiggle on an unforecastable series is worse than a flat line. On level shift everything degrades, and the ranking becomes mostly a lottery, which is the honest answer to "how do these models handle regime change".

Contamination, and the fact that everyone knows

The zero-shot claim rests entirely on the target data being absent from pretraining, and every author in this space knows the guarantee is soft. The disclosures are worth reading in full because they set the standard for what "honest" looks like here.

Chronos, in a footnote about Benchmark II: "From a rigorous standpoint, to prevent information leakage, the start time of any dataset within this category must be after the timestamp of the last observation from the pretraining dataset and Benchmark I. Nevertheless, we consider the risk to be minimal given that the datsets bear no overlap beyond high-level conceptual categorization."

Chronos, about a competitor it is beating: "the evaluation setup may have been advantageous for Moirai-1.0-R as many datasets in Benchmark II were part of its pretraining corpus." A paper flagging that its own comparison flatters itself.

TimesFM, about the Darts benchmark: "since these datasets are used in numerous time series blog posts for illustrative purposes, data contamination for llmtime cannot be ruled out." An LLM prompted with a famous series may simply have read it.

And the most quietly admirable one — Chronos's own changelog: "We found an off-by-one error in the decoded bin indices for Chronos models which had led to artificially worse results for Chronos models in the previous version. Upon fixing this issue, the results for Chronos models improved significantly." A one-index bug in dequantization, disclosed in public, that had been understating their own numbers. Note what it also implies for you: if you implement Chapter 4's quantization yourself and are off by one bin, you will lose accuracy in a way that looks like a modelling problem and is not.

The cost side of the ledger

Chronos measures inference time for a single series and reports that "the inference speed of the larger Chronos models is comparable to some statistical local models", while conceding that "a potential limitation of the larger Chronos models is their inference speed compared to task-specific deep learning models."

Read that comparison carefully, because it is doing real work. A statistical local model is fitted per series — that fit is the expensive part, and it is what the comparison includes. A task-specific deep model is trained once and then evaluated cheaply per series — but that training run is not in the number. So:

The third column is where Chronos pays for its distribution: twenty sampling passes per forecast, each autoregressive over the horizon. The paper's own list of remedies is a list of things the NLP community already built — "quantization… and faster decoding techniques, including speculative and lookahead decoding" — which is the clearest possible illustration of what you buy by making your model architecturally identical to a language model.

Chapter 7: Embeddings From Forecasters

A forecaster consumes a context and emits a future. Somewhere in the middle it holds a fixed-size representation of that context — and if that representation is good enough to forecast from, the obvious question is what else it is good enough for.

This chapter is about pulling the vector out of the middle and using it: to classify a series, to find similar series, to score how anomalous a window is. It is also where the honest limits of "one model for everything" become visible, because the objective a model was trained on determines what its representation throws away, and forecasting throws away exactly the things classification wants.

Where the vector is, and what shape it has

Take MOMENT, the model in this family built explicitly for general-purpose analysis rather than forecasting. Its configuration is stated precisely: "All models take an input time series of length T = 512, breaking it into N = 64 disjoint patches of length P = 8. We mask 30% of the patches uniformly at random during pre-training."

So a 512-point window becomes a 1024-number vector in the Large model. That is a 2:1 compression in raw count — not impressive as compression — but the point was never compression. The point is that the 1024 numbers live in a space where distance means similarity of dynamics, and the original 512 do not.

Retrieval, worked

Once you have vectors, similarity is a dot product. Do one by hand in four dimensions so nothing is hidden. Let a query window and three library windows have pooled embeddings:

cosine retrieval over pooled embeddings, by handq                = [ 0.8,  0.5, -0.2,  0.3]   |q| = sqrt(0.64+0.25+0.04+0.09) = 1.0100

v1 ECG normal    = [ 0.7,  0.6, -0.1,  0.4]   |v1| = 1.0100
   dot = 0.8(0.7) + 0.5(0.6) + (-0.2)(-0.1) + 0.3(0.4)
       = 0.56 + 0.30 + 0.02 + 0.12 = 1.0000
   cos = 1.0000 / (1.0100 * 1.0100) = 0.9804        # nearest

v2 ECG arrhythmia= [ 0.2, -0.5,  0.8,  0.1]   |v2| = 0.9695
   dot = 0.16 - 0.25 - 0.16 + 0.03 = -0.2200
   cos = -0.2200 / (1.0100 * 0.9695) = -0.2247

v3 motor vibration=[-0.6,  0.3,  0.5, -0.2]   |v3| = 0.8602
   dot = -0.48 + 0.15 - 0.10 - 0.06 = -0.4900
   cos = -0.4900 / (1.0100 * 0.8602) = -0.5640      # most dissimilar

Three lines of arithmetic and you have a nearest-neighbour search over an archive of sensor windows. Index a million of them and the same dot product becomes semantic search over machine behaviour: show me every window that looks like the four hours before the last failure. There is no forecasting anywhere in that query, and it is arguably the most valuable thing the checkpoint does.

The three downstream tasks, and how each reduces to the same encoder

Notice how tightly the objective and the task list are coupled. Masked reconstruction gives you imputation for free, anomaly detection almost for free (reconstruction error is already the loss), and classification cheaply (pool and probe). Causal next-patch prediction gives you forecasting for free and the others awkwardly. The objective is not a detail; it is the product roadmap.

The honest result you should know about

MOMENT's imputation section reports that with linear probing it "achieved the lowest reconstruction error on all ETT datasets", and then adds: "In the zero-shot setting, MOMENT consistently outperformed all statistical interpolation methods with the exception of linear interpolation."

Sit with that. A 385-million-parameter foundation model, pretrained on a corpus assembled from four public archives, loses zero-shot imputation to drawing a straight line between the two neighbouring points.

Can you use a forecaster as an embedding model?

Chronos says maybe, and is careful to label it a hypothesis rather than a result: "We hypothesize that the representations learned by the encoders of Chronos-T5 models are universal and can be used for these tasks. An exploration of Chronos-T5 representations for various downstream tasks would constitute interesting future work."

You can reason about how well that will go by asking, for each task, what the forecasting objective was paid to keep.

The two "critically / no" rows are where forecaster embeddings will disappoint on classification, and they are also fixable: centre and phase-align before pooling, or pool over several offset windows and average, which washes out phase the way a bag-of-frames does in audio. That is a real technique, not a hedge — and it is exactly the kind of adaptation that separates using a checkpoint from understanding it.

Chronos also has a structural advantage worth naming: it is a T5, so it has a genuine encoder whose output is a bidirectional representation of the context. A pure decoder-only model like TimesFM has only causal states, so the representation at position j has never seen positions after j — you would pool the final state, or the mean of states, and accept that early tokens were encoded with less information than late ones.

Anomaly detection, with the threshold arithmetic

The reconstruction route deserves to be spelled out, because it is the one people deploy first and the one whose failure mode is least obvious.

The procedure: slide a window of 512 points along the series with some stride; run each window through the encoder and the reconstruction head; score the window by its mean squared reconstruction error. Rare dynamics score high because the model never learned to reproduce them.

Then you need a threshold, and the honest way to set one is to calibrate on a clean split rather than to pick a number.

calibrating an anomaly threshold, by hand# 1. run every window of a KNOWN-CLEAN period through the modelrecon errors on clean windows :  mean mu = 0.021 ,  sd = 0.006

# 2. pick a working point. three sigma is the usual first guess:threshold = mu + 3 * sd = 0.021 + 0.018 = 0.039

# 3. score a suspect windowe = 0.058
z = (0.058 - 0.021) / 0.006 = 6.17      # six sigma — flag it

# 4. sanity-check the alert rate BEFORE shipping.
#    at 3 sigma under a normal model, roughly 1 window in 741 fires.
#    with a 512-point window and a stride of 64 on 1 Hz data, that is
#    one window every 64 seconds -> about 65 false alarms per day, per sensor.
#    across 500 sensors that is 32,500 pages. The threshold is not the
#    hard part; the alert budget is.

How you pool matters more than you would guess

Every task above starts with "reduce (N, D) token representations to one (D,) vector". That reduction is a modelling choice, and different choices keep different things.

The fourth row is the cheap win people miss. Concatenating the standard deviation across tokens costs one line and doubles the dimension, and it captures "how much did the dynamics change within this window" — which is precisely the discriminating feature in a great many sensor problems, and which mean pooling averages away by construction.

python — pulling embeddings out of a patched encoderimport torch

# x: (B, T) univariate windows, T = 512x = revin.normalize(x)                      # (B, 512)
patches = x.unfold(-1, 8, 8)                  # (B, 64, 8)   P = 8, disjoint
tok     = patch_proj(patches) + pos             # (B, 64, D)
h       = encoder(tok)                          # (B, 64, D)   D = 768 for Base

emb_mean = h.mean(dim=1)                      # (B, D)      the usual embedding
emb_rich = torch.cat([h.mean(1), h.std(1)], -1)  # (B, 2D)     dynamics + variability
emb_norm = torch.nn.functional.normalize(emb_mean, dim=-1)

# retrieval over an archive of M windows is now one matmulscores = emb_norm @ archive_norm.T              # (B, M) cosines
top    = scores.topk(10, dim=-1)                # nearest neighbours

Nine lines from a raw window to a semantic search over a sensor archive. The heaviest thing in that block is the encoder call, and it runs once per window, offline, exactly like indexing documents.

Which checkpoint for which job

Chapter 8: The Design Axes

Eight chapters in, you have seen five models. It would be easy to file them as five separate inventions. They are not. They are five points in a small, enumerable design space, and once you can see the axes you can read any new paper in this area in about ten minutes — and, more to the point, you can propose the combinations nobody has shipped.

There are five axes that matter.

The five models, placed

Read the table by columns rather than rows and the field's actual state of knowledge appears. Column five is nearly constant — four of five models simply do not model channels — which tells you where the open problem is. Column three is nearly constant too, which tells you that scaling is considered solved (Chapter 4 says it is not). Columns one and two are where all the variation lives, which is why this lesson spent five chapters there.

Axis 1 in depth: the tokenization ladder

Now look at the fourth and third rows together. The quantized token has the property everyone wants — a genuine discrete vocabulary — and the property nobody wants, a sequence as long as the series. The patched token has the opposite pair. Nothing in these five papers has both. That empty cell is the most obvious open recombination in the space, and it has a name in every neighbouring field: quantize the patch, not the value. Learn a codebook of K prototype shapes, assign each patch to its nearest code, and you get a short sequence of genuine discrete tokens.

Axis 2 in depth: what each objective buys and forecloses

The last column is the one to memorise, because it is your debugging checklist. When a foundation model produces a bad forecast, the first question is not "is it undertrained" but which of these signatures am I looking at. A flat, shrunken forecast is an objective problem. A staircase is a tokenizer problem. A forecast that drifts increasingly wrong after step 128 is a rollout problem. Three different fixes.

A design exercise: build the model these five cannot

Chapter 6 named intermittent demand as the clearest loss for this family. Design a model for it, on purpose, one axis at a time. Do not skip to the answer; the value is in watching each constraint propagate.

The data. Weekly demand for spare parts. A typical series is mostly zeros with occasional counts of 1–4, and the zeros are meaningful — they are not missing data. Horizon 13 weeks; what the business needs is P(demand > 0) and the size distribution, not a point estimate.

Notice what happened. Every choice was forced by a property of the data that a general-purpose model had ignored, and the result is not a small tweak — it is a different model with a different loss, a different scaler and a different output head. That is the honest scope of "foundation model for time series" today: a very good default, and a starting point for the twenty percent of problems that have structure the default was never shown.

The sixth axis, which nobody calls an axis: the corpus

Architecture papers get read; data sections get skimmed. In this family the data section is where most of the variance actually lives, and the five models differ on it more than they differ on anything else.

Look at Chronos versus TimesFM. Chronos has roughly three orders of magnitude less real data and closes the gap with two generators — convex mixtures of real series, and samples from randomly composed GP kernels. Both are cheap, both are auditable, and neither requires access to a search engine's logs. If you are outside a large company, that row is the one to copy.

And the contamination point from Chapter 6 is really a corpus point. "Zero-shot" is a claim about the relationship between two datasets, so it is only as strong as the corpus documentation. A model whose corpus is a proprietary log stream cannot support an auditable zero-shot claim, no matter how good its numbers are.

The empty cells

Cross the axes and most combinations have never been shipped. Here are the four that look most load-bearing, with what each would buy and what it would cost.

How to read the next one of these in ten minutes

A new time-series foundation model appears roughly every few months. Six questions, in this order, will tell you almost everything before you reach the results table.

If you can answer all six, you can predict the failure modes without running anything — and you will know which of the ten rows in Chapter 6's win-and-lose table your problem sits in.

Chapter 9: Legacy & Cheat Sheet

Three papers, eighteen months, one question answered three ways. What is actually different about forecasting now?

Not the accuracy, mostly. Chapter 6 was blunt about that: on a single clean seasonal series, a tuned classical model is within significance. What changed is the unit of deployment. Before, the unit was a fitted model per series. After, the unit is a checkpoint per organisation. That is the same relocation that language modelling went through, and it has the same consequence: the interesting engineering moves from "how do I fit this" to "what do I ask it".

The symbol table

The equations, in order

The numbers worth remembering

Build it yourself — the weekend recipe

Where to go from here

References

1. Nie, Y., Nguyen, N. H., Sinthong, P., Kalagnanam, J. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers," ICLR 2023 — arXiv:2211.14730. Patching, channel independence, and the masked-patch pretraining that everything after it uses.
2. Das, A., Kong, W., Sen, R., Zhou, Y. "A decoder-only foundation model for time-series forecasting," 2023 — arXiv:2310.10688. TimesFM: 200M parameters, O(100B) timepoints, the longer output patch and the first-patch masking trick.
3. Ansari, A. F., Stella, L., Türkmen, C. et al. "Chronos: Learning the Language of Time Series," 2024 — arXiv:2403.07815. Mean scaling plus uniform binning, an unmodified T5, TSMixup and KernelSynth, and a candid limitations section.
4. Goswami, M., Szafer, K., Choudhry, A. et al. "MOMENT: A Family of Open Time-series Foundation Models," ICML 2024 — arXiv:2402.03885. The Time Series Pile, masked pretraining for general-purpose analysis, and the imputation result that linear interpolation wins.
5. Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., Sahoo, D. "Unified Training of Universal Time Series Forecasting Transformers" (Moirai), ICML 2024 — arXiv:2402.02592. LOTSA, multi patch-size projections, and Any-variate Attention — the only serious attempt here at the channel problem.
6. Kim, T. et al. "Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift," ICLR 2022. RevIN, the normalisation layer three of the five models use.
7. Zerveas, G. et al. "A Transformer-based Framework for Multivariate Time Series Representation Learning," KDD 2021. The pointwise masked design PatchTST argues against, and the source of the BatchNorm-over-LayerNorm finding.
8. Zeng, A. et al. "Are Transformers Effective for Time Series Forecasting?" (DLinear), 2022. The linear baseline that forced this whole line of work to justify itself. Still the first thing to run.
9. Godahewa, R. et al. "Monash Time Series Forecasting Archive," NeurIPS 2021 Datasets and Benchmarks. The 30-dataset archive TimesFM tops zero-shot.
10. Gruver, N. et al. "Large Language Models Are Zero-Shot Time Series Forecasters" (LLMTime), NeurIPS 2023. The prompt-an-LLM alternative that TimesFM beats by more than 25% on Monash at a tiny fraction of the cost.
11. He, K. et al. "Masked Autoencoders Are Scalable Vision Learners," CVPR 2022. Where the "mask must exceed the correlation length" argument was made most sharply.