Latent Variable Models for Neural Data

One recorded trial gives you thousands of spikes and one hidden signal behind them. Factor analysis and Gaussian-process smoothing recover that signal from a single trial.

Learn how a few hidden numbers can explain a whole population of noisy neurons, and how to infer them trial by trial.

Press each method in turn, then drag the neuron count down to 4. Watch the orange estimate chase the blue hidden signal. Then we build, piece by piece, the model that does it: the hidden variable, the loadings, the fitting loop and the smooth prior.

You need a feel for averages and variance (how spread out numbers are) and a little matrix notation. We build the rest from zero.

Follows Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024, Jonathan Pillow), lecture 15 (slides) and lecture 19 (notes), and Yu et al. 2009 on GPFA. Every idea is re-taught here in our own words and devices.

Recover the hidden signal

10

A toy trial, not recorded data: one hidden signal, 50 bins of 20 ms, Poisson spike counts, square-root transformed as in Yu et al. 2009. Loadings are set from the simulation, as if learned on earlier trials. Smoothing uses a 40 ms Gaussian; GPFA picks its own timescale by likelihood. Errors are root-mean-square, in units of the signal’s own spread.

Chapter 0

Why Look for a Hidden Signal

See why a single trial of spikes looks like noise, and why a few hidden numbers can explain it

Picture a person with an electrode array in motor cortex (the strip of brain that plans movement). A cursor appears on a screen. They think about moving it to the right. The array records 96 channels, and you watch the spikes arrive.

A spike is a brief electrical pulse that one neuron fires. A spike train is the list of times at which it fired. Each neuron fires a little more or a little less as the plan unfolds. But any single neuron is erratic. It fires twice in one 20 ms window, then not at all in the next two.

Now you want to know what the brain is doing right now, on this one attempt. You do not have a hundred repeats to average. You have one trial: one try at the movement, from start to finish. This lesson is about getting a clean answer from a single trial.

The hunch: many neurons, few causes

Here is the idea in plain words. The neurons are not 96 separate stories. They share a few causes. The plan to reach right is one cause. The urgency of the reach is another. Each cause pushes many neurons at once, some up and some down.

Think of a stadium crowd doing a wave. Each person stands and sits on their own, and each person is a little late or a little early. But the wave itself is one smooth thing. If you watch ten people you can guess where the wave is. If you watch a thousand, you can see it clearly.

The wave is a latent variable. “Latent” means hidden: you never record it directly. You record only the people, and you infer the wave from them. In neural data, the people are neurons and the wave is the shared signal that drives them. Pillow’s lecture states the general idea this way: latent variable models explain observed data with some hidden structure.

The lecture draws the picture with simulated data. A raster shows 50 neurons over time as rows of dots. It looks like static. Then a second picture shows a few smooth curves, the inferred latent variables, that the dots were secretly following. The raster is the data. The curves are the explanation.

Why a trial average is not enough

The old answer is to repeat the trial many times and average. If the signal is the same each time, noise cancels and the signal stays. That works, and it built the field. But it has a hole.

The signal is not the same each time. On one attempt the person reaches fast. On the next, slowly. The hidden bump arrives a little early or a little late. Averaging lines the trials up in time and then smears them together. The average is lower and wider than any real trial.

Try it. The device below draws six trials of the same hidden signal, each shifted in time by a small random amount. The thick line is their average. Drag the timing spread up and watch the average lose its peak.

Average the trials and watch the peak melt

Thin blue lines: the same hidden signal on six trials, each shifted in time. Thick orange line: the trial average. Drag the spread of timing between trials.

80 ms

A toy signal, not recorded data. The six shifts are fixed multiples of the spread (about −1.2, −0.6, −0.2, 0.3, 0.8 and 1.3 times it). Each trial’s signal is scaled to the same overall spread before averaging.

The cure is to read the hidden signal on each trial separately. Then each trial keeps its own speed and shape. Averaging after you have read it is still available, if you want it. But now you can also ask new questions: why was this reach slow, and what was the brain doing when it was?

The same trick appears all over neuroscience

The lecture lists three other places where a hidden variable explains messy data. In electrophysiology, spike sorting infers which neuron fired each pulse that the electrode saw. In calcium imaging, you infer where the cells are and when they fired from a blurry movie. In EEG, source separation infers the brain sources that mixed into each scalp sensor. Our own lessons on independent component analysis and spike sorting live in the same family.

What is new in this lesson is the neural population on one trial. We need three tools, in this order. First, the plain idea of a latent variable model and the two jobs it must do. Second, factor analysis, the simplest model that finds shared structure and ignores private noise. Third, a smooth prior over time, which gives Gaussian-process factor analysis, or GPFA.

The road

  1. The recipe: a prior over the hidden variable and a mapping from it to data.
  2. Why the likelihood is hard: you must sum over every hidden value.
  3. Factor analysis: shared variance through a few factors, private variance per neuron.
  4. Factor analysis against PCA: why noisy neurons should count for less.
  5. Inference and EM: reading the hidden variable, and fitting the model by alternating.
  6. Probabilistic PCA and how many factors: the closed-form cousin, and held-out error.
  7. Time: smooth first, reduce second, and the trade-off that hides.
  8. GPFA: smoothing and reduction in one model.
  9. Single-trial trajectories: the hidden path of the population, one reach at a time.
  10. Field Guide: the numbers, the code, and where to go next.

The hero at the top is the finished instrument. Chapter 8 explains its third button, and Chapter 7 its second.

Why can a trial average hide what the brain did on a single attempt?

Chapter 1

The Recipe: A Prior and a Mapping

Build any latent variable model from two parts, then use it to run a guess backwards from data to cause

You find a number on a scrap of paper: 0.8. A friend tells you it came from one of two machines. Machine A prints numbers near −1.5. Machine B prints numbers near +1.5. Both are a bit noisy. Which machine printed your 0.8?

You answer without thinking: probably B, because 0.8 is closer to 1.5. You have just done latent variable inference. The machine is the hidden variable. The number is the data. You ran the story backwards, from the number you saw to the machine you did not.

Two parts make a model

Pillow’s lecture says every latent variable model is defined by two pieces. The first is a prior over the latent variable: how likely each hidden value is before you see any data. For the two machines, it is how often each one is used. The second is a mapping: given the hidden value, how likely is each data value? The lecture calls it the conditional probability of the data given the latent. For machine B, it is a bell curve around +1.5.

Multiply them and you get the joint distribution, the probability of a hidden value and a data value together. This is how the model would generate data: draw a hidden value from the prior, then draw data from the mapping.

p(x, z)
The joint: how likely it is that the hidden value is z and the data value is x.
p(z)
The prior over the hidden value. Which machine do we use, and how often?
p(x | z)
The mapping. The bar reads “given”. If machine z is running, how likely is the number x?

Two jobs, and they are not the same

The lecture names two things we want to do with such a model. The first is inference, also called recognition: given one data point, work out the hidden value. That is the scrap of paper. The second is fitting: given a whole dataset, work out the settings of the prior and the mapping, such as where each machine’s bell curve sits. We deal with fitting in Chapter 5.

Recognition is Bayes’ rule. Take the joint at the number you saw, and divide by the total over every hidden value, so the answers add up to 1. The result is the recognition distribution, also called the posterior.

p(z | x)
The recognition distribution: how likely each hidden value is, now that you have seen x.
p(x | z) p(z)
Mapping times prior, the joint at the data you saw. A hidden value scores high if it is common and also explains x well.
∑ p(x | z′) p(z′)
The same product added over every hidden value. It rescales the scores to sum to 1. Chapter 2 explains why this sum is the hard part.

Try it on the two machines. The lecture works this exact case, a binary mixture of Gaussians, on the board. Move the observed number and watch the posterior split.

Guess which machine printed the number

Purple bump: machine A, weighted by how often it runs. Orange bump: machine B. The teal line is the number you observed. The bar under it splits the posterior.

0.8
50%

A toy: machine A centred at −1.5 and machine B at +1.5, both with spread 1. The numbers are illustrative; the rule is Bayes’ rule, as in the lecture’s binary mixture example.

At 0.8 with equal use, the bell of B is worth 0.312 there and the bell of A only 0.028. Divide: 0.312 / (0.312 + 0.028) = 0.917. The model says B with 92% probability. Slide the prior toward A and the answer moves. A rarely used machine needs strong evidence.

One recipe, many models

The lecture then makes a point that organises the rest of this series. Change the prior and the mapping, and you get a different famous model. A discrete prior with Gaussian mapping is a mixture of Gaussians, which is clustering. A Gaussian prior with a linear Gaussian mapping is factor analysis, and PCA is a special case. A Gaussian prior that moves in time with linear dynamics is a linear dynamical system, the engine inside the Kalman filter. A discrete prior that hops between states is a hidden Markov model.

ModelPrior over the hidden variableMapping to data
Mixture of GaussiansdiscreteGaussian
Factor analysisGaussianlinear, Gaussian
Linear dynamical systemlinear Gaussian dynamicslinear, Gaussian
Hidden Markov modeldiscrete transitionsany

We have lessons on the cousins: EM for Gaussian mixtures, the hidden Markov model, and the Kalman filter. This lesson takes the second row and then adds time to it, which lands between rows two and three.

In a latent variable model, which pair of ingredients defines the model?

Chapter 2

Why the Likelihood Is Hard

See how a hidden variable turns a simple sum into an integral, and how a Gaussian model escapes it

In Chapter 1 you scored two machines. You multiplied, you added two numbers, you divided. Easy. Now the machine is not A or B. It is a dial, and it can sit at any value between −5 and 5. How many “machines” are there? Infinitely many.

That one change is the source of most of the pain in this field. Let us see exactly where it bites.

A GLM where you do not know the input

Recall the generalised linear model (GLM), and the spiking version in the point-process GLM lesson. A GLM predicts a neuron’s spikes y from an input x that you know: a stimulus, or the cursor velocity. To fit it, you write the likelihood, the probability the model gives to the spikes you recorded, and you make it large. For a Poisson neuron, the log-likelihood is a plain sum over time bins. Nothing is hidden. Pillow’s slide puts it this way: fit using the sum of yt log λt minus λt, where each λt is a fixed function of x and the weights.

The lecture then says: a latent variable model is a GLM where you do not know the input. The “stimulus” is the hidden signal, and you must find it from the spikes themselves. That is the chicken-and-egg problem on the slide. To learn the weights you need the input. To find the input you need the weights.

Sum over every hidden value

The way out is to ask: what probability does the model give to the data, whatever the hidden value was? You add up the joint over every possible hidden value. For two machines that was two terms. For a continuous hidden variable, the sum becomes an integral: the area under a curve. This quantity is the marginal likelihood, or evidence.

p(x)
The marginal likelihood: how probable the data are, averaged over every hidden value. This is what we want large when fitting.
∫
The integral sign: add up the thin slices of area as z moves along its whole range.
p(x | z)
How well the hidden value z explains the data x. It is high when z predicts x closely.
p(z) dz
How likely z was to begin with, times a thin slice of width dz.

The device draws the two curves that get multiplied for one neuron. The neuron’s count is x = A z + noise. You set the loading A (how strongly the hidden value drives the neuron), the noise variance ψ, and the count you observed.

Add up every hidden value

Grey line: the prior over the hidden value z. Blue: how well each z explains the observed count. Orange fill: their product. Its area is the evidence. Drag the sliders.

1.0
1.0
2.0

A toy with one neuron and one hidden value, prior N(0, 1). Each curve is stretched to fill the plot; the evidence is computed from the true heights with 400 slices, then checked against the closed form N(x; 0, A² + ψ).

Take A = 1, ψ = 1 and x = 2. The slices add to 0.1038. Now look at the closed form from the lecture: the evidence is a Gaussian centred at 0 with variance A² + ψ = 2. At x = 2 that is exp(−4/4) / √(4π) = 0.3679 / 3.545 = 0.1038. The same number, with no slicing.

Why this matters, and why it usually fails

This closed form is a gift of the Gaussian. Gaussians multiply into Gaussians, and a Gaussian integrates to a Gaussian. Change one ingredient and the gift is gone. If the mapping is a Poisson spike model, with rate eA z, the integral has no formula. You must approximate it. The lecture puts the problem bluntly: for a model whose latent runs through time, the likelihood needs a high-dimensional integral, one dimension for every time bin.

For that reason the lecture lists three ways to fit a latent variable model:

  1. Sampling. Draw hidden values and parameters in turn from their conditional distributions (Gibbs sampling). Slow, but fully Bayesian.
  2. Expectation maximisation (EM). Alternate between guessing the hidden values given the parameters and improving the parameters given the guesses. Chapter 5.
  3. Variational inference. Optimise a lower bound on the quantity you want. The lecture on factor analysis uses this view to derive EM.

This lesson uses the second, because for Gaussian models both steps are exact. Our EM for Gaussian mixtures lesson shows the same loop on clusters. The variational auto-encoder uses the third idea with neural networks.

Why is a latent variable model harder to fit than a GLM with a known input?

Chapter 3

Factor Analysis: Shared Variance and Private Noise

Build the simplest hidden-variable model of a population, and read it as a model of the covariance

Two neurons sit side by side in motor cortex. When one fires more than usual in a 20 ms bin, the other tends to fire more too. A third neuron does the opposite. Is that a coincidence? Over thousands of bins it is not. Something shared pushes them together, and something private makes each one wobble on its own.

Factor analysis puts those two kinds of variation into one model. It is the oldest and plainest latent variable model for continuous data, and it is the engine inside GPFA.

The model, one line at a time

We have n neurons. In one time bin they give a vector x of n numbers (spike counts, or their square roots, which we explain in Chapter 7). We say a small number d of hidden values z drive all of them. Here is the generative story in the notation of Pillow’s lecture notes.

z ~ N(0, Id)
The hidden vector. Each of its d entries is an independent bell-curve number with mean 0 and variance 1. These are the factors.
x = A z + ε
The data. A is an n-by-d matrix of factor loadings. Column j says how strongly factor j pushes each neuron, and in which direction.
ε ~ N(0, Ψ)
Private noise. Ψ is a diagonal matrix, so each neuron has its own noise variance ψi, called its uniqueness, and the noise of different neurons is independent.

Shapes help. For 96 neurons and 10 factors, z is 10 numbers, A is a 96 by 10 table, and x is 96 numbers. A tall, thin matrix turns a short list into a long list. That is what “a few causes, many effects” means in code. The parameters are θ = {A, Ψ}.

What the model says about covariance

Take the model and ask what the data look like when you forget z. The lecture notes show the answer by the standard Gaussian rules: the data are Gaussian with mean 0 and covariance A AT + Ψ.

p(x)
The marginal: the distribution of the data once the hidden vector is integrated out (Chapter 2).
A AT
The shared part. It has rank at most d, so it is a few directions of joint variation. Every off-diagonal entry comes from here.
Ψ
The private part. It is diagonal, so it adds to each neuron’s own variance and to nothing else.

So factor analysis is a model of the covariance (the table of how every pair of neurons varies together). It writes that table as low rank plus diagonal. Move the sliders to see the table and the split for each neuron.

Split each neuron’s variance

Left: the covariance of eight neurons. Purple cells co-vary up, blue cells co-vary in opposite directions. Right: each neuron’s variance, purple for shared, grey for private. Drag the two sliders.

1.0
1.0

A toy with one factor: loadings 0.9, −0.6, 0.7, 0.3, −0.8, 0.5, 1.0, −0.4 times the shared strength, and private variances 0.4 to 0.7 times the private-noise slider. All numbers are illustrative.

Read neuron 3 at the default settings. Its loading is 0.7, so the factor gives it shared variance 0.7² = 0.49. Its private variance is 0.5. Total: 0.99. About half of what neuron 3 does is shared. Turn the private noise to 3 and the shared share drops to 25%, while the off-diagonal cells do not change at all. Noise lives only on the diagonal.

Why not just keep the covariance?

Count parameters. A full covariance for 96 neurons has 96 × 97 / 2 = 4,656 free numbers. Factor analysis with 10 factors has 96 × 10 loadings plus 96 uniquenesses, which is 1,056 numbers, and a bit fewer after the freedom described below. A recording of a few hundred trials cannot pin down 4,656 numbers well. It can pin down about a thousand much better. The factor model is a deliberate simplification, a bet that the world has low-dimensional shared structure.

In factor analysis, which part of the covariance can the hidden factors explain?

Chapter 4

Factor Analysis Against PCA

Find out why a noisy neuron should count for less, and why PCA cannot know that

Two thermometers hang in the same room. Both track the room’s temperature. One is good. The other is faulty and jumps around by ten degrees. You want the room’s temperature. Which reading do you trust?

You trust the good one, of course. Now suppose a program must find “the main direction of variation” in the pair of readings. If it simply picks the direction with the most variance, it will point at the faulty thermometer. That is the one that varies the most.

Two ways to model a covariance

You know PCA: find the directions in which the data spread the most. Pillow’s notes say both PCA and factor analysis are models of the covariance, and they differ in one choice. PCA says the covariance is low rank: a few directions carry it all. Factor analysis says it is low rank plus a diagonal: a few shared directions, plus an independent amount of noise for each neuron.

That diagonal is a “fudge factor” in the notes’ words. It lets each neuron have its own noise. PCA has no such slot, so it must treat all variance as signal.

The lecture’s two-neuron example

The notes build the smallest case. One hidden value drives two neurons. Both load equally, with loading 1, so A = [1, 1]T. But the noise on neuron 1 has ten times the standard deviation of the noise on neuron 2: Ψ = diag(100, 1). The data covariance is then A AT + Ψ, which is a 2 by 2 table with 101 and 2 on the diagonal and 1 off it.

PCA looks at that table and sees that axis 1 has variance 101 and axis 2 has variance 2. Its top direction points almost straight along neuron 1, about 0.6 degrees off the axis. But the truth is that the hidden value pushes both neurons equally, so the shared direction is the diagonal [1, 1]. Factor analysis recovers that, because it knows neuron 1’s big variance is private noise.

Drag the noise on neuron 1 and watch the two arrows. The cloud is 250 draws from the model.

Make one neuron noisy

Each dot is one time bin: neuron 1 on the horizontal axis, neuron 2 on the vertical. The orange line is PCA’s top direction. The purple line is the factor loading. Drag the noise of neuron 1.

10.0

The notes’ example: loadings [1, 1], noise std of neuron 2 fixed at 1. At std 10 the covariance is [[101, 1], [1, 2]]. The cloud is simulated; the angles and weights are exact.

At noise 10 the PCA arrow lies 0.6 degrees from neuron 1’s axis, and the factor line stays at 45 degrees. At noise 1 both neurons are equally noisy, and the two lines agree. The disagreement grows exactly as one neuron gets less trustworthy.

The same idea in the inference weights

The device also shows the weights used to guess the hidden value from one data point. Chapter 5 derives them. For this example, the weight on neuron 1 is 0.005 and the weight on neuron 2 is 0.498. The model almost ignores the noisy neuron. If you see x = [3, 2], the estimate is 0.005 × 3 + 0.498 × 2 = 1.01. A projection on PCA’s axis would give nearly 3. The notes say it plainly: project onto [0, 1] instead of [1, 0].

Two invariances that decide which to use

The notes end this comparison with two facts worth keeping.

  1. PCA ignores rotation. Spin the data and PCA returns the same components, spun. FA does not. Spinning moves shared variance onto the axes, where FA reads it as private.
  2. FA ignores per-axis scale. Multiply neuron i by α and FA scales that row of A by α and that noise by α², and changes nothing else. PCA can change completely.

For neurons, the second fact is the one that matters. Neurons fire at different rates, so their variances differ by large factors. Each neuron is its own axis with its own units. A method that survives rescaling an axis is a better fit than one that lets the loudest neuron decide. The first fact is a smaller cost: “neuron 5” is a real thing, so the axes are not arbitrary.

One neuron has ten times more private noise than the others. Which method will tend to point its main axis at that neuron?

Chapter 5

Reading the Hidden Variable, and Fitting the Model

Derive the recognition distribution, then alternate between guessing hidden values and improving the loadings

You have a factor model, and a data vector arrives. First question: what was the hidden value? Second question, harder: you do not yet have the loadings A or the noise Ψ. How do you get them?

Each question seems to need the answer to the other. If you knew A and Ψ, you could read the hidden values. If you knew the hidden values, you could fit A and Ψ by plain regression. Expectation maximisation breaks the circle by taking turns.

Step one: the recognition distribution

Fix A and Ψ. Chapter 1 told us the recognition distribution is the mapping times the prior, rescaled. Both factors are Gaussian, so the product is Gaussian. The notes find it by completing the square: write the exponent as a quadratic in z, collect the z2 terms and the z terms, and read off the mean and the covariance.

The exponent of the product is −½ (A z − x)T Ψ−1 (A z − x) − ½ zT z. Expand it. The z2 terms are zT (AT Ψ−1 A + I) z. The terms linear in z are −2 zT AT Ψ−1 x. A Gaussian in z with covariance Λ and mean μ has exponent −½ zT Λ−1 z + zT Λ−1 μ. Match the pieces.

Λ
The posterior covariance: how unsure we still are about z. It does not depend on x. Quiet neurons (small ψ) with large loadings shrink it.
μ
The posterior mean: our best guess for z. It first multiplies x by Ψ−1, so each neuron counts in inverse proportion to its noise, then projects with AT.

Run the numbers from the last chapter. A = [1, 1], Ψ = diag(100, 1). Then AT Ψ−1 A = 1/100 + 1 = 1.01, so Λ = 1 / 2.01 = 0.4975. For x = [3, 2], μ = 0.4975 × (3/100 + 2/1) = 0.4975 × 2.03 = 1.010. The mean is 1.01 and the spread is 0.4975 (a standard deviation of 0.70). The noisy neuron barely moved it.

Step two: improve the loadings

Now treat the guesses as if they were data. The notes use a Gaussian guess for every sample, N(μi, Λ). The new loadings are a ridge-regression solution from guessed hidden values to the data. The new noise is the leftover error, plus a correction for our remaining doubt about z.

Â
The new loadings. The term NΛ is the doubt: it keeps the loadings from growing as if the guesses were exact.
Ψ̂
The new uniquenesses. Only the diagonal is kept, because the model says noise is independent across neurons.

A tiny case by hand. Keep the A and Ψ above and take two samples, x1 = [3, 2] and x2 = [−1, −2]. The E-step gives μ1 = 1.010 and μ2 = 0.4975 × (−0.01 − 2) = −1.000. Sum of μ² is 1.020 + 1.000 = 2.020, and NΛ = 0.995, so the denominator is 3.015. The numerator for neuron 1 is 3 × 1.010 + (−1)(−1.000) = 4.030, so its new loading is 1.337. For neuron 2 it is 4.020 / 3.015 = 1.333. The new uniquenesses are 2.307 for neuron 1 and 1.320 for neuron 2.

Noise estimates moved toward the data: neuron 1 went from 100 down to 2.3, because two samples show it was never that noisy. With so little data, treat this as a drill and not as a result.

Repeat the two steps. The lecture shows why each round cannot make the fit worse: both steps raise a lower bound called the negative free energy, which touches the true log-likelihood after the E-step. Step the loop on a population of four neurons below.

Take turns until the fit settles

Left: log-likelihood per sample after each EM round. Right: estimated loading (orange bar) against the loading that made the data (blue tick). Press Step, then Run 20.

150 simulated samples of four neurons from one factor. True loadings 1.0, 0.8, −0.6, 0.5 and uniquenesses 0.3, 0.2, 0.4, 1.0 are labelled; the start is loadings 0.2, 0.1, 0.15, 0.05. The sign of a factor is arbitrary, so the bars are flipped to match the true signs.

Watch the first rounds. The log-likelihood jumps, then flattens: by round 10 it sits within 0.01 of its round-40 value. Each round costs one pass over the data, and no round ever lowers the fit.

In the factor analysis posterior mean μ = Λ AT Ψ−1 x, what does the Ψ−1 do?

Chapter 6

Probabilistic PCA and How Many Factors

Get a closed-form factor model by fixing the noise to be equal, then pick the number of factors from held-out data

EM took many rounds in Chapter 5. Sometimes you want an answer in one line of linear algebra. And whichever method you use, one question has been hanging since Chapter 3: how many factors? One? Ten? Forty?

This chapter answers both. Make the noise equal on every neuron and the fit has a formula. Then test each number of factors on data the model has never seen.

Equal noise gives a formula

Probabilistic PCA (PPCA) is factor analysis with one restriction: Ψ = σ² I. Every neuron has the same private noise. It is less flexible than FA, since it cannot down-weight a noisy neuron. In return, Tipping and Bishop showed in 1999 that it needs no iteration. The notes give the recipe. Take the sample covariance and its eigenvectors bi and eigenvalues λi, sorted largest first. Keep d of them.

  1. Directions. The loading directions are the top d eigenvectors, exactly as in PCA.
  2. Noise floor. The noise variance σ² is the average of the eigenvalues you throw away.
  3. Lengths. Each loading has length √(λi − σ²). It is the eigenvalue with the noise floor taken off.

That last step is the whole difference from PCA. PCA reports λi as the strength of a direction. PPCA says part of that is noise that every direction carries, and subtracts it. If the discarded eigenvalues are tiny, the two methods agree.

The device below simulates 20 neurons driven by three real factors with different private noise. Its top three eigenvalues are 28.05, 14.81 and 9.04. The next is 1.13, then 0.99 and so on down to 0.21. With d = 3, the 17 discarded eigenvalues average 0.62, so σ² = 0.62 and the first loading has length √(28.05 − 0.62) = 5.24.

What goes in and what comes out

Follow the data flow once. The input is a matrix of 200 training samples by 20 neurons. You centre it and form the 20 by 20 sample covariance. An eigen-decomposition returns 20 eigenvalues and a 20 by 20 matrix of eigenvectors. For a chosen d, the output is a 20 by d loading matrix A and one number, σ². Nothing is iterated, so the whole fit costs one eigen-decomposition, a few milliseconds here.

Compare this with Chapter 5, where EM needed dozens of passes over the data. The price of the shortcut is the equal-noise assumption. If one neuron is much noisier than the rest, as in Chapter 4, PPCA will blame the wrong direction, and you should return to factor analysis.

The test that picks d

The scree of eigenvalues shows a cliff after three, but real data rarely cooperate so cleanly. Yu and colleagues use a more honest test, and so should you. Split the trials into folds. Fit the model on some of them. On held-out trials, hide one neuron, infer the hidden value from the other neurons, and predict the hidden neuron’s activity. Add up the squared errors across neurons. Repeat for every candidate d. That is leave-one-neuron-out cross-validation.

The prediction step is Chapter 5 again, now without neuron j: μ = (A−jT A−j + σ² I)−1 A−jT x−j, and the guess for neuron j is Aj μ. Because neuron j is never used to find μ, a model cannot score well by memorising noise.

Choose how many factors to keep

Top: the eigenvalues of the covariance. Orange bars are kept, grey bars are discarded; the dashed line is the noise floor PPCA sets. Bottom: held-out prediction error for each number of factors. Drag the slider.

3

A simulation, not recorded data: 20 neurons, three true factors, private noise 0.3 to 1.0, 200 training and 200 test samples. Error is mean squared leave-one-neuron-out prediction on the test samples. Yu et al. 2009 ran the same test with 4-fold cross-validation on 61 real units.

Slide to d = 1. The noise floor jumps to 1.81, because two real factors are hiding in the “discarded” part and are being counted as noise. The held-out error is 1.96. Slide to d = 3. The floor drops to 0.62, and the error to 0.76, the lowest. Slide on to d = 8. The error creeps back up to 0.79: extra factors fit noise from the training set that the test set does not share.

Yu et al. ran this kind of test on 61 real motor and premotor units. For the reach target they showed, the prediction error of their reduced GPFA model bottomed out at 10 dimensions. Across the 14 reach targets, the best dimension ranged from 8 to 12, with 10 the most common.

In probabilistic PCA, how is the noise variance σ² estimated?

Chapter 7

Time: Smooth First, Reduce Second

Turn spikes into numbers a Gaussian model can use, then find the smoothing width that the two-stage recipe hides

Factor analysis has so far treated each time bin as a separate sample. It would give exactly the same answer if you shuffled the 50 bins of a trial. But the brain is not shuffled. The hidden signal at 300 ms is close to the hidden signal at 320 ms. A model that ignores this wastes information.

Before we fix that, we need to make the data fit the model, and we need the old fix everybody tried first.

From spikes to Gaussian-friendly numbers

Factor analysis assumes Gaussian noise with the same variance on every sample. Spikes break that. First we bin the spike train: count the spikes in each 20 ms window. Yu and colleagues used 20 ms bins. A count is a small whole number, 0, 1, 2, and its noise is not Gaussian.

It is also not constant. A Poisson count has variance equal to its mean, so a busy neuron is noisier than a quiet one, and a neuron is noisier at the peak of its response than at the trough. That breaks the model’s promise of a fixed ψi per neuron.

The usual fix is the square-root transform: replace each count by its square root. A quick simulation shows why. Counts drawn at rate 0.3 have variance 0.30. At rate 10, they have variance 10.0. After the square root, the variance stays between 0.22 and 0.40 over rates from 0.3 to 10, and settles near one quarter at high rates. The transform makes the noise nearly the same for all firing rates, which is what Ψ assumes. Yu et al. square-rooted their 20 ms counts for this reason.

The old recipe: smooth, then reduce

The square-rooted counts still jitter from bin to bin. The natural repair is to smooth each neuron’s train in time. A Gaussian kernel does that: each bin is replaced by a weighted average of its neighbours, with weights that follow a bell curve. The width of the bell (its standard deviation) sets the smoothing. Wide kernels average many bins and look calm. Narrow ones keep sharp features and stay noisy.

Then you run PCA, PPCA or FA on the smoothed data. Yu et al. call these two-stage methods. In our toy, step two is the posterior mean from Chapter 5, applied bin by bin: μt = (1 + s)−1 Σi Ai (zit − di) / ψi, where s = Σi Ai² / ψi and di is the neuron’s baseline.

Try it. The top panel shows one trial’s estimate at the chosen width. The bottom panel shows the error against width, averaged over 12 trials, with a dashed line for the method of the next chapter.

Find the width that fits

Top: the hidden signal (blue, dashed) and the two-stage estimate (orange) for one trial. Bottom: error against kernel width over 12 trials; the dashed line is what GPFA gets without any width. Drag the width.

40 ms

A toy: 12 simulated neurons, 20 ms bins, 12 trials, square-root counts, loadings set from the simulation. Error is root-mean-square against the hidden signal, in its own units. The best width here is shaped by this toy’s signal; Yu et al. found about 40 ms best for PPCA and FA on their real data.

Where the recipe goes wrong

Look at the U. Symptom: with no smoothing, the estimate jitters and the error is 0.40 or more. With a very wide kernel, the estimate is a flat smear and the error is worse than none. Measurement: the curve has a clear minimum, so a width exists, but the data do not announce it. Someone has to scan for it with held-out data, as Yu et al. did.

Worse, the recipe makes three silent choices. The same width applies to every neuron, though some neurons carry fast features and some slow. The same width applies to every hidden dimension, though one factor may drift slowly while another oscillates. And the smoothing never sees the factor model, so the model cannot ask for more smoothing where it is unsure.

The fix is to put both stages in one model, so that the smoothness is a parameter learned together with the loadings. That is GPFA. If you know the RTS smoother from Kalman filtering, the spirit is similar: the estimate at each time borrows from its neighbours, and how much it borrows is set by a model, not by hand.

Why do we take the square root of spike counts before factor analysis?

Chapter 8

GPFA: Smoothing and Reduction in One Model

Give each hidden factor a smooth random-curve prior, so that the model learns how smooth it should be

Draw a wobbly line on paper without lifting the pen. You can draw a lazy line that barely bends. You can draw a jittery line that changes direction every few millimetres. Either way, the line is continuous: where it is now tells you something about where it will be a moment later. How much it tells you depends on the line’s character.

A hidden neural signal behaves like that line. Gaussian-process factor analysis, from Yu, Cunningham, Santhanam, Ryu, Shenoy and Sahani (2009), turns that into a prior.

A random curve with a memory length

A Gaussian process (GP) is a rule for generating random curves. Instead of giving each time bin its own independent bell curve, it says: the values at all times together are one big Gaussian whose covariance depends on how far apart the times are. Our Gaussian processes lesson builds this in depth.

GPFA uses the squared-exponential kernel. The covariance between the hidden value at time t and at time t′ is

K(t, t′)
How strongly the hidden value at time t moves with the value at t′. Close times give a number near 1. Far times give a number near 0.
σf²
The signal variance. The paper fixes it to 1 − σn², so K(t, t) = 1 and each factor has unit scale.
τ
The timescale: the memory length of the curve. A larger τ gives slower, smoother curves. This is the only kernel setting the paper learns, one for each factor.
σn²
A tiny nugget, fixed at 10−3 in the paper. It keeps the matrix well-behaved and adds almost nothing.

Work out one value. With τ = 4 bins (80 ms), two bins 3 apart have covariance e−9/32 = 0.755. Bins 8 apart have e−64/32 = 0.135. So a value says a lot about its near neighbours and almost nothing about values 160 ms away. The top of the device draws random curves from this prior at the τ you choose.

The whole model, and what changes from Chapter 3

The observation model is the factor model you know, now with a baseline: yt = C xt + d + ε, with ε ~ N(0, R) and R diagonal (so each neuron has its own noise). Here yt holds the square-rooted counts of q neurons in bin t, xt is the p-dimensional hidden state, C is q by p, and d is each neuron’s baseline. The one change from factor analysis is the prior. Instead of drawing xt independently in each bin, each of the p rows of x gets its own GP, over all T bins together.

This is how smoothing and reduction share one model. The loadings C reduce the dimension. The kernel makes the hidden curves smooth. Both are fitted at once by EM, and the E-step is still exact because everything is Gaussian. The M-step updates C, d and R in closed form, as in factor analysis, while each τ is updated by gradient steps. As with Chapter 5, EM converges to a local optimum.

Reading the hidden curve

For one factor the posterior mean has a neat form. Gather each bin’s evidence into bt = Σi Ci (yit − di) / Ri, and let s = Σi Ci² / Ri. These are the same quantities as in Chapter 5. Then the whole curve is the vector K (I + s K)−1 b: one T by T linear solve. For T = 50 bins it is a 50 by 50 system, instant in a browser. The same solve gives the likelihood of each τ, which is how the hero picks its timescale.

Set how much the hidden curve remembers

Top: three random curves drawn from the prior at your τ. Bottom: the hidden signal (blue, dashed) and the GPFA estimate (orange) from the same spikes. Drag τ.

100 ms

One simulated trial with 10 neurons (not recorded data), loadings set from the simulation. Likelihood is the marginal likelihood of the spikes under each τ; it is what the paper’s EM climbs. In Yu et al. the first five learned timescales were 54, 160, 293, 173 and 111 ms.

At τ = 20 ms the curves in the prior are nearly white noise, and the estimate follows every blip in the spikes. At τ = 400 ms the curves are nearly straight lines and the estimate cannot bend to follow the peak. Between them the likelihood has a peak. That is the value EM would learn, and it is also near the lowest error.

On real data the timescales differ by factor. Yu et al. report learned timescales of 54, 160, 293, 173 and 111 ms for their first five dimensions (with 15 fitted). A two-stage method has one width for all of them. GPFA gives every factor its own.

What does GPFA change compared with plain factor analysis?

Chapter 9

Single-Trial Trajectories

Draw the path a whole population takes on each trial, and read what the paper found when it did

Imagine a monkey seated at a screen. A target lights up. The animal must wait for a go signal, then reach. Between the light and the reach there is a delay, and in that delay nothing moves. But motor cortex is busy. The population is getting ready.

Yu and colleagues recorded that delay. A 96-channel array sat over dorsal premotor and motor cortex of one rhesus macaque. After removing ten channels, they kept 61 units in one session (18 single neurons and 43 multi-unit signals). They had 784 trials of a delayed centre-out reach to 14 target positions. Their question: what does the population do, trial by trial, while the animal prepares?

A population is a point that moves

With p = 3 hidden factors, the state of the population at one moment is a point in a 3-D box. A moment later the point has moved. Join the points over a trial and you get a neural trajectory: the path of the population through its own state space. Our neural manifolds lesson explains why such paths live on a low-dimensional surface.

GPFA gives one such path for every trial, from just that trial’s spikes. The model has already been fitted on many trials, so that C, d, R and the timescales are known. The trajectory for a new trial is then the posterior mean from Chapter 8, a single linear solve.

Making the axes mean something

Chapter 3 warned that factors are only known up to a rotation. For drawing, the paper settles the choice with a singular value decomposition of the fitted C. That orthonormalises the loadings and orders the dimensions by how much data covariance each explains. The first axis of the plot is the most important one. That is why the paper fits 15 factors, orthonormalises, and keeps the top few for display or for prediction.

Now the device. It simulates three targets with five trials each. Every trial follows its target’s path, but with its own timing and size, so no two trials match. Choose the method and the neuron count. The faint thick lines are the true paths. Dots mark every 100 ms.

Draw each trial’s path

Each colour is one target, with five trials. Faint thick lines are the true paths; thin lines are the recovered paths of single trials. Pick a method, then drag the neuron count down.

8

A toy, not recorded data: 15 simulated trials, spikes from the sine-and-bump paths above, square-root counts, 40 ms kernel for the two-stage method, τ = 100 ms for GPFA, loadings known. Both methods are given the best linear alignment to the true axes, which no experimenter has. Distances are in the latent’s own units.

With 24 neurons both methods land closest to the true paths. With 8, the estimates wander, and the average distance to the truth grows for both. In this toy, GPFA stays a little closer on most draws. Do not over-read that: the gap is small, and both methods here get the help of an alignment to the truth. The paper’s test is harder and fairer: predicting a held-out neuron. On that test, the best two-stage method (FA with a 40 ms kernel) still had higher error than GPFA, and the authors note it remains to be seen how much that matters for behaviour.

What the paper found

On the real data, the authors report that single-trial trajectories converged over time during motor planning. In plain words: trials to the same target start with a scattered cloud of states and end the delay in a tighter cloud. They measured this with a ratio of the volumes of the covariance ellipsoids of the trial cloud, taken over the top 10 dimensions and reduced to a ratio per dimension (the 10th root), on trials with delays over 400 ms. Across the 14 targets, the ratio per dimension was 1.31 ± 0.13 from target onset to the go cue and 1.17 ± 0.14 from the go cue to movement onset, both above 1 (P < 0.001). The authors read a ratio above 1 as convergence.

They also show one odd trial. On one trial the animal was slow to move, with a reaction time of 844 ms, far longer than usual. Its trajectory looked very different from the others. Yet its activity around movement onset matched the other trials well. The paper presents this as an illustration that single-trial paths may relate to behaviour, and leaves subtler links for later work. A trial average could never have shown it.

What this model cannot do

The paper is frank about its assumptions. All of PCA, PPCA, FA and GPFA assume a linear link in square-root count space, so the true surface may be curved. GPFA specifies no dynamics: it says the path is smooth, not what pushes it along. The authors list extensions: coupling the kernels of different factors, kernels that change over time, nonlinear maps, and point-process spike models. The learned timescales depend on the starting values. Over four starts, the mean of the 15 learned timescales ranged from 125 to 155 ms, while the likelihoods and trajectories stayed very similar. And plotting three dimensions may hide structure in the rest. The next lessons in this series pick up exactly these gaps: dynamics (the Kalman and linear dynamical systems of the Kalman decoder) and nonlinear latents.

Why can a single-trial trajectory show something a trial average cannot?

Chapter 10

Field Guide

Keep the numbers, the terms, the checklist and the code, and find where to go next

A latent variable model explains many noisy measurements by a few hidden numbers. You define it by a prior over the hidden variable and a mapping from hidden to data. Fitting needs a sum over all hidden values, so we alternate: guess the hidden values, then improve the parameters (EM). Factor analysis is the Gaussian case. It writes the covariance as low rank plus diagonal, which lets each neuron have its own noise. Probabilistic PCA fixes the noise equal and has a closed form. The number of factors comes from held-out prediction. Spike counts need binning and a square root first. The two-stage recipe smooths and then reduces, with a width chosen by hand. GPFA gives every factor a Gaussian-process prior, so smoothness is learned with the loadings, and it returns one path per trial.

The numbers that matter

WhatNumberSource and why it matters
Two-neuron example, covariance[[101, 1], [1, 2]]NEU 560 L19 notes; PCA points at the noisy neuron
Recognition weights, same example0.005 and 0.498; estimate 1.01 for x = [3, 2]Derived from the notes’ formula; noisy neurons count less
Parameters, 96 neurons4,656 covariance entries against 1,056 for 10 factorsCounted here; why the factor model helps
PPCA noise floormean of the discarded eigenvaluesNotes, section 6 (Tipping and Bishop 1999)
Recording61 units (18 single, 43 multi), 784 trials, 14 targets, 20 ms binsYu et al. 2009
Best dimensionality10 (range 8 to 12 over 14 targets)Yu et al. 2009, leave-one-neuron-out, reduced GPFA
Best two-stage kernelabout 40 ms for PPCA and FAYu et al. 2009
GP kernelK(t, t) = 1, nugget 10−3, one learned τ per factorYu et al. 2009
First five learned timescales54, 160, 293, 173, 111 msYu et al. 2009
Convergence ratio1.31 ± 0.13 then 1.17 ± 0.14Yu et al. 2009; above 1 read as convergence in planning

The terms

TermPlain meaning
Latent variableA number the model invents to explain the data; never recorded directly
Prior and mappingHow likely each hidden value is, and how it produces the data
Recognition distributionThe posterior over the hidden value, given the data
Marginal likelihoodThe data’s probability with the hidden value integrated out
Loadings, uniquenessesHow each factor pushes each neuron; each neuron’s private noise variance
EMAlternate the E-step (guess hidden values) and the M-step (improve parameters)
PPCAFactor analysis with equal noise on every neuron; closed form
Two-stage methodSmooth each neuron in time, then reduce dimension
Gaussian processA rule for random curves; nearby times move together
GPFAFactor analysis with a Gaussian-process prior over time on every factor
Neural trajectoryThe path of the hidden state through one trial

The core as code

Two functions carry the lesson. The first is factor analysis by EM. The second is the GPFA posterior mean for a trial whose loadings and timescales are already fitted. Both follow the equations of Chapters 5 and 8. We ran the first on one simulated set (150 samples, 4 neurons, one factor, seed 1): it returned loadings of size 0.94, 0.67, 0.47, 0.52 (the overall sign is arbitrary) against the true 1.0, 0.8, 0.6, 0.5. With 150 samples the estimates can miss by up to about 0.15.

pythonimport numpy as np

def fa_em(X, d, iters=100, seed=0):
    """Factor analysis by EM. X is N x n (samples by neurons)."""
    N, n = X.shape
    X = X - X.mean(0)
    rng = np.random.default_rng(seed)
    A = 0.1 * rng.standard_normal((n, d))            # n x d loadings
    psi = X.var(0)                                   # n private variances
    for _ in range(iters):
        Lam = np.linalg.inv(A.T @ (A / psi[:, None]) + np.eye(d))   # d x d, posterior covariance
        U = (X / psi) @ A @ Lam                                      # N x d, posterior means
        A = (X.T @ U) @ np.linalg.inv(U.T @ U + N * Lam)             # M-step: loadings
        psi = np.mean((X - U @ A.T) ** 2, 0) + np.einsum('ij,jk,ik->i', A, Lam, A)
    return A, psi

def se_kernel(T, tau, sn=1e-3):
    t = np.arange(T)
    return (1 - sn) * np.exp(-(t[:, None] - t[None, :]) ** 2 / (2 * tau ** 2)) + sn * np.eye(T)

def gpfa_posterior_mean(Y, C, d, R, taus):
    """Y: q x T square-rooted counts. C: q x p. d, R: length q. taus: length p, in bins."""
    q, T = Y.shape
    p = C.shape[1]
    b = (C.T @ ((Y - d[:, None]) / R[:, None])).reshape(-1)     # p*T evidence, index a*T + t
    G = C.T @ (C / R[:, None])                                  # p x p
    K = np.zeros((p * T, p * T))
    for a, tau in enumerate(taus):
        K[a * T:(a + 1) * T, a * T:(a + 1) * T] = se_kernel(T, tau)
    M = np.eye(p * T) + np.kron(G, np.eye(T)) @ K               # I + (G kron I) K
    return (K @ np.linalg.solve(M, b)).reshape(p, T)            # p x T hidden path

A checklist before you trust a latent model

  1. Transform the counts. Bin, then square-root, so the noise is about equal across rates.
  2. Hold out neurons. Choose the number of factors by leave-one-neuron-out error, not by a bend in the scree.
  3. Restart EM. Try several starting loadings and keep the highest likelihood; it is a local optimum.
  4. Look at the space, not the axis. Factors are known up to rotation; order them (for example by SVD) before you read them.
  5. Say what is linear. The map is linear in square-root counts; the true surface may be curved.
  6. Do not smooth by eye. A calmer curve is not a truer one; score it on held-out data.

Where this sits

GPFA gave a practical way to read single trials. It has no dynamics: it says the path is smooth, not what pushes it. The Kalman filter and RTS smoother add linear dynamics, and the Kalman decoder uses them to read intent in real time. Later models make both the dynamics and the map nonlinear. The sequential auto-encoder LFADS (Pandarinath et al., Nature Methods 2018, doi) and the flexible latent-dynamics model DFINE (Abbaspourazad et al., Nature Biomedical Engineering 2024, doi) are examples. The Neural Latents Benchmark (Pei et al., NeurIPS Datasets and Benchmarks 2021, arXiv:2109.04463) scores such models with one shared test, co-smoothing in bits per spike.

Keep going

Sources

All simulated trials, neurons and trajectories in the devices are our own toys, labelled as such. The square-root variance check in Chapter 7 is a quick simulation of ours.

The takeaway. A population of noisy neurons is often a few hidden numbers seen through a lens. Factor analysis finds the lens, a smooth prior reads the numbers trial by trial, and held-out prediction tells you whether you chose well.
You have recorded a new session. Which plan follows the lesson?

Now press Teach and explain, from memory, why a noisy neuron should count for less, how EM breaks the chicken-and-egg loop, and what GPFA adds to factor analysis. Then go back to the hero and move the neuron slider with fresh eyes.

Latent Variable Models for Neural Data
NEU 560 course site · Yu et al. 2009 · Back to the Kalman decoder