One recorded trial gives you thousands of spikes and one hidden signal behind them. Factor analysis and Gaussian-process smoothing recover that signal from a single trial.
Learn how a few hidden numbers can explain a whole population of noisy neurons, and how to infer them trial by trial.
Press each method in turn, then drag the neuron count down to 4. Watch the orange estimate chase the blue hidden signal. Then we build, piece by piece, the model that does it: the hidden variable, the loadings, the fitting loop and the smooth prior.
You need a feel for averages and variance (how spread out numbers are) and a little matrix notation. We build the rest from zero.
Follows Princeton NEU 560: Statistical Modeling and Analysis of Neural Data (Spring 2024, Jonathan Pillow), lecture 15 (slides) and lecture 19 (notes), and Yu et al. 2009 on GPFA. Every idea is re-taught here in our own words and devices.
A toy trial, not recorded data: one hidden signal, 50 bins of 20 ms, Poisson spike counts, square-root transformed as in Yu et al. 2009. Loadings are set from the simulation, as if learned on earlier trials. Smoothing uses a 40 ms Gaussian; GPFA picks its own timescale by likelihood. Errors are root-mean-square, in units of the signal’s own spread.
Chapter 0
See why a single trial of spikes looks like noise, and why a few hidden numbers can explain it
Picture a person with an electrode array in motor cortex (the strip of brain that plans movement). A cursor appears on a screen. They think about moving it to the right. The array records 96 channels, and you watch the spikes arrive.
A spike is a brief electrical pulse that one neuron fires. A spike train is the list of times at which it fired. Each neuron fires a little more or a little less as the plan unfolds. But any single neuron is erratic. It fires twice in one 20 ms window, then not at all in the next two.
Now you want to know what the brain is doing right now, on this one attempt. You do not have a hundred repeats to average. You have one trial: one try at the movement, from start to finish. This lesson is about getting a clean answer from a single trial.
Here is the idea in plain words. The neurons are not 96 separate stories. They share a few causes. The plan to reach right is one cause. The urgency of the reach is another. Each cause pushes many neurons at once, some up and some down.
Think of a stadium crowd doing a wave. Each person stands and sits on their own, and each person is a little late or a little early. But the wave itself is one smooth thing. If you watch ten people you can guess where the wave is. If you watch a thousand, you can see it clearly.
The wave is a latent variable. “Latent” means hidden: you never record it directly. You record only the people, and you infer the wave from them. In neural data, the people are neurons and the wave is the shared signal that drives them. Pillow’s lecture states the general idea this way: latent variable models explain observed data with some hidden structure.
The lecture draws the picture with simulated data. A raster shows 50 neurons over time as rows of dots. It looks like static. Then a second picture shows a few smooth curves, the inferred latent variables, that the dots were secretly following. The raster is the data. The curves are the explanation.
The old answer is to repeat the trial many times and average. If the signal is the same each time, noise cancels and the signal stays. That works, and it built the field. But it has a hole.
The signal is not the same each time. On one attempt the person reaches fast. On the next, slowly. The hidden bump arrives a little early or a little late. Averaging lines the trials up in time and then smears them together. The average is lower and wider than any real trial.
Try it. The device below draws six trials of the same hidden signal, each shifted in time by a small random amount. The thick line is their average. Drag the timing spread up and watch the average lose its peak.
Thin blue lines: the same hidden signal on six trials, each shifted in time. Thick orange line: the trial average. Drag the spread of timing between trials.
A toy signal, not recorded data. The six shifts are fixed multiples of the spread (about −1.2, −0.6, −0.2, 0.3, 0.8 and 1.3 times it). Each trial’s signal is scaled to the same overall spread before averaging.
The cure is to read the hidden signal on each trial separately. Then each trial keeps its own speed and shape. Averaging after you have read it is still available, if you want it. But now you can also ask new questions: why was this reach slow, and what was the brain doing when it was?
The lecture lists three other places where a hidden variable explains messy data. In electrophysiology, spike sorting infers which neuron fired each pulse that the electrode saw. In calcium imaging, you infer where the cells are and when they fired from a blurry movie. In EEG, source separation infers the brain sources that mixed into each scalp sensor. Our own lessons on independent component analysis and spike sorting live in the same family.
What is new in this lesson is the neural population on one trial. We need three tools, in this order. First, the plain idea of a latent variable model and the two jobs it must do. Second, factor analysis, the simplest model that finds shared structure and ignores private noise. Third, a smooth prior over time, which gives Gaussian-process factor analysis, or GPFA.
The hero at the top is the finished instrument. Chapter 8 explains its third button, and Chapter 7 its second.
Chapter 1
Build any latent variable model from two parts, then use it to run a guess backwards from data to cause
You find a number on a scrap of paper: 0.8. A friend tells you it came from one of two machines. Machine A prints numbers near −1.5. Machine B prints numbers near +1.5. Both are a bit noisy. Which machine printed your 0.8?
You answer without thinking: probably B, because 0.8 is closer to 1.5. You have just done latent variable inference. The machine is the hidden variable. The number is the data. You ran the story backwards, from the number you saw to the machine you did not.
Pillow’s lecture says every latent variable model is defined by two pieces. The first is a prior over the latent variable: how likely each hidden value is before you see any data. For the two machines, it is how often each one is used. The second is a mapping: given the hidden value, how likely is each data value? The lecture calls it the conditional probability of the data given the latent. For machine B, it is a bell curve around +1.5.
Multiply them and you get the joint distribution, the probability of a hidden value and a data value together. This is how the model would generate data: draw a hidden value from the prior, then draw data from the mapping.
The lecture names two things we want to do with such a model. The first is inference, also called recognition: given one data point, work out the hidden value. That is the scrap of paper. The second is fitting: given a whole dataset, work out the settings of the prior and the mapping, such as where each machine’s bell curve sits. We deal with fitting in Chapter 5.
Recognition is Bayes’ rule. Take the joint at the number you saw, and divide by the total over every hidden value, so the answers add up to 1. The result is the recognition distribution, also called the posterior.
Try it on the two machines. The lecture works this exact case, a binary mixture of Gaussians, on the board. Move the observed number and watch the posterior split.
Purple bump: machine A, weighted by how often it runs. Orange bump: machine B. The teal line is the number you observed. The bar under it splits the posterior.
A toy: machine A centred at −1.5 and machine B at +1.5, both with spread 1. The numbers are illustrative; the rule is Bayes’ rule, as in the lecture’s binary mixture example.
At 0.8 with equal use, the bell of B is worth 0.312 there and the bell of A only 0.028. Divide: 0.312 / (0.312 + 0.028) = 0.917. The model says B with 92% probability. Slide the prior toward A and the answer moves. A rarely used machine needs strong evidence.
The lecture then makes a point that organises the rest of this series. Change the prior and the mapping, and you get a different famous model. A discrete prior with Gaussian mapping is a mixture of Gaussians, which is clustering. A Gaussian prior with a linear Gaussian mapping is factor analysis, and PCA is a special case. A Gaussian prior that moves in time with linear dynamics is a linear dynamical system, the engine inside the Kalman filter. A discrete prior that hops between states is a hidden Markov model.
| Model | Prior over the hidden variable | Mapping to data |
|---|---|---|
| Mixture of Gaussians | discrete | Gaussian |
| Factor analysis | Gaussian | linear, Gaussian |
| Linear dynamical system | linear Gaussian dynamics | linear, Gaussian |
| Hidden Markov model | discrete transitions | any |
We have lessons on the cousins: EM for Gaussian mixtures, the hidden Markov model, and the Kalman filter. This lesson takes the second row and then adds time to it, which lands between rows two and three.
Chapter 2
See how a hidden variable turns a simple sum into an integral, and how a Gaussian model escapes it
In Chapter 1 you scored two machines. You multiplied, you added two numbers, you divided. Easy. Now the machine is not A or B. It is a dial, and it can sit at any value between −5 and 5. How many “machines” are there? Infinitely many.
That one change is the source of most of the pain in this field. Let us see exactly where it bites.
Recall the generalised linear model (GLM), and the spiking version in the point-process GLM lesson. A GLM predicts a neuron’s spikes y from an input x that you know: a stimulus, or the cursor velocity. To fit it, you write the likelihood, the probability the model gives to the spikes you recorded, and you make it large. For a Poisson neuron, the log-likelihood is a plain sum over time bins. Nothing is hidden. Pillow’s slide puts it this way: fit using the sum of yt log λt minus λt, where each λt is a fixed function of x and the weights.
The lecture then says: a latent variable model is a GLM where you do not know the input. The “stimulus” is the hidden signal, and you must find it from the spikes themselves. That is the chicken-and-egg problem on the slide. To learn the weights you need the input. To find the input you need the weights.
The way out is to ask: what probability does the model give to the data, whatever the hidden value was? You add up the joint over every possible hidden value. For two machines that was two terms. For a continuous hidden variable, the sum becomes an integral: the area under a curve. This quantity is the marginal likelihood, or evidence.
The device draws the two curves that get multiplied for one neuron. The neuron’s count is x = A z + noise. You set the loading A (how strongly the hidden value drives the neuron), the noise variance ψ, and the count you observed.
Grey line: the prior over the hidden value z. Blue: how well each z explains the observed count. Orange fill: their product. Its area is the evidence. Drag the sliders.
A toy with one neuron and one hidden value, prior N(0, 1). Each curve is stretched to fill the plot; the evidence is computed from the true heights with 400 slices, then checked against the closed form N(x; 0, A² + ψ).
Take A = 1, ψ = 1 and x = 2. The slices add to 0.1038. Now look at the closed form from the lecture: the evidence is a Gaussian centred at 0 with variance A² + ψ = 2. At x = 2 that is exp(−4/4) / √(4π) = 0.3679 / 3.545 = 0.1038. The same number, with no slicing.
This closed form is a gift of the Gaussian. Gaussians multiply into Gaussians, and a Gaussian integrates to a Gaussian. Change one ingredient and the gift is gone. If the mapping is a Poisson spike model, with rate eA z, the integral has no formula. You must approximate it. The lecture puts the problem bluntly: for a model whose latent runs through time, the likelihood needs a high-dimensional integral, one dimension for every time bin.
For that reason the lecture lists three ways to fit a latent variable model:
This lesson uses the second, because for Gaussian models both steps are exact. Our EM for Gaussian mixtures lesson shows the same loop on clusters. The variational auto-encoder uses the third idea with neural networks.
Chapter 3
Build the simplest hidden-variable model of a population, and read it as a model of the covariance
Two neurons sit side by side in motor cortex. When one fires more than usual in a 20 ms bin, the other tends to fire more too. A third neuron does the opposite. Is that a coincidence? Over thousands of bins it is not. Something shared pushes them together, and something private makes each one wobble on its own.
Factor analysis puts those two kinds of variation into one model. It is the oldest and plainest latent variable model for continuous data, and it is the engine inside GPFA.
We have n neurons. In one time bin they give a vector x of n numbers (spike counts, or their square roots, which we explain in Chapter 7). We say a small number d of hidden values z drive all of them. Here is the generative story in the notation of Pillow’s lecture notes.
Shapes help. For 96 neurons and 10 factors, z is 10 numbers, A is a 96 by 10 table, and x is 96 numbers. A tall, thin matrix turns a short list into a long list. That is what “a few causes, many effects” means in code. The parameters are θ = {A, Ψ}.
Take the model and ask what the data look like when you forget z. The lecture notes show the answer by the standard Gaussian rules: the data are Gaussian with mean 0 and covariance A AT + Ψ.
So factor analysis is a model of the covariance (the table of how every pair of neurons varies together). It writes that table as low rank plus diagonal. Move the sliders to see the table and the split for each neuron.
Left: the covariance of eight neurons. Purple cells co-vary up, blue cells co-vary in opposite directions. Right: each neuron’s variance, purple for shared, grey for private. Drag the two sliders.
A toy with one factor: loadings 0.9, −0.6, 0.7, 0.3, −0.8, 0.5, 1.0, −0.4 times the shared strength, and private variances 0.4 to 0.7 times the private-noise slider. All numbers are illustrative.
Read neuron 3 at the default settings. Its loading is 0.7, so the factor gives it shared variance 0.7² = 0.49. Its private variance is 0.5. Total: 0.99. About half of what neuron 3 does is shared. Turn the private noise to 3 and the shared share drops to 25%, while the off-diagonal cells do not change at all. Noise lives only on the diagonal.
Count parameters. A full covariance for 96 neurons has 96 × 97 / 2 = 4,656 free numbers. Factor analysis with 10 factors has 96 × 10 loadings plus 96 uniquenesses, which is 1,056 numbers, and a bit fewer after the freedom described below. A recording of a few hundred trials cannot pin down 4,656 numbers well. It can pin down about a thousand much better. The factor model is a deliberate simplification, a bet that the world has low-dimensional shared structure.
Chapter 4
Find out why a noisy neuron should count for less, and why PCA cannot know that
Two thermometers hang in the same room. Both track the room’s temperature. One is good. The other is faulty and jumps around by ten degrees. You want the room’s temperature. Which reading do you trust?
You trust the good one, of course. Now suppose a program must find “the main direction of variation” in the pair of readings. If it simply picks the direction with the most variance, it will point at the faulty thermometer. That is the one that varies the most.
You know PCA: find the directions in which the data spread the most. Pillow’s notes say both PCA and factor analysis are models of the covariance, and they differ in one choice. PCA says the covariance is low rank: a few directions carry it all. Factor analysis says it is low rank plus a diagonal: a few shared directions, plus an independent amount of noise for each neuron.
That diagonal is a “fudge factor” in the notes’ words. It lets each neuron have its own noise. PCA has no such slot, so it must treat all variance as signal.
The notes build the smallest case. One hidden value drives two neurons. Both load equally, with loading 1, so A = [1, 1]T. But the noise on neuron 1 has ten times the standard deviation of the noise on neuron 2: Ψ = diag(100, 1). The data covariance is then A AT + Ψ, which is a 2 by 2 table with 101 and 2 on the diagonal and 1 off it.
PCA looks at that table and sees that axis 1 has variance 101 and axis 2 has variance 2. Its top direction points almost straight along neuron 1, about 0.6 degrees off the axis. But the truth is that the hidden value pushes both neurons equally, so the shared direction is the diagonal [1, 1]. Factor analysis recovers that, because it knows neuron 1’s big variance is private noise.
Drag the noise on neuron 1 and watch the two arrows. The cloud is 250 draws from the model.
Each dot is one time bin: neuron 1 on the horizontal axis, neuron 2 on the vertical. The orange line is PCA’s top direction. The purple line is the factor loading. Drag the noise of neuron 1.
The notes’ example: loadings [1, 1], noise std of neuron 2 fixed at 1. At std 10 the covariance is [[101, 1], [1, 2]]. The cloud is simulated; the angles and weights are exact.
At noise 10 the PCA arrow lies 0.6 degrees from neuron 1’s axis, and the factor line stays at 45 degrees. At noise 1 both neurons are equally noisy, and the two lines agree. The disagreement grows exactly as one neuron gets less trustworthy.
The device also shows the weights used to guess the hidden value from one data point. Chapter 5 derives them. For this example, the weight on neuron 1 is 0.005 and the weight on neuron 2 is 0.498. The model almost ignores the noisy neuron. If you see x = [3, 2], the estimate is 0.005 × 3 + 0.498 × 2 = 1.01. A projection on PCA’s axis would give nearly 3. The notes say it plainly: project onto [0, 1] instead of [1, 0].
The notes end this comparison with two facts worth keeping.
For neurons, the second fact is the one that matters. Neurons fire at different rates, so their variances differ by large factors. Each neuron is its own axis with its own units. A method that survives rescaling an axis is a better fit than one that lets the loudest neuron decide. The first fact is a smaller cost: “neuron 5” is a real thing, so the axes are not arbitrary.
Chapter 5
Derive the recognition distribution, then alternate between guessing hidden values and improving the loadings
You have a factor model, and a data vector arrives. First question: what was the hidden value? Second question, harder: you do not yet have the loadings A or the noise Ψ. How do you get them?
Each question seems to need the answer to the other. If you knew A and Ψ, you could read the hidden values. If you knew the hidden values, you could fit A and Ψ by plain regression. Expectation maximisation breaks the circle by taking turns.
Fix A and Ψ. Chapter 1 told us the recognition distribution is the mapping times the prior, rescaled. Both factors are Gaussian, so the product is Gaussian. The notes find it by completing the square: write the exponent as a quadratic in z, collect the z2 terms and the z terms, and read off the mean and the covariance.
The exponent of the product is −½ (A z − x)T Ψ−1 (A z − x) − ½ zT z. Expand it. The z2 terms are zT (AT Ψ−1 A + I) z. The terms linear in z are −2 zT AT Ψ−1 x. A Gaussian in z with covariance Λ and mean μ has exponent −½ zT Λ−1 z + zT Λ−1 μ. Match the pieces.
Run the numbers from the last chapter. A = [1, 1], Ψ = diag(100, 1). Then AT Ψ−1 A = 1/100 + 1 = 1.01, so Λ = 1 / 2.01 = 0.4975. For x = [3, 2], μ = 0.4975 × (3/100 + 2/1) = 0.4975 × 2.03 = 1.010. The mean is 1.01 and the spread is 0.4975 (a standard deviation of 0.70). The noisy neuron barely moved it.
Now treat the guesses as if they were data. The notes use a Gaussian guess for every sample, N(μi, Λ). The new loadings are a ridge-regression solution from guessed hidden values to the data. The new noise is the leftover error, plus a correction for our remaining doubt about z.
A tiny case by hand. Keep the A and Ψ above and take two samples, x1 = [3, 2] and x2 = [−1, −2]. The E-step gives μ1 = 1.010 and μ2 = 0.4975 × (−0.01 − 2) = −1.000. Sum of μ² is 1.020 + 1.000 = 2.020, and NΛ = 0.995, so the denominator is 3.015. The numerator for neuron 1 is 3 × 1.010 + (−1)(−1.000) = 4.030, so its new loading is 1.337. For neuron 2 it is 4.020 / 3.015 = 1.333. The new uniquenesses are 2.307 for neuron 1 and 1.320 for neuron 2.
Noise estimates moved toward the data: neuron 1 went from 100 down to 2.3, because two samples show it was never that noisy. With so little data, treat this as a drill and not as a result.
Repeat the two steps. The lecture shows why each round cannot make the fit worse: both steps raise a lower bound called the negative free energy, which touches the true log-likelihood after the E-step. Step the loop on a population of four neurons below.
Left: log-likelihood per sample after each EM round. Right: estimated loading (orange bar) against the loading that made the data (blue tick). Press Step, then Run 20.
150 simulated samples of four neurons from one factor. True loadings 1.0, 0.8, −0.6, 0.5 and uniquenesses 0.3, 0.2, 0.4, 1.0 are labelled; the start is loadings 0.2, 0.1, 0.15, 0.05. The sign of a factor is arbitrary, so the bars are flipped to match the true signs.
Watch the first rounds. The log-likelihood jumps, then flattens: by round 10 it sits within 0.01 of its round-40 value. Each round costs one pass over the data, and no round ever lowers the fit.
Chapter 6
Get a closed-form factor model by fixing the noise to be equal, then pick the number of factors from held-out data
EM took many rounds in Chapter 5. Sometimes you want an answer in one line of linear algebra. And whichever method you use, one question has been hanging since Chapter 3: how many factors? One? Ten? Forty?
This chapter answers both. Make the noise equal on every neuron and the fit has a formula. Then test each number of factors on data the model has never seen.
Probabilistic PCA (PPCA) is factor analysis with one restriction: Ψ = σ² I. Every neuron has the same private noise. It is less flexible than FA, since it cannot down-weight a noisy neuron. In return, Tipping and Bishop showed in 1999 that it needs no iteration. The notes give the recipe. Take the sample covariance and its eigenvectors bi and eigenvalues λi, sorted largest first. Keep d of them.
That last step is the whole difference from PCA. PCA reports λi as the strength of a direction. PPCA says part of that is noise that every direction carries, and subtracts it. If the discarded eigenvalues are tiny, the two methods agree.
The device below simulates 20 neurons driven by three real factors with different private noise. Its top three eigenvalues are 28.05, 14.81 and 9.04. The next is 1.13, then 0.99 and so on down to 0.21. With d = 3, the 17 discarded eigenvalues average 0.62, so σ² = 0.62 and the first loading has length √(28.05 − 0.62) = 5.24.
Follow the data flow once. The input is a matrix of 200 training samples by 20 neurons. You centre it and form the 20 by 20 sample covariance. An eigen-decomposition returns 20 eigenvalues and a 20 by 20 matrix of eigenvectors. For a chosen d, the output is a 20 by d loading matrix A and one number, σ². Nothing is iterated, so the whole fit costs one eigen-decomposition, a few milliseconds here.
Compare this with Chapter 5, where EM needed dozens of passes over the data. The price of the shortcut is the equal-noise assumption. If one neuron is much noisier than the rest, as in Chapter 4, PPCA will blame the wrong direction, and you should return to factor analysis.
The scree of eigenvalues shows a cliff after three, but real data rarely cooperate so cleanly. Yu and colleagues use a more honest test, and so should you. Split the trials into folds. Fit the model on some of them. On held-out trials, hide one neuron, infer the hidden value from the other neurons, and predict the hidden neuron’s activity. Add up the squared errors across neurons. Repeat for every candidate d. That is leave-one-neuron-out cross-validation.
The prediction step is Chapter 5 again, now without neuron j: μ = (A−jT A−j + σ² I)−1 A−jT x−j, and the guess for neuron j is Aj μ. Because neuron j is never used to find μ, a model cannot score well by memorising noise.
Top: the eigenvalues of the covariance. Orange bars are kept, grey bars are discarded; the dashed line is the noise floor PPCA sets. Bottom: held-out prediction error for each number of factors. Drag the slider.
A simulation, not recorded data: 20 neurons, three true factors, private noise 0.3 to 1.0, 200 training and 200 test samples. Error is mean squared leave-one-neuron-out prediction on the test samples. Yu et al. 2009 ran the same test with 4-fold cross-validation on 61 real units.
Slide to d = 1. The noise floor jumps to 1.81, because two real factors are hiding in the “discarded” part and are being counted as noise. The held-out error is 1.96. Slide to d = 3. The floor drops to 0.62, and the error to 0.76, the lowest. Slide on to d = 8. The error creeps back up to 0.79: extra factors fit noise from the training set that the test set does not share.
Yu et al. ran this kind of test on 61 real motor and premotor units. For the reach target they showed, the prediction error of their reduced GPFA model bottomed out at 10 dimensions. Across the 14 reach targets, the best dimension ranged from 8 to 12, with 10 the most common.
Chapter 7
Turn spikes into numbers a Gaussian model can use, then find the smoothing width that the two-stage recipe hides
Factor analysis has so far treated each time bin as a separate sample. It would give exactly the same answer if you shuffled the 50 bins of a trial. But the brain is not shuffled. The hidden signal at 300 ms is close to the hidden signal at 320 ms. A model that ignores this wastes information.
Before we fix that, we need to make the data fit the model, and we need the old fix everybody tried first.
Factor analysis assumes Gaussian noise with the same variance on every sample. Spikes break that. First we bin the spike train: count the spikes in each 20 ms window. Yu and colleagues used 20 ms bins. A count is a small whole number, 0, 1, 2, and its noise is not Gaussian.
It is also not constant. A Poisson count has variance equal to its mean, so a busy neuron is noisier than a quiet one, and a neuron is noisier at the peak of its response than at the trough. That breaks the model’s promise of a fixed ψi per neuron.
The usual fix is the square-root transform: replace each count by its square root. A quick simulation shows why. Counts drawn at rate 0.3 have variance 0.30. At rate 10, they have variance 10.0. After the square root, the variance stays between 0.22 and 0.40 over rates from 0.3 to 10, and settles near one quarter at high rates. The transform makes the noise nearly the same for all firing rates, which is what Ψ assumes. Yu et al. square-rooted their 20 ms counts for this reason.
The square-rooted counts still jitter from bin to bin. The natural repair is to smooth each neuron’s train in time. A Gaussian kernel does that: each bin is replaced by a weighted average of its neighbours, with weights that follow a bell curve. The width of the bell (its standard deviation) sets the smoothing. Wide kernels average many bins and look calm. Narrow ones keep sharp features and stay noisy.
Then you run PCA, PPCA or FA on the smoothed data. Yu et al. call these two-stage methods. In our toy, step two is the posterior mean from Chapter 5, applied bin by bin: μt = (1 + s)−1 Σi Ai (zit − di) / ψi, where s = Σi Ai² / ψi and di is the neuron’s baseline.
Try it. The top panel shows one trial’s estimate at the chosen width. The bottom panel shows the error against width, averaged over 12 trials, with a dashed line for the method of the next chapter.
Top: the hidden signal (blue, dashed) and the two-stage estimate (orange) for one trial. Bottom: error against kernel width over 12 trials; the dashed line is what GPFA gets without any width. Drag the width.
A toy: 12 simulated neurons, 20 ms bins, 12 trials, square-root counts, loadings set from the simulation. Error is root-mean-square against the hidden signal, in its own units. The best width here is shaped by this toy’s signal; Yu et al. found about 40 ms best for PPCA and FA on their real data.
Look at the U. Symptom: with no smoothing, the estimate jitters and the error is 0.40 or more. With a very wide kernel, the estimate is a flat smear and the error is worse than none. Measurement: the curve has a clear minimum, so a width exists, but the data do not announce it. Someone has to scan for it with held-out data, as Yu et al. did.
Worse, the recipe makes three silent choices. The same width applies to every neuron, though some neurons carry fast features and some slow. The same width applies to every hidden dimension, though one factor may drift slowly while another oscillates. And the smoothing never sees the factor model, so the model cannot ask for more smoothing where it is unsure.
The fix is to put both stages in one model, so that the smoothness is a parameter learned together with the loadings. That is GPFA. If you know the RTS smoother from Kalman filtering, the spirit is similar: the estimate at each time borrows from its neighbours, and how much it borrows is set by a model, not by hand.
Chapter 8
Give each hidden factor a smooth random-curve prior, so that the model learns how smooth it should be
Draw a wobbly line on paper without lifting the pen. You can draw a lazy line that barely bends. You can draw a jittery line that changes direction every few millimetres. Either way, the line is continuous: where it is now tells you something about where it will be a moment later. How much it tells you depends on the line’s character.
A hidden neural signal behaves like that line. Gaussian-process factor analysis, from Yu, Cunningham, Santhanam, Ryu, Shenoy and Sahani (2009), turns that into a prior.
A Gaussian process (GP) is a rule for generating random curves. Instead of giving each time bin its own independent bell curve, it says: the values at all times together are one big Gaussian whose covariance depends on how far apart the times are. Our Gaussian processes lesson builds this in depth.
GPFA uses the squared-exponential kernel. The covariance between the hidden value at time t and at time t′ is
Work out one value. With τ = 4 bins (80 ms), two bins 3 apart have covariance e−9/32 = 0.755. Bins 8 apart have e−64/32 = 0.135. So a value says a lot about its near neighbours and almost nothing about values 160 ms away. The top of the device draws random curves from this prior at the τ you choose.
The observation model is the factor model you know, now with a baseline: yt = C xt + d + ε, with ε ~ N(0, R) and R diagonal (so each neuron has its own noise). Here yt holds the square-rooted counts of q neurons in bin t, xt is the p-dimensional hidden state, C is q by p, and d is each neuron’s baseline. The one change from factor analysis is the prior. Instead of drawing xt independently in each bin, each of the p rows of x gets its own GP, over all T bins together.
This is how smoothing and reduction share one model. The loadings C reduce the dimension. The kernel makes the hidden curves smooth. Both are fitted at once by EM, and the E-step is still exact because everything is Gaussian. The M-step updates C, d and R in closed form, as in factor analysis, while each τ is updated by gradient steps. As with Chapter 5, EM converges to a local optimum.
For one factor the posterior mean has a neat form. Gather each bin’s evidence into bt = Σi Ci (yit − di) / Ri, and let s = Σi Ci² / Ri. These are the same quantities as in Chapter 5. Then the whole curve is the vector K (I + s K)−1 b: one T by T linear solve. For T = 50 bins it is a 50 by 50 system, instant in a browser. The same solve gives the likelihood of each τ, which is how the hero picks its timescale.
Top: three random curves drawn from the prior at your τ. Bottom: the hidden signal (blue, dashed) and the GPFA estimate (orange) from the same spikes. Drag τ.
One simulated trial with 10 neurons (not recorded data), loadings set from the simulation. Likelihood is the marginal likelihood of the spikes under each τ; it is what the paper’s EM climbs. In Yu et al. the first five learned timescales were 54, 160, 293, 173 and 111 ms.
At τ = 20 ms the curves in the prior are nearly white noise, and the estimate follows every blip in the spikes. At τ = 400 ms the curves are nearly straight lines and the estimate cannot bend to follow the peak. Between them the likelihood has a peak. That is the value EM would learn, and it is also near the lowest error.
On real data the timescales differ by factor. Yu et al. report learned timescales of 54, 160, 293, 173 and 111 ms for their first five dimensions (with 15 fitted). A two-stage method has one width for all of them. GPFA gives every factor its own.
Chapter 9
Draw the path a whole population takes on each trial, and read what the paper found when it did
Imagine a monkey seated at a screen. A target lights up. The animal must wait for a go signal, then reach. Between the light and the reach there is a delay, and in that delay nothing moves. But motor cortex is busy. The population is getting ready.
Yu and colleagues recorded that delay. A 96-channel array sat over dorsal premotor and motor cortex of one rhesus macaque. After removing ten channels, they kept 61 units in one session (18 single neurons and 43 multi-unit signals). They had 784 trials of a delayed centre-out reach to 14 target positions. Their question: what does the population do, trial by trial, while the animal prepares?
With p = 3 hidden factors, the state of the population at one moment is a point in a 3-D box. A moment later the point has moved. Join the points over a trial and you get a neural trajectory: the path of the population through its own state space. Our neural manifolds lesson explains why such paths live on a low-dimensional surface.
GPFA gives one such path for every trial, from just that trial’s spikes. The model has already been fitted on many trials, so that C, d, R and the timescales are known. The trajectory for a new trial is then the posterior mean from Chapter 8, a single linear solve.
Chapter 3 warned that factors are only known up to a rotation. For drawing, the paper settles the choice with a singular value decomposition of the fitted C. That orthonormalises the loadings and orders the dimensions by how much data covariance each explains. The first axis of the plot is the most important one. That is why the paper fits 15 factors, orthonormalises, and keeps the top few for display or for prediction.
Now the device. It simulates three targets with five trials each. Every trial follows its target’s path, but with its own timing and size, so no two trials match. Choose the method and the neuron count. The faint thick lines are the true paths. Dots mark every 100 ms.
Each colour is one target, with five trials. Faint thick lines are the true paths; thin lines are the recovered paths of single trials. Pick a method, then drag the neuron count down.
A toy, not recorded data: 15 simulated trials, spikes from the sine-and-bump paths above, square-root counts, 40 ms kernel for the two-stage method, τ = 100 ms for GPFA, loadings known. Both methods are given the best linear alignment to the true axes, which no experimenter has. Distances are in the latent’s own units.
With 24 neurons both methods land closest to the true paths. With 8, the estimates wander, and the average distance to the truth grows for both. In this toy, GPFA stays a little closer on most draws. Do not over-read that: the gap is small, and both methods here get the help of an alignment to the truth. The paper’s test is harder and fairer: predicting a held-out neuron. On that test, the best two-stage method (FA with a 40 ms kernel) still had higher error than GPFA, and the authors note it remains to be seen how much that matters for behaviour.
On the real data, the authors report that single-trial trajectories converged over time during motor planning. In plain words: trials to the same target start with a scattered cloud of states and end the delay in a tighter cloud. They measured this with a ratio of the volumes of the covariance ellipsoids of the trial cloud, taken over the top 10 dimensions and reduced to a ratio per dimension (the 10th root), on trials with delays over 400 ms. Across the 14 targets, the ratio per dimension was 1.31 ± 0.13 from target onset to the go cue and 1.17 ± 0.14 from the go cue to movement onset, both above 1 (P < 0.001). The authors read a ratio above 1 as convergence.
They also show one odd trial. On one trial the animal was slow to move, with a reaction time of 844 ms, far longer than usual. Its trajectory looked very different from the others. Yet its activity around movement onset matched the other trials well. The paper presents this as an illustration that single-trial paths may relate to behaviour, and leaves subtler links for later work. A trial average could never have shown it.
The paper is frank about its assumptions. All of PCA, PPCA, FA and GPFA assume a linear link in square-root count space, so the true surface may be curved. GPFA specifies no dynamics: it says the path is smooth, not what pushes it along. The authors list extensions: coupling the kernels of different factors, kernels that change over time, nonlinear maps, and point-process spike models. The learned timescales depend on the starting values. Over four starts, the mean of the 15 learned timescales ranged from 125 to 155 ms, while the likelihoods and trajectories stayed very similar. And plotting three dimensions may hide structure in the rest. The next lessons in this series pick up exactly these gaps: dynamics (the Kalman and linear dynamical systems of the Kalman decoder) and nonlinear latents.
Chapter 10
Keep the numbers, the terms, the checklist and the code, and find where to go next
A latent variable model explains many noisy measurements by a few hidden numbers. You define it by a prior over the hidden variable and a mapping from hidden to data. Fitting needs a sum over all hidden values, so we alternate: guess the hidden values, then improve the parameters (EM). Factor analysis is the Gaussian case. It writes the covariance as low rank plus diagonal, which lets each neuron have its own noise. Probabilistic PCA fixes the noise equal and has a closed form. The number of factors comes from held-out prediction. Spike counts need binning and a square root first. The two-stage recipe smooths and then reduces, with a width chosen by hand. GPFA gives every factor a Gaussian-process prior, so smoothness is learned with the loadings, and it returns one path per trial.
| What | Number | Source and why it matters |
|---|---|---|
| Two-neuron example, covariance | [[101, 1], [1, 2]] | NEU 560 L19 notes; PCA points at the noisy neuron |
| Recognition weights, same example | 0.005 and 0.498; estimate 1.01 for x = [3, 2] | Derived from the notes’ formula; noisy neurons count less |
| Parameters, 96 neurons | 4,656 covariance entries against 1,056 for 10 factors | Counted here; why the factor model helps |
| PPCA noise floor | mean of the discarded eigenvalues | Notes, section 6 (Tipping and Bishop 1999) |
| Recording | 61 units (18 single, 43 multi), 784 trials, 14 targets, 20 ms bins | Yu et al. 2009 |
| Best dimensionality | 10 (range 8 to 12 over 14 targets) | Yu et al. 2009, leave-one-neuron-out, reduced GPFA |
| Best two-stage kernel | about 40 ms for PPCA and FA | Yu et al. 2009 |
| GP kernel | K(t, t) = 1, nugget 10−3, one learned τ per factor | Yu et al. 2009 |
| First five learned timescales | 54, 160, 293, 173, 111 ms | Yu et al. 2009 |
| Convergence ratio | 1.31 ± 0.13 then 1.17 ± 0.14 | Yu et al. 2009; above 1 read as convergence in planning |
| Term | Plain meaning |
|---|---|
| Latent variable | A number the model invents to explain the data; never recorded directly |
| Prior and mapping | How likely each hidden value is, and how it produces the data |
| Recognition distribution | The posterior over the hidden value, given the data |
| Marginal likelihood | The data’s probability with the hidden value integrated out |
| Loadings, uniquenesses | How each factor pushes each neuron; each neuron’s private noise variance |
| EM | Alternate the E-step (guess hidden values) and the M-step (improve parameters) |
| PPCA | Factor analysis with equal noise on every neuron; closed form |
| Two-stage method | Smooth each neuron in time, then reduce dimension |
| Gaussian process | A rule for random curves; nearby times move together |
| GPFA | Factor analysis with a Gaussian-process prior over time on every factor |
| Neural trajectory | The path of the hidden state through one trial |
Two functions carry the lesson. The first is factor analysis by EM. The second is the GPFA posterior mean for a trial whose loadings and timescales are already fitted. Both follow the equations of Chapters 5 and 8. We ran the first on one simulated set (150 samples, 4 neurons, one factor, seed 1): it returned loadings of size 0.94, 0.67, 0.47, 0.52 (the overall sign is arbitrary) against the true 1.0, 0.8, 0.6, 0.5. With 150 samples the estimates can miss by up to about 0.15.
pythonimport numpy as np
def fa_em(X, d, iters=100, seed=0):
"""Factor analysis by EM. X is N x n (samples by neurons)."""
N, n = X.shape
X = X - X.mean(0)
rng = np.random.default_rng(seed)
A = 0.1 * rng.standard_normal((n, d)) # n x d loadings
psi = X.var(0) # n private variances
for _ in range(iters):
Lam = np.linalg.inv(A.T @ (A / psi[:, None]) + np.eye(d)) # d x d, posterior covariance
U = (X / psi) @ A @ Lam # N x d, posterior means
A = (X.T @ U) @ np.linalg.inv(U.T @ U + N * Lam) # M-step: loadings
psi = np.mean((X - U @ A.T) ** 2, 0) + np.einsum('ij,jk,ik->i', A, Lam, A)
return A, psi
def se_kernel(T, tau, sn=1e-3):
t = np.arange(T)
return (1 - sn) * np.exp(-(t[:, None] - t[None, :]) ** 2 / (2 * tau ** 2)) + sn * np.eye(T)
def gpfa_posterior_mean(Y, C, d, R, taus):
"""Y: q x T square-rooted counts. C: q x p. d, R: length q. taus: length p, in bins."""
q, T = Y.shape
p = C.shape[1]
b = (C.T @ ((Y - d[:, None]) / R[:, None])).reshape(-1) # p*T evidence, index a*T + t
G = C.T @ (C / R[:, None]) # p x p
K = np.zeros((p * T, p * T))
for a, tau in enumerate(taus):
K[a * T:(a + 1) * T, a * T:(a + 1) * T] = se_kernel(T, tau)
M = np.eye(p * T) + np.kron(G, np.eye(T)) @ K # I + (G kron I) K
return (K @ np.linalg.solve(M, b)).reshape(p, T) # p x T hidden path
GPFA gave a practical way to read single trials. It has no dynamics: it says the path is smooth, not what pushes it. The Kalman filter and RTS smoother add linear dynamics, and the Kalman decoder uses them to read intent in real time. Later models make both the dynamics and the map nonlinear. The sequential auto-encoder LFADS (Pandarinath et al., Nature Methods 2018, doi) and the flexible latent-dynamics model DFINE (Abbaspourazad et al., Nature Biomedical Engineering 2024, doi) are examples. The Neural Latents Benchmark (Pei et al., NeurIPS Datasets and Benchmarks 2021, arXiv:2109.04463) scores such models with one shared test, co-smoothing in bits per spike.
All simulated trials, neurons and trajectories in the devices are our own toys, labelled as such. The square-root variance check in Chapter 7 is a quick simulation of ours.
Now press Teach and explain, from memory, why a noisy neuron should count for less, how EM breaks the chicken-and-egg loop, and what GPFA adds to factor analysis. Then go back to the hero and move the neuron slider with fresh eyes.