One trial of spikes is a noisy dice roll. LFADS learns the hidden dynamics behind it and returns smooth rates, so a decoder reads hand velocity at R² 0.90 where raw bins give 0.34.
Learn how a recurrent generator, an encoder and a variational objective turn a few hundred spike trains into clean single-trial neural trajectories.
Pick a method, then drag Neurons recorded down. Raw bins and smoothing do not care how many neurons you have. The generator fit does. Then we build, piece by piece, the spikes, the dynamics, the network, the objective and the tricks that make it train.
You need a feel for a probability and what a recurrent network is (a network that feeds its own state back in at every time step). We teach the rest from zero.
Based on Pandarinath et al., “Inferring single-trial neural population dynamics using sequential auto-encoders”, Nature Methods 2018, and the AutoLFADS follow-up. Part of the series on brain-computer interfaces. The hero uses a toy world with made-up neurons, labelled as such.
Toy world, not data: 30 made-up neurons whose rates are a hidden rotating pair of numbers (the dynamics) pushed through a log-linear readout, spikes drawn as Poisson counts in 10 ms bins. The “generator fit” knows the rotation and fits only the starting point by maximum a posteriori, which is the LFADS idea without the neural network. The real model is in Chapters 3 to 5; the paper’s numbers are in Chapter 7.
Chapter 0
See why a brain-computer interface cannot average its way out of noisy spikes, and what it needs instead
A man with paralysis sits in front of a screen. A small array of electrodes sits in the motor cortex, the strip of brain that plans and drives movement. He imagines moving a cursor to the right. Over the next half second, a few hundred neurons each fire a handful of times.
A spike is one brief electrical pulse from one neuron. A computer counts the spikes of each neuron in short time slices, here 10 ms long. Those counts are the only thing the interface ever sees. The earlier lesson on spikes to rates built that count table. This lesson starts where it ended.
The trouble is that spikes are sparse and random. A neuron that fires at 20 spikes per second fires on average 0.2 times in a 10 ms slice. Most slices hold zero spikes. A few hold one. The pattern you see in a single slice tells you almost nothing about what the neuron is “really” doing.
Neuroscientists have an old cure for noise: trial averaging. Ask the animal to do the same reach 50 times. Line the 50 trials up in time. Add the spike counts and divide by 50. The noise shrinks, because random errors cancel, and a smooth curve appears. This curve is the neuron’s firing rate: the number of spikes per second it tends to produce at that moment.
For a BCI user this cure fails, for two reasons. First, the interface must act on this trial, right now, not after 49 more. Second, no two trials are the same. The user may start the reach a little earlier, aim a little differently, or correct halfway. The average describes a trial that never happened.
So the real goal is harder. Given the spikes of one trial, estimate the smooth rates that produced that trial. This is called single-trial inference. Try the problem below. The device records one neuron on many trials. Each trial has its own hidden starting state, so each trial has its own true rate curve.
The blue ticks are the spikes of one neuron on trial 1, drawn from a hidden rate (dashed). The warm line is the average of the first few trials. Drag the number of trials averaged and watch whether the warm line moves toward the dashed one.
Toy world: one made-up neuron, 48 bins of 10 ms, and a hidden starting state that differs on every trial (the same toy world as the hero). The numbers are computed live from the spikes you see. Real motor cortex shows the same effect: the average of many trials hides trial-to-trial differences.
Drag the slider to 40. The warm curve becomes smooth, and it settles on the average rate. But the dashed line, the truth for trial 1, stays apart from it. More trials make the average better as a description of the typical trial and no better as a description of this trial. At one trial, the warm line is just the raw counts: jagged and useless.
Here is the key observation behind this lesson. A few hundred neurons do not act alone. They are wired into a circuit, and the circuit has dynamics: a rule that says where the whole population will be a moment from now, given where it is now. If the rule is simple, the true rate of every neuron on a trial is fixed by only a few numbers: where the population started, and what pushed it along.
That turns denoising into a search. You do not need to guess 200 neurons times 100 time bins of rate, which is 20,000 numbers, from a few thousand spikes. You need to find a handful of starting numbers that make a rule-following trajectory explain all the spikes. Fewer unknowns means less noise gets through.
LFADS stands for latent factor analysis via dynamical systems. Latent means hidden: the factors are never recorded, only inferred. Dynamical system means a rule for how a state changes with time. The method, from Pandarinath and colleagues (2018), builds a neural network that learns the rule and then, for each new trial, infers the hidden starting state and any hidden pushes. The output is a smooth rate for every neuron on that trial.
It is a heavy tool, and the paper reports a heavy payoff. On a monkey reaching task, hand velocity decoded from LFADS rates reached R² of 0.90 (R² is the share of the variation that the decoder explains, 1 is perfect). Decoded from raw binned counts it reached 0.34. We rebuild that gap in Chapter 7.
Each chapter below builds one part of the model. We use a small toy world with a known answer so every claim can be measured. The real model and the paper’s numbers appear alongside, and we say which is which.
Chapter 1
Learn the one noise model LFADS uses, and the score it gives to any guess about the hidden rates
Imagine standing under a tree in light rain. Over one second you feel drops land on your hand. You count 3. Over the next second you count 6. Then 2. The rain did not change. Each second is a fresh random draw from the same steady shower.
A neuron behaves much like that rain. At a given moment it has a rate: say 40 spikes per second, on average. Whether a spike lands in a given 10 ms slice is a matter of chance. The rate sets the odds. It does not set the outcome.
Statisticians have a name for counts of random, independent events at a steady average rate: the Poisson distribution. It has one number, the expected count, written λ (lambda). For a bin of width Δt seconds and a rate r spikes per second, λ = r · Δt. At 40 spikes per second in a 10 ms bin, λ = 40 × 0.01 = 0.4.
The chance that the bin holds exactly k spikes is a short formula. Here it is, with every symbol named.
λ = 0.4: P(0) = e−0.4 = 0.670, P(1) = 0.4 × 0.670 = 0.268, P(2) = 0.08 × 0.670 = 0.054
Read the worked line. With λ = 0.4, two bins out of three are empty. About one in four holds a single spike. Only one in twenty holds two. This is why a raw spike train looks like sparse static, even when the rate behind it is strong.
The Poisson has a famous fingerprint: its variance (how widely the counts scatter around their mean) equals its mean. Double the rate and the scatter grows in step. Real neurons are somewhat different, but LFADS still uses the Poisson as its observation model, and the paper states that choice directly. Play with it below.
Top: 24 repeats of one neuron at a constant rate, one row each. Bottom: bars show how many bins held 0, 1, 2 or more spikes; the line is the formula. Drag the rate, choose a bin width, then press Roll again.
Illustrative: spikes are drawn by the page’s own random generator, 24 rows by one second of bins. The line is the exact Poisson formula, not a fit.
Set the rate to 40 and the bin to 10 ms: two thirds of the bins are empty. Widen the bin to 50 ms and the empty share drops to about 14%. A wider bin collects more spikes, so the count tells you more about the rate. But a wide bin also blurs when things happen. LFADS keeps bins narrow and gets its power from somewhere else.
Now flip the question. You see counts, and you want to judge a guess about the rate. The tool is the likelihood: the probability of the counts you actually saw, if the guess were true. A guess that makes the observed counts probable is a good guess.
Take one bin that held y = 3 spikes. Guess A says λ = 0.4. Guess B says λ = 2.5. Using the formula: P(3 | 0.4) = 0.4³ × e−0.4 / 6 = 0.064 × 0.670 / 6 = 0.0071. P(3 | 2.5) = 15.6 × 0.0821 / 6 = 0.214. Guess B is 30 times more likely. In practice we add up logarithms over all neurons and bins, so the score is a sum, not a product of thousands of tiny numbers.
Keep this formula in mind. It is the whole of LFADS’s data term. Every other part of the model exists to propose good rates λ for this score to judge. The paper writes the model’s rates as exp of a linear readout, so λ is always positive: a neuron cannot fire a negative number of spikes.
Chapter 2
Find the few hidden numbers that steer hundreds of neurons, and watch a simple rule turn them into rates
Think of a marching band on a field. Two hundred people, each with a position, yet you do not need 400 coordinates to say what the band is doing. The drum major says “turn left”, and every player follows a fixed pattern. A few numbers describe the whole show.
Motor cortex may work like that. Two hundred recorded neurons look independent when you stare at one. But they share inputs, they excite and inhibit each other, and they are all driven by the same plan. So their rates move together. Compare it to the marching band: a few hidden numbers set the band’s pattern, and each player is a fixed mixture of those numbers.
Call the hidden numbers the latent state (latent means hidden). In LFADS the readout of the state is called the factors, a short list of numbers at each moment. The Maze model in the paper uses 40 factors to explain 202 neurons. A toy world can use just two.
Each neuron reads the factors through a weight vector, a list of numbers that says how strongly this neuron cares about each factor. The neuron adds a baseline, then turns the sum positive with an exponential. The result is the rate, which LFADS then feeds to the Poisson of Chapter 1.
If you know PCA (see the PCA lesson and neural manifolds), this is its cousin. PCA finds a few directions that capture most variation. Here the same idea runs the other way: the model starts from the few directions and builds the neurons from them.
Factors alone give you a snapshot. A dynamical system adds the rule that moves the state from one time bin to the next. Write the state as x(t). The rule says x(t+1) depends on x(t), and possibly on a push from outside.
The simplest interesting rule is a rotation. Each bin the state turns by a small angle ω around the origin. Shenoy, Sahani and Churchland (2013) review the case for this dynamical view of motor cortex, and Churchland and colleagues (2012) reported rotation-like patterns in population activity during reaching. Rotation is a toy here, not a claim that every reach is a circle.
x(t+1) = R(ω) x(t), ω = 0.26 rad per bin
Work one neuron by hand. Let its weights be w = (1, 0), its baseline give 0.3 spikes per bin, and the start be x(0) = (1, 0). At t = 0 the dot product is 1, so the expected count is 0.3 × e1 = 0.82, which is 82 spikes per second. After 6 bins the state has turned 6 × 0.26 = 1.56 radians, nearly a quarter turn. Now x = (0.011, 1.000), the dot product is 0.011, and the rate falls to 0.30 per bin, 30 per second. A neuron with weights (0, 1) does the opposite: it sits at baseline first and peaks at bin 6.
That is already enough to see the main trick. Change only the start of the state, and every neuron’s whole rate curve changes in a coordinated way. Try it.
Left: the hidden plane. The purple dot is the start and the curve is the path the rule makes. Right: the rates of four neurons over one trial. Drag the start angle and radius.
Toy world: 30 made-up neurons with weights spread around the circle, a rotation of 0.26 rad per 10 ms bin, and baselines of 12 to 40 spikes per second. Nothing here is recorded data. The neuron numbers shown are the first four of the thirty.
Slide the angle and the four curves shift in time: the peaks move earlier or later. Slide the radius and the peaks grow taller or shallower. Two numbers set 30 neurons times 48 bins, or 1,440 rates. That is the compression LFADS wants: instead of estimating 1,440 free rates from sparse spikes, estimate two.
Look at what the rule does not allow. Once the start is fixed, the future is fixed. A population that follows a pure rule has no freedom left. Its whole trial is a function of its initial condition, the state at the first time bin. That is why LFADS spends so much effort inferring one number list per trial: the initial condition, called g0 in the paper.
Real motor cortex is not a pure rotation. The rule must be learned, it is nonlinear, and sometimes something outside the population pushes it, such as a sudden change in what the animal sees. Chapter 3 swaps the rotation for a recurrent neural network that can learn almost any rule. Chapter 5 adds the pushes.
One more thing to keep in mind is what failure looks like. If the chosen rule is wrong, then no start state fits the spikes well. The fit will leave structure in the leftover error. In the toy world, that would mean the generator-fit line missing a bump that the spikes clearly show. Watching for that mismatch is how you learn the model is too simple.
Chapter 3
Replace the hand-made rotation with a recurrent network that learns the rule, and follow every tensor from start state to spikes
In Chapter 2 we wrote the rule by hand: turn by 0.26 radians each bin. Real brains do not publish their rules. We need a machine that can learn almost any rule from examples. A recurrent neural network (RNN) is such a machine. It keeps a list of numbers called its hidden state, and at every time bin it computes a new hidden state from the old one.
Think of a spreadsheet row that you re-compute once per tick. Each tick, every cell is overwritten with a formula of the old row. Learn the formulas and you learn the rule. That is all an RNN is, and it is why the same weights can run for 50 bins or for 500: you just keep ticking.
Plain RNNs forget quickly. The LFADS paper uses a GRU (gated recurrent unit; see the RNN and GRU lesson for the full build). A GRU adds two gates, numbers between 0 and 1 that act like dimmer switches. The update gate decides how much of the old state to keep. The reset gate decides how much of the old state to look at when proposing the new one. With the gates open or closed, a GRU can hold a value for a long time or overwrite it at once.
You do not need the gate equations to follow LFADS. You need the contract: a GRU takes the old state (and an optional input) and returns a new state of the same length. In the paper’s Maze model the generator is a GRU with 100 units.
The generator in the paper is built from three small steps. The state gt goes through a GRU to become gt+1. A linear readout turns gt into the factors ft. A second readout, plus an exponential, turns the factors into rates rt. Poisson draws then give the spikes. The tensor shapes in the Maze model are in this table.
| Name | What it is | Shape in the Maze model | Where it comes from |
|---|---|---|---|
| g0 | Initial condition of the generator | 100 numbers | Sampled from the encoder’s guess (Chapter 4) |
| gt | Generator hidden state at bin t | 100 numbers per bin | GRU: gt = GRU(gt−1, ut) |
| ft | Factors | 40 numbers per bin | Linear readout of gt |
| rt | Firing rates | 202 numbers per bin | exp(linear readout of ft) |
| yt | Spike counts (the data) | 202 integers per bin | Poisson draws around rt |
The numbers 100, 40 and 202 are from the paper’s Methods (Table 1 and Sections 1.3 to 1.5). Over a trial of T bins the whole output is a table of 202 × T rates. Notice the funnel: 100 numbers of state squeeze to 40 factors, then widen to 202 neurons. The narrow waist is the low-dimensional bottleneck of Chapter 2, now learned.
Now see it run. The device below uses a tiny network with 6 units, built by hand from rotating pairs and a saturating tanh. It is not trained and not a GRU, but it has the same shape of computation: start state in, ticks, factors, rates, spikes. Choose a start and step through.
Each column is one time bin. Bars show the hidden state (6 numbers, purple), the factors (2 numbers, purple), the rates (8 neurons, warm) and one random spike draw (blue ticks). Pick a start, then press Step or Play.
Toy network, not trained: gt+1 = tanh(1.15 · R gt) where R is three fixed rotation blocks (0.10, 0.16 and 0.23 rad), factors are two fixed mixtures of the 6 units, and 8 neurons read the factors through the exp readout of Chapter 2. The real generator has 100 units, 40 factors and 202 neurons and is a trained GRU.
Run the three starts. All three share the same weights, and all three produce different rate curves. The rule is stored in the weights. The trial is stored in g0. Training will change the weights once, over thousands of trials. Inference, which we meet in Chapter 4, picks one g0 per trial.
A network with enough units can memorise anything, including noise. The paper guards against that with plain tools. It applies dropout (randomly switching off some connections during training) and an L2 penalty on the generator’s recurrent weights, which pushes the weights toward small values and keeps the rule simple. It also ramps both the L2 and a KL penalty (Chapter 4) up from zero over the first 2,000 training steps, so the network first learns to fit, then learns to be tidy.
What does failure look like here? Symptom: the model reconstructs the training spikes beautifully and does badly on trials it has not seen. Measurement: the likelihood score on held-out trials falls while the score on training trials keeps rising. Fix: more dropout, a stronger L2 penalty, fewer units, or more data. The paper holds out one fifth of trials, a 4 to 1 train and validation split, to watch for exactly this.
Chapter 4
Guess the starting state from the spikes, and pay a price for every bit of detail the guess carries
So far the generator runs forward from a starting state g0. But nobody hands us g0. We have spikes and must work backward. Think of a detective at a scene: the footprints (spikes) are noisy, and the detective must reconstruct where the suspect started (g0). A good detective also admits uncertainty: “probably here, give or take a metre.”
An autoencoder is a pair of networks. The encoder squeezes the data into a short code. The decoder expands the code back into the data. Training makes the output match the input. A variational autoencoder (VAE; see the VAE lesson) changes one thing: the encoder does not output one code but a distribution over codes, a mean and a spread. The decoder receives a random sample from that distribution.
LFADS is a sequential VAE. The decoder is the generator of Chapter 3. The code is g0. The encoder reads the whole trial of spike counts with a bidirectional RNN: one RNN runs forward in time, one runs backward, and their final states are combined. A linear readout of that combined state gives the mean and variance of a Gaussian guess for g0.
Training needs gradients, but a random draw has no gradient. The reparameterization trick moves the randomness aside. Draw a fixed-noise number ε from a standard bell curve, then compute g0 = μ + σε. Now μ and σ are ordinary numbers that gradients can flow through, and ε is just an input. With μ = 0.8, σ = 0.3 and ε = −0.5, the sample is 0.8 + 0.3 × (−0.5) = 0.65.
What should the VAE maximise? Two things, pulling against each other. The paper writes the bound as L = Lx − LKL.
The prior is what we believe about g0 before seeing any spikes. In the paper it is a Gaussian centred at zero with a fixed spread parameter, κ = 0.1. The KL divergence measures how far the encoder’s Gaussian is from the prior, in nats (a unit of information). For two one-dimensional bell curves it has a short form.
KL( N(μ, σ²) ‖ N(0, 1) ) = ½ ( σ² + μ² − 1 − ln σ² )
Work it with μ = 0.8 and σ = 0.3: ½ (0.09 + 0.64 − 1 + 2.408) = ½ (2.138) = 1.07 nats. Now an encoder that ignores the data and outputs the prior itself, μ = 0 and σ = 1: ½ (1 + 0 − 1 − 0) = 0. Zero cost, zero information. Information about the trial always costs KL.
That is the trade. A sharp, confident guess far from zero fits the spikes better, so Lx rises, but KL rises too. A vague guess near zero costs nothing but explains little. The best guess balances them. This balance is the main defence against noise: the encoder is not allowed to chase every random fluctuation unless it pays for it.
Training a real encoder takes hours. We can still see the trade-off in the toy world, because there the best guess for the starting state can be found directly. The device finds the most probable start given the spikes and the prior (a MAP estimate, with a Gaussian spread from the curvature of the score). A trained encoder learns to produce this same kind of answer in one pass. The weight on the KL term is a knob you can turn.
The plane shows the prior (dashed circle), the encoder’s guess for the start (warm ellipse), and the hidden truth (plain dot). The bars show the score pieces. Drag the KL weight and the number of time bins the encoder sees.
Toy world: 30 made-up neurons, a rotation rule, a prior of variance 1 on the start (the paper’s prior is the same kind of Gaussian with κ = 0.1), and a Gaussian guess found by Newton’s method instead of by a trained encoder. Lx is evaluated at the guess’s centre, without the constant log y! term. The KL weight is on a log scale; 1.0 is the plain bound.
Slide the KL weight toward 0.01. The guess leaps to wherever the spikes alone push it, the ellipse sits at the truth plus noise, and Lx is as high as it gets. Slide toward 100. The guess collapses toward the origin, KL falls toward zero, and the fit gets worse. Cut the number of time bins to 6 and you see the other pull: less evidence makes a broader ellipse, so the prior matters more.
Symptom: every trial in the output looks the same, a smooth average-like trace. Measurement: the KL term sits near zero, which means the encoder is returning the prior and the generator is running without any trial-specific information. Fix: do not let the KL penalty bite too early. The paper ramps the KL and L2 penalties up linearly, reaching their maximum at 2,000 training steps, so the network learns to use g0 before it is taxed for it. The limit is the same as turning the KL weight too high in the device above.
Chapter 5
Let the model explain a surprise by adding a push, and see how it finds when the push arrived
A monkey reaches for a target. Halfway there, the cursor on the screen suddenly jumps sideways. The animal must correct. In the motor cortex, the population changes course mid-trial. A pure rule of the kind we built in Chapters 2 and 3 cannot explain that: once g0 is set, the whole future is fixed.
The cortex is not a closed box. It receives signals from other brain areas and from the senses. Those signals push its state. LFADS needs a way to say “something outside nudged the population here.” The paper calls these nudges inferred inputs: hidden signals, one list of numbers per time bin, that the model must infer from the spikes alone. Nobody records them. Nobody tells the model when they happen.
Where does the model get these inputs? From a second network. In the paper, a controller RNN sits beside the generator. At each time bin it receives two things: the encoder’s summary of the trial, and the generator’s own factors from the previous bin, ft−1. It compares what the generator is doing with what the encoder says happened, and it outputs the mean and spread of a Gaussian for the input ut. The generator then takes ut as an extra input to its GRU.
Why feed in the previous factors? Because the controller must know what the generator already predicts, to say what is missing. It acts like a navigator who sees the planned route and the actual position, and shouts a correction only when they differ.
Like g0, each ut has a prior and a KL cost. The paper uses an AR(1) prior, in which each input is expected to be close to the previous one. The KL term in the bound of Chapter 4 now sums over g0 and over every ut. Pushes are not free. A push has to buy enough extra likelihood to pay its KL price.
What did the paper find? In a cursor-jump task on Monkey J, perturbations occurred on 25% of trials. A model with 4 inferred inputs was fitted. The authors embedded the inferred inputs with t-SNE, a tool that lays out high-dimensional points on a plane, and report that they cluster by perturbation type: unperturbed, left or right. No ground truth for the inputs exists in real data, so that is evidence of structure, not a proof of what each input means.
In the toy world we know the truth, so we can check the answer. Each trial has a hidden rotation and, optionally, a kick: a sudden sideways shove at one time bin. The device fits two models by maximising the penalised score on the spikes. The first lets only the start g0 be free. The second also lets a push appear at every bin, at a cost. It is a direct penalised fit of the real generative structure, a rotation with inputs, not a trained network.
Top: the first hidden coordinate over the trial. The dashed line is the truth, the warm line is the fit. Bottom: the size of the inferred input at each bin (pink); the marker shows when the kick really landed. Choose the model, then drag the kick size.
Toy world: the rotation of Chapter 2, 30 neurons, a kick that lands at a random bin between 14 and 32. The fit minimises the negative log-posterior with a Gaussian penalty on the start (variance 1) and on each input (variance 0.1), using Newton’s method on 96 numbers; the paper uses an AR(1) prior and a trained controller. The error is the root-mean-square distance between the true and fitted hidden points.
Set the kick to 1.4 and switch between the two models. With only a starting state, the warm line follows the early rotation and then misses the shove. With inputs on, the line bends where the dashed line bends, and the pink bars cluster close to the true landing bin. Set the kick to 0 with inputs on: the bars shrink to small, scattered values, because there is nothing to explain and each push costs.
An input channel is a powerful tool, and any powerful tool can be misused. If the controller may push as hard as it likes at any bin, it can absorb the noise itself. Then the generator never has to learn a proper rule.
Symptom: the inferred inputs are large on every trial, even trials with no known perturbation, and the autonomous dynamics learned by the generator are weak. Measurement: compare the KL cost of the inputs on trials with and without a known event. If both are large, the controller is doing the generator’s job. Fix: limit the number of input dimensions (the paper used 4 for the cursor-jump task), keep the KL penalty on, and prefer a prior that expects slow inputs. Try it on the device: drag the kick to 0 and watch whether the bars stay small.
Chapter 6
Compare three ways to turn one trial of spikes into rates, and see why no amount of smoothing matches a model that knows the rule
After training, LFADS is used like a filter. You hand it one trial of spike counts, and it hands back one table of smooth rates. The path inside is the path from Chapters 3 to 5: the encoder guesses g0 (and the controller guesses the inputs), the generator runs forward, and the readout produces rates. The paper takes the posterior mean: it samples many times from the encoder’s guess, runs the generator for each, and averages the rates. The paper’s example uses 512 samples.
How good is that answer? We should compare it with the simplest alternatives. There are three on the table. The first does nothing: use the raw counts. The second blurs the counts in time. The third uses a model of the dynamics.
Divide each count by the bin width. A bin with 1 spike in 10 ms becomes a rate of 100 spikes per second. A bin with 0 spikes becomes 0. The estimate is unbiased on average, yet any single bin is wildly wrong. The expected count is 0.3, but the bin can only hold 0 or 1 or 2, so every estimate is off by at least 30 spikes per second.
Replace each bin by a weighted average of its neighbours. The usual weights follow a Gaussian bell curve whose width is σ bins. Take σ = 2 bins. The centre bin gets weight 0.20, its neighbours 0.18, then 0.12, 0.07, and so on out to about 7 bins. A lone spike in a 10 ms bin, which a raw estimate calls 100 spikes per second, becomes a gentle bump with a peak of 20 spikes per second spread over about 14 bins.
Smoothing trades two kinds of error. A narrow window leaves most of the noise: variance, error that changes from trial to trial. A wide window removes the noise but also flattens real peaks: bias, error that is the same every time. You pick the width where the two balance, but the best balance is still a compromise. Smoothing treats every neuron separately and does not know that the neurons are linked.
A generator model uses everyone’s spikes at once. All 30 neurons vote on a handful of hidden numbers, and the rule makes the rates in every bin follow from them. The noise in each neuron averages out across neurons, with no loss of timing. The cost is an assumption: that the rule is roughly right.
Use the device to see all three. The top panel shows one neuron on one trial. The bottom panel shows the root-mean-square error of smoothing as a function of width, averaged over 40 simulated trials, with the generator fit’s error as a flat line.
Top: spikes (blue ticks), the true rate (dashed) and the estimate (warm) for one neuron. Bottom: error against smoothing width; the yellow line is where the smoothing width sits. Choose a method, then move the width.
Toy world: 30 made-up neurons, bins of 10 ms, a rotation rule with a fitted start. The error is the root-mean-square gap between the estimated and true rate, over all neurons and bins of the shown trial; the bottom curve averages 40 other simulated trials. The generator fit here knows the rule (a stand-in for a trained LFADS), so it is the best case for the method.
Start with the smoothed method at 0.5 bins: it is almost the raw trace, error high. Drag the width up. The error falls, reaches a floor near 3 to 4 bins, and then rises again as peaks flatten. That U-shape is the bias and variance trade. Now switch to the generator fit. The error line sits below the bottom of the U, and this is not luck: it uses information that smoothing throws away, namely what the other neurons are doing.
Real data have no known true rates, so the paper scores a different thing: how well the model predicts spikes of neurons it was not shown. LFADS factors predicted held-out neurons with a better single-trial likelihood than GPFA (a method that smooths a few latent traces with Gaussian processes) for all population sizes tested (p < 10−8). This is a fair test, because neither method saw those neurons’ spikes during fitting. The paper does not give a single headline number for it in the text we checked, so we do not print one.
Failure mode, first the symptom: the denoised rate has a bump that the spikes do not support. Measurement: compare the model’s likelihood on held-out neurons against smoothing; if the model loses, the rule is wrong. Fix: allow inputs (Chapter 5), more factors, or a better-tuned model (Chapter 8). A model-based denoiser is only as good as its rule.
Chapter 7
Turn denoised rates into hand velocity, read the paper’s R² numbers, and see why a model can beat a method with eight times the neurons
A BCI exists to move something. The final step is a decoder: a function that reads neural activity and outputs a command, here the velocity of the hand. Denoising is only worth the trouble if the decoder gets better. Pandarinath and colleagues tested exactly that.
The score is R² (R squared): the share of the variation in the true hand velocity that the decoder explains. R² = 1 means a perfect copy. R² = 0 means the decoder is no better than always guessing the average. Work a tiny case. The true velocities are 1, 2 and 3. The decoder says 1.2, 1.9 and 3.3. The squared errors are 0.04, 0.01 and 0.09, which sum to 0.14. The squared distances of the true values from their mean, 2, are 1, 0 and 1, which sum to 2. R² = 1 − 0.14 / 2 = 0.93.
The data are from a monkey (Monkey J) doing a maze reaching task, with 202 neurons recorded from motor cortex (M1 and PMd) by two 96-channel arrays, and about 2,300 trials across 108 reach conditions. The paper decodes hand velocity from rates estimated four ways. These are the average R² values from the paper with the full 202-neuron population.
| Method | What it does | Average R² for hand velocity |
|---|---|---|
| Binning | Raw spike counts per bin | 0.34 |
| Smoothing | Blur each neuron in time | 0.66 |
| GPFA | Smooth a few shared hidden traces with Gaussian processes | 0.69 |
| LFADS | Sequential autoencoder with a generator | 0.90 |
Two more results from the same paper matter. In single sessions, LFADS improved mean R² over smoothing by 0.32 and over GPFA by 0.27 (both p < 10−8, Wilcoxon signed-rank test). These come from per-session summaries, so they do not subtract exactly from the table. And, in the most striking comparison, LFADS using only 25 neurons (for X velocity) or 50 neurons (for Y velocity) beat the other methods applied to all 202.
Read that last claim twice. A model with an eighth or a quarter of the neurons did better than a method that used every one. Why is it plausible? Because the model does not need each neuron to speak for itself. A small group of neurons is enough to pin down the starting state and the pushes. The rule then fills in what everyone else is doing.
We can test this idea in our toy world, where the answer is built in. The device decodes a hidden “hand velocity” that is a fixed mix of the two hidden coordinates. It uses a linear decoder trained on 50 trials and scored on 50 other trials. Raw bins, smoothed rates and the generator fit take turns as the input to the decoder. Switch to the toy view and drag the number of neurons.
Choose the paper’s bars or the toy world’s live curves. In the toy view the vertical line marks the number of neurons you choose, and the read-out gives the three R² values there.
Paper bars: Pandarinath et al. 2018, Maze dataset, average R² for hand velocity with 202 neurons (0.34, 0.66, 0.69, 0.90). Toy curves are live: 100 simulated trials of 30 made-up neurons, a ridge-regression decoder trained on 50 and tested on 50, hidden hand velocity equal to the two hidden coordinates. The generator fit knows the rule, so it flatters the method; the paper trained the network.
Drag the toy slider down. The raw and smoothed curves drop quickly as neurons are removed, because each neuron carried its own share of the information. The generator curve falls far more slowly: the rule lets a few neurons support a good estimate. The shape matches the paper’s claim. The numbers differ, and they should: the toy is a different world.
Electrodes die. Signal quality drifts. Neurons come and go from the recording. A decoder that needs every channel at its best is fragile. A model that infers a handful of hidden numbers from whatever channels remain is sturdier. That is the practical meaning of the paper’s result, and the reason the idea spread. It is also why the NoMAD work in Chapter 9 builds on LFADS-style dynamics to keep decoders stable over months.
Failure mode, in order. Symptom: R² on the training set is high and drops on new days. Measurement: compare R² on trials from the same day with trials from a later day. Fix: align the latent space across days (Chapter 9), or retrain with new data. A high R² on one day is not a promise about the next.
Chapter 8
Spot the trap where the network simply copies its input, and learn the two fixes that made LFADS reliable: coordinated dropout and AutoLFADS
A student is asked to “predict” the answers on a test. The answer sheet is taped to the desk. Of course the student gets 100%. Nobody calls it learning.
An autoencoder faces the same temptation. Its job is to rebuild the spikes it is given. If the network is large enough, the easy path is to pass each spike straight from the input to the output, with no dynamics in between. The paper’s follow-up work calls this identity overfitting: the model finds trivial solutions that pass individual spikes from input to output. It looks perfect and has learned nothing about the rule.
The usual defence against overfitting is a validation set: trials the network never trained on. If training loss falls and validation loss rises, you stop. But the copying trick works on any trial, new or old, because it copies whatever it is handed. The follow-up paper makes this point: a model that acts like an identity transformation performs similarly on training and validation data, so standard validation loss cannot detect it.
You can see why. The validation trials arrive with their spikes as input, and the model copies them to the output, and the score is computed against the same spikes. Training and validation look the same: both look wonderful.
Keshtkaran and Pandarinath (2019) proposed a fix with a simple core. Take the table of spike counts, neurons by time bins. At each training step, randomly hide some of the entries from the network: set them to zero in the input, and scale up the rest, as in ordinary dropout. Now comes the coordinated part. The score is computed using only entries the network did not see. For the entries it did see, the gradient is blocked.
The effect is neat. To score well on hidden entries, the network cannot copy; it has nothing to copy for those cells. It must predict them from the visible ones. The only way to do that is to learn the structure that links neurons and time bins: the dynamics. In the synthetic test in the paper, a coordinated-dropout rate of 0.3 meant 70% of the data served as input and the remaining 30% as the target for the likelihood.
Try it. The device compares two models. The copier returns its input wherever it has one. The dynamics model is the generator fit of earlier chapters, fitted only on the entries it is allowed to see. Then both are scored on the entries they saw and on the entries they did not.
The grid is one trial: 30 neurons by 48 bins. Blue cells are shown to the model. Grey cells are hidden. The bars give the average Poisson log-likelihood per cell (closer to 0 is better) on the cells shown and on the cells hidden. Drag the share hidden.
Toy world: 30 made-up neurons, a rotation rule, 10 ms bins. The copier predicts a rate equal to the count it was shown (with a floor of 0.02 per bin) and the floor where the cell is hidden. The dynamics model is the maximum-a-posteriori start fit on shown cells only. Scores are the full Poisson log-likelihood per cell, averaged over the trial.
The copier wins on the cells it can see. Its score there is the best it can be. That is the number a plain training loss would report, and it hides the problem. On hidden cells the copier collapses. The dynamics model scores about the same on both groups. Coordinated dropout puts the loss where the copier loses. Move the slider and press the button a few times: the order of the bars does not change.
LFADS has many knobs: the scale of the KL penalty, the scale of the L2 penalty, the dropout rate, the coordinated dropout rate and the learning rate. A bad setting wastes a day of training. Early users tuned by hand, an art that took months. Keshtkaran and colleagues (Nature Methods 2022) automated it with population-based training (PBT).
Picture a garden of 40 plants, each grown with slightly different settings. Every so often, the weakest plants are replaced by copies of the strongest, and the copies get small random changes to their settings. Over many rounds the whole garden drifts toward settings that work. In AutoLFADS, each generation is 50 epochs, and the maze runs used 40 workers. Training stops when the best validation score improves by less than 0.05% over four generations.
What did it buy? On small subsets of the maze data (20%, 10% and 5% of trials), AutoLFADS beat manual tuning at all three sizes (p < 0.05). On a different dataset, area 2 of somatosensory cortex, its rates gave a better likelihood than a GLM for 110 of 121 neurons. The authors did not cross-validate the GLMs on purpose, so read this as a ceiling comparison, not a fair held-out test. The paper reports the R² numbers for the maze in a figure, so we do not print them.
Failure mode, in order. Symptom: a model trains fine but the rates look jagged, like the spikes themselves. Measurement: score the model on hidden entries, not on shown ones, and check whether the score matches the shown-entry score. Fix: raise the coordinated dropout rate, strengthen the KL and L2 penalties, or let AutoLFADS search for you.
Chapter 9
Meet the catch that keeps the original model offline, and follow the work that grew from it: stitched days, aligned dynamics and transformers
A user moves a cursor with thought. The screen refreshes every 20 ms or so. At each refresh the interface must decide where the cursor goes now, using spikes that have arrived so far. It cannot wait for the user to finish the reach and then look back.
That requirement is called causality: the output at time t may depend only on inputs up to time t. LFADS, as published, is not causal. Its encoder is a bidirectional RNN. One half runs backward from the end of the trial to the start. The estimate of g0, and with it every rate at every bin, depends on the whole trial, including the future.
For science this is a gift. When you want the best picture of what a population did in an experiment, using the future makes the picture sharper. For a closed-loop BCI it is a problem. The original model is an offline analysis tool. Making it run in real time is a separate engineering task, and we do not claim the paper solved it.
We can measure the cost in the toy world. The device fits the starting state using only the bins seen so far (the causal choice), then uses the rule to read off the hidden state at the current bin. It compares that with a fit that sees all 48 bins. The curve shows the average error over 40 simulated trials, bin by bin.
Warm: the error when only bins up to the marker are used. Plain line: the error when the whole trial is used. Drag the marker to a time in the trial and see how long the causal fit takes to catch up.
Toy world: 30 made-up neurons, the rotation rule of Chapter 2, 10 ms bins, 40 trials. Error is the root-mean-square distance between the true hidden point at the chosen bin and its estimate (the fit from the bins seen, run forward by the rule). The whole-trial line reads out its estimate at the same bin. The rule is known to the fit, so real systems will do worse.
Look at the first 10 bins. The causal fit has few spikes, so its guess about the start is poor and its error is high. As bins arrive, the error falls. The whole-trial fit has the full evidence at every bin, so its line is flat and low. The gap is the price of acting in real time. Notice that the causal curve does not need all 48 bins: it is usually close to the whole-trial line well before the end. The right place to cut depends on the task and on how many neurons you have.
Implanted arrays record from slightly different neurons each day. A decoder trained on Monday may fail on Tuesday. The paper offers an answer it calls stitching. In their example (Monkey P), they trained one LFADS model on 44 recording sessions spread over 162 days. Each session got its own readout from factors to neurons, since the neurons differ. But all sessions shared the same generator, the same dynamics. The shared part learns from every day.
The reported gains: stitched models, with a single shared decoder, improved mean R² by 0.22 over single-session LFADS (p < 10−8), and improved prediction of reaction time by 0.15 in correlation (p < 10−7). The idea is plain: the rule is a property of the brain circuit, and the circuit is the same on Tuesday.
These are pointers, not a survey. We have read the abstracts and cited the venues. We have not tested their claims here, and we leave the details to those papers.
Symptom: the decoded cursor drifts after a few days, though nothing in the code changed. Measurement: compare the latent trajectories of the new day with the old day on the same behaviour. If their shapes match but sit in different places, the circuit is steady and only the recording moved. Fix: realign the latent space (the NoMAD idea), or fit a fresh readout for each day on top of a shared generator (the stitching idea). A decoder built on unstable electrodes may be fixed faster than the brain is.
Chapter 10
Keep the numbers, the checklist, the core code and the sources, and know where to go next
LFADS treats spike counts as Poisson draws from rates. The rates come from a low-dimensional state that follows a learned rule. An encoder infers the starting state, a controller infers outside pushes, and a generator runs the rule forward. A variational bound trains everything at once. The result is a smooth rate for each neuron on each single trial, and in the paper a decoder reading those rates reached R² 0.90 where raw bins gave 0.34.
| Quantity | Value | Where it comes from |
|---|---|---|
| Maze neurons, arrays | 202 neurons, two 96-channel arrays | Pandarinath 2018, Monkey J, M1 and PMd |
| Maze trials | about 2,300 trials, 108 reach conditions | Pandarinath 2018 (2,296 used in analysis) |
| Bin width | 5 ms | Pandarinath 2018, Methods Table 1 |
| Generator, factors | 100 GRU units, 40 factors | Pandarinath 2018, Maze model |
| Prior on g0 | Gaussian, mean 0, κ = 0.1 | Pandarinath 2018, Methods 1.3 |
| KL and L2 ramp | linear, full at 2,000 steps | Pandarinath 2018, Methods 1.10 |
| Optimiser | Adam, rate 0.01, clip 200, 4:1 split | Pandarinath 2018, Methods 1.10 |
| Hand-velocity R² | 0.34 bins, 0.66 smoothing, 0.69 GPFA, 0.90 LFADS | Pandarinath 2018, Maze, 202 neurons |
| Fewer neurons | 25 (X) or 50 (Y) beat the others on 202 | Pandarinath 2018 |
| Stitching | 44 sessions, 162 days; R² +0.22 | Pandarinath 2018, Monkey P |
| Cursor jump | 4 inferred inputs, 25% perturbed trials | Pandarinath 2018, Monkey J |
| Coordinated dropout | rate 0.3: 70% input, 30% scored | Keshtkaran 2022, synthetic test |
| AutoLFADS search | 40 workers, 50 epochs per generation | Keshtkaran 2022, maze runs |
A compact PyTorch sketch of the model and its loss. It is our own condensed version for teaching, not the authors’ code, and we ran it for three training steps on random data to check that it runs. It leaves out the AR(1) prior on the inputs (it uses a plain Gaussian), the L2 penalty, dropout and the penalty ramp. See lfads-torch for a full implementation.
pythonimport torch, torch.nn as nn, torch.nn.functional as F def kl_gauss(mu, logvar, prior_var): class="cm"># KL( N(mu, exp(logvar)) || N(0, prior_var) ), summed over the last axis return 0.5 * ((logvar.exp() + mu**2) / prior_var - 1 - logvar + torch.log(torch.tensor(prior_var))).sum(-1) class LFADS(nn.Module): def __init__(s, n=202, e=64, c=64, g=100, f=40, u=4): super().__init__() s.g, s.u = g, u s.enc = nn.GRU(n, e, batch_first=True, bidirectional=True) s.g0_head = nn.Linear(2 * e, 2 * g) class="cm"># mean and log-variance of g0 s.ctrl = nn.GRUCell(2 * e + f, c) class="cm"># sees encoder output and last factors s.u_head = nn.Linear(c, 2 * u) class="cm"># mean and log-variance of u_t s.gen = nn.GRUCell(u, g) s.fac = nn.Linear(g, f) s.rate = nn.Linear(f, n) class="cm"># log-rate readout def forward(s, y): class="cm"># y: (B, T, n) spike counts B, T, _ = y.shape h, hn = s.enc(y) class="cm"># h: (B, T, 2e), hn: (2, B, e) mu, lv = s.g0_head(torch.cat([hn[0], hn[1]], -1)).chunk(2, -1) g = mu + torch.randn_like(mu) * (0.5 * lv).exp() class="cm"># reparameterization kl = kl_gauss(mu, lv, 0.1) class="cm"># prior variance kappa = 0.1 ct = h.new_zeros(B, s.ctrl.hidden_size) f = s.fac(g) logr = [] for t in range(T): ct = s.ctrl(torch.cat([h[:, t], f], -1), ct) um, ul = s.u_head(ct).chunk(2, -1) ut = um + torch.randn_like(um) * (0.5 * ul).exp() kl = kl + kl_gauss(um, ul, 0.1) class="cm"># simple prior; the paper uses AR(1) g = s.gen(ut, g) f = s.fac(g) logr.append(s.rate(f)) return torch.stack(logr, 1), kl class="cm"># (B, T, n), (B,) def loss_fn(model, y, kl_scale, keep=None): logr, kl = model(y if keep is None else y * keep / keep.mean()) nll = F.poisson_nll_loss(logr, y, log_input=True, reduction="none") if keep is not None: nll = nll * (1 - keep) class="cm"># score only the hidden cells return nll.sum((1, 2)).mean() + kl_scale * kl.mean()
We read the Pandarinath and Keshtkaran papers through their open-access texts. The other papers are cited from their abstracts and listings. Every toy device is labelled as a toy, and none of its numbers come from the papers.
Now press Teach and draw the model from memory: the spikes, the encoder, the guess for g0, the generator, the factors and the rates, and the two parts of the bound. Then go back to the hero and drag the neuron count again.