The Kalman Decoder

A cursor must glide, but spike counts jitter. The Kalman decoder fuses a physics prior with 70 ms of spikes and turns noisy counts into a smooth path.

Learn how a brain-computer interface turns the firing of a few dozen motor-cortex neurons into the position of a cursor, one 70 ms step at a time.

Switch between Each bin alone and Kalman, then drag the neuron count down. Watch which path stays near the blue truth. Then we build, piece by piece, the prior, the measurement model, the fit from data and the gain that does the fusing.

You need a feel for averages and noise and nothing else. We teach the state, the Gaussian and the matrices from zero.

Grounded in W. Wu, Y. Gao, E. Bienenstock, J. P. Donoghue and M. J. Black, “Bayesian population decoding of motor cortical activity using a Kalman filter”, Neural Computation 18, 80 to 118 (2006). The general filter is in our Kalman filter lesson; this lesson is about using it on spikes. Every idea is re-taught here in our own words and devices.

Decode a hand from spikes

24

Illustrative simulation, not recorded data: a smooth 12 s hand path, neurons tuned to position and velocity with Gaussian noise, 70 ms bins, A, W, H and Q fitted by least squares on 3.5 minutes of separate simulated training (the length of the paper’s pinball training set). “Each bin alone” is the best fit of the state to one bin’s spikes, with no memory. The error is the mean distance between the decoded and the true position.

Chapter 0

From Firing to Motion

Follow one 70 ms slice of brain activity from the electrode to the cursor, and see why a single slice is not enough

A woman with a spinal cord injury sits in front of a screen. A small array of electrodes rests on her motor cortex, the strip of brain that plans hand movements. She cannot move her arm. She imagines moving it to the right, and a cursor on the screen drifts to the right.

Nothing magic happens between the thought and the cursor. The electrodes pick up spikes: brief electrical pulses that a neuron fires. Software counts the spikes of each neuron. A program then guesses where the hand would be. That program is the decoder.

The data flow, with its shapes

Count spikes in short windows. A window is a bin, and this lesson uses bins of 70 ms, the size Wu and colleagues used in their pinball task. A neuron with 3 spikes in a 70 ms bin has a firing rate of about 43 spikes per second. We write the rates of all C neurons in one bin as a list of C numbers, zk ∈ RC. The paper had C = 42 units in the pinball task and C = 25 in the pursuit task.

The decoder must output the state of the hand: the few numbers that say where it is and where it is going. Our state has four numbers, xk = [px, py, vx, vy] ∈ R4: position in centimetres and velocity in centimetres per second. The paper used six, adding acceleration. So every 70 ms the decoder turns 42 numbers into 4 numbers (or 6).

That looks like one multiplication. If we knew a matrix G of shape 4 × C, we could compute x̂k = G zk in each bin and be done. Try it in the hero at the top: the pink path is exactly this idea, with the best possible G. It jitters. It also wanders off the true path when you remove neurons.

Why one bin is not enough

Spikes are random. A neuron that fires 20 spikes per second on average does not fire exactly 1.4 spikes in every 70 ms bin. It fires 0, 1, 2 or 3, by chance. That chance is noise. A decoder that trusts each bin fully copies the noise into the cursor.

The obvious fix is to average several bins. Longer windows hold more spikes, and the noise shrinks like one over the square root of the window length. Four times the window halves the noise.

But averaging has a price. The hand keeps moving while you wait. An average over the last 300 ms describes where the hand was 150 ms ago, on average. The cursor lags the hand. Noise falls as the window grows, and lag grows with it. Move the slider below and find the best compromise.

Find the best window

Pink: error from spike noise. Yellow: error from lag. White: the two combined. Drag the window and look for the lowest point of the white curve.

70 ms

Illustrative toy, not the paper’s data. Noise error = 4.2 cm/s × √(70 ms / window). Lag error = half the window times a typical acceleration of 30 cm/s². The two add as squares. Chapter 6 shows the paper’s own bin-width numbers.

With these toy numbers the best window is near 140 ms. Even there the error is not small. A plain average cannot be both quick and quiet. We need a second source of information that costs no lag.

The missing information: hands obey physics

A hand does not teleport. If it moves right at 10 cm/s now, it will still move right 70 ms from now. That fact costs nothing to use and carries no spike noise. It is a prior: what you believe before you look at the new spikes.

The Kalman decoder runs on this idea. At every bin it does two things. First it predicts: it moves the last estimate forward with a simple physical rule. Then it corrects: it nudges the prediction toward what the new spikes say, by an amount that depends on how much it trusts each source. The rest of this lesson builds each piece.

The road

  1. The prior: how a hand moves from one bin to the next, and how uncertainty grows.
  2. Spikes as measurements: what one neuron says, and what thirty say together.
  3. Predict and correct: the filter, with every number of one worked step.
  4. Fit it from data: A, W, H and Q by least squares in a few minutes.
  5. The steady-state gain: why the filter turns into a fixed rule after a second.
  6. Lag and bin width: the paper’s own numbers.
  7. When it drifts: nonstationary neurons, and recalibration.
  8. Against the field: population vector, linear filter, Kalman, and what came next.
  9. Field Guide: numbers, checklist, code, sources.
A decoder averages the last 8 bins instead of 1. What does it gain and what does it lose?

Chapter 1

The Prior: How a Hand Moves

Write the physics of a hand as one matrix and one noise term, and watch the uncertainty fan out as you look further ahead

Pause a video of someone reaching for a cup. Where will the hand be one frame later? You can guess well. It will be a little further along the line it is already travelling. You did not need new information. You used what a hand is.

The decoder needs that same guess in numbers. It starts from the state from Chapter 0: position and velocity, written xk for bin number k. The question becomes: given xk, what is xk+1, the state 70 ms later?

The rule, in plain words

Two facts do most of the work. First, position changes by velocity times time. At 10 cm/s, a hand moves 0.7 cm in 70 ms. Second, velocity mostly stays the same, but it fades a little in each bin as the hand slows toward a stop. We say velocity keeps about 94% of its value.

Real hands also surprise us. A target appears, and the hand speeds up. The rule cannot know that. So we add a random kick to each step and say how big the kicks usually are. The kick is the process noise. It is our honest way to say: “the physics is approximate.”

xk+1
The hand state in the next bin: position and velocity, 70 ms later.
A
A 4 × 4 matrix that holds the physics: “new position = old position + 0.07 × velocity” and “new velocity = 0.94 × old velocity”.
xk
The hand state now.
wk
The random kick: what the physics cannot predict. Drawn fresh in every bin.
W
The covariance of the kick: how large the kicks are, and which state numbers they push together. The notation N(0, W) means a bell-curve draw centred on zero.

A bell curve with a size: the Gaussian

A Gaussian is the familiar bell curve. It has a centre, the mean, and a width, the variance. The square root of the variance is the standard deviation, written sd; about two thirds of draws fall within one sd of the mean. For several numbers at once, the variance becomes a table called the covariance matrix. Its diagonal holds one variance per number. Its other cells say whether two numbers rise and fall together.

The Gaussian has one property that makes the whole lesson work. Push a Gaussian through a linear rule like A x and it stays Gaussian. Add an independent Gaussian kick and it still stays Gaussian. So the decoder can track its belief with just two objects: a mean and a covariance.

A worked step

Take one axis. The state is [p, v]. Start at p = 0 cm and v = 10 cm/s. Use Δt = 0.07 s, the 94% fade, and a kick of variance 4 (cm/s)2 on velocity. These are toy numbers close to what we fit later.

p1 = 0 + 0.07 × 10 = 0.7 cm    v1 = 0.94 × 10 = 9.4 cm/s    (the mean moves; no kick yet)

The spread after one step is just the kick: variance 4, so an sd of 2 cm/s. After a second step the old spread fades (4 × 0.942 = 3.53) and a new kick adds 4, so the variance is 7.53 and the sd is 2.7 cm/s. The spread grows, then levels off, because fading and kicking balance. The final level is 4 / (1 − 0.942) = 34.4, an sd of 5.9 cm/s.

This is the key picture. The prior is sharp for the next bin and vague for the distant future. The covariance after each step is Pk+1 = A Pk AT + W: the old uncertainty, pushed through the physics, plus a new kick. Play with the fan below.

Roll the prior forward

The hand starts at 0 cm with the velocity you set. Thin purple lines are possible futures from the prior alone. The band is one sd. Change the speed and the size of the kicks.

10 cm/s
×1.00

Toy numbers: one axis, 70 ms bins, velocity fades by 6% per bin, velocity kick variance 4 (cm/s)² times the knob, no kick on position. The forty lines are seeded draws; the band is computed exactly from P = A P Aᵀ + W.

What belongs in the state

Wu and colleagues did not stop at velocity. Their state held position, velocity and acceleration, six numbers in 2D. They tested the choice on the pinball data (their Table 3). A state with position only gave a mean squared error of 7.72 cm2. Adding velocity cut it to 6.31. Adding acceleration cut it to 5.87. Higher derivatives helped less each time: 5.72, 5.61, 5.60. The BIC score, which charges for extra parameters, was best for the version that also held the third derivative (jerk). The authors still used position, velocity and acceleration in most experiments, as a good trade between accuracy and model size.

The lesson: every extra state number adds parameters to fit and needs more training data. Stop when the gain stops paying for the cost.

Two ways the prior goes wrong

If W is too small, the decoder believes its physics too much. The cursor glides past a sudden turn, because the prior keeps saying “straight on”. If W is too large, the prior says almost nothing and the cursor jitters as in Chapter 0. Slide the kick knob and see the fan widen or tighten. No setting is right in the abstract. Chapter 4 fits W from recorded movement so that the kicks match what real hands do.

The decoder predicts one bin ahead, then ten bins ahead, from the physics alone. What happens to its uncertainty?

Chapter 2

Spikes as Noisy Measurements

Treat each neuron as a crude, noisy ruler for hand velocity, and see how thirty rulers agree on one answer

Listen to one neuron in motor cortex while a monkey moves its hand. The clicks come faster when the hand moves up and to the right. They slow down when the hand moves down and to the left. The neuron has a favourite direction. Scientists call it the preferred direction, and the way its rate rises and falls with direction is its tuning. Georgopoulos and colleagues described this pattern in 1986, and it is the root of every motor decoder.

A tuned neuron works like a ruler for velocity. If its rate is high, the hand probably moves toward the preferred direction. The ruler is poor, though. It is noisy, it reads only one direction, and its marks are not the same for every neuron.

The measurement equation

Wu and colleagues made the ruler precise with a straight-line rule. A neuron’s rate is a weighted sum of the hand state, plus noise. They plotted real rates against position and velocity (their Figure 2) and found that a line is “a crude but reasonable approximation”. For all C neurons together:

zk
The spike measurement in bin k: one firing-rate number per neuron, C numbers in all.
H
A C × 4 matrix. Row i holds neuron i’s weights: how much its rate rises per cm of position and per cm/s of velocity. It is the neuron’s tuning, written as numbers.
xk
The hand state, the same four numbers as in Chapter 1.
qk
Measurement noise: whatever the line does not explain. One value per neuron, redrawn in every bin.
Q
A C × C covariance matrix. Its diagonal says how noisy each neuron is. Its other cells say which neurons err together.

Read the equation backwards from how a decoder uses it. The brain picks a hand state. The state sets the average rates through H. Chance adds noise q. The decoder sees only z and must work out x. This is a generative model: it says how the data are made, and decoding runs it in reverse.

Two small repairs: centre and square-root

The paper applied two fixes before fitting. First, it subtracted each mean, so every neuron and every state number is centred on zero. This removes the baseline firing and means the line needs no intercept. Second, it took the square root of each firing rate.

Why the square root? Spike counts are close to a Poisson process, where the noise variance equals the mean. A busy neuron is noisier than a quiet one, which breaks our assumption of a fixed-size Gaussian error. The square root evens out the variance. The authors report that this step helps only slightly. It is a repair, not a requirement.

One ruler, then many

Take velocity along one axis. Let a neuron read z = h v + noise, with h = 0.2 per cm/s and a noise sd of 1.0. Suppose it reads z = 1.4. Invert the ruler: v = 1.4 / 0.2 = 7 cm/s. How sure can we be? Noise of 1.0 in z is noise of 1.0 / 0.2 = 5 cm/s in v. So one neuron says “7, give or take 5”. That is nearly useless.

Now ask thirty neurons. Each gives its own “v, give or take”. Multiply the beliefs: that is Bayes’ rule with independent evidence. The result is a Gaussian whose precision (one over variance) is the sum of the precisions. With neurons of equal quality the sd falls like 5 / √N. Four neurons give 2.5 cm/s. Twenty-five give 1.0. Drag the slider and watch the bumps narrow.

Add neurons, narrow the answer

Thin pink bumps: what each neuron alone says about velocity. Thick pink: what they say together. The blue line is the true velocity. Add neurons, then raise the noise.

3
1.00

Illustrative: true velocity 6 cm/s; thirty seeded neurons with weights from −0.25 to +0.25 per cm/s; independent Gaussian noise. Joint mean = Σ h z / σ² over Σ h² / σ², and joint sd = 1 / √(Σ h² / σ²). Neurons with a larger weight narrow the answer more.

Two details of the picture matter. A neuron with a small weight h gives a very wide bump, so it adds little. And a neuron with a negative weight is as useful as a positive one: it just reads the ruler backwards. What counts is the size of h compared with the noise.

When neurons share noise

The sum-of-precisions rule assumed independent noise. Real neurons are not independent. Two cells that receive the same input make the same mistake at the same time. Count them as two witnesses and you count one fact twice.

Work a number. Two neurons with noise correlation 0.8 and unit noise each. Their average has variance (1 + 0.8) / 2 = 0.9, almost as noisy as one neuron (variance 1). Two independent neurons would give 0.5. The pair is worth 1.1 independent neurons, not 2.

That is why Wu and colleagues made Q a full matrix, with every correlation, and not a diagonal. They tested both. On the pinball task the mean squared error was 5.87 cm2 with the full matrix and 6.91 cm2 with a diagonal one (their Table 5). On the pursuit task it was 5.99 against 6.35. The cost is size: Q has C(C + 1) / 2 free numbers, 903 for 42 neurons, and the paper warns that fitting it is hard for large populations.

What the measurement can and cannot see

In practice the weights on velocity are much larger than on position. The spikes tell the decoder how fast the hand moves, not so well where it is. Position comes mostly from adding up velocity through the physics of Chapter 1. This is why the pink path in the hero drifts and jitters in position while the Kalman path does not: the prior supplies the integration that the raw spikes lack.

Two neurons have noise that is strongly correlated. A decoder that treats their noise as independent will...

Chapter 3

Predict, Then Correct

Merge the physics and the spikes with a single trust weight, and follow every number through one full step

You are driving and two friends give directions. One has watched the road so far and says, “Keep going straight, like before.” The other glanced at a map and says, “Turn left.” Whom do you believe? A sensible driver asks how reliable each friend has been, and leans toward the more reliable one.

The Kalman filter makes that judgement exact. The first friend is the prior from Chapter 1. The second is the spike measurement from Chapter 2. Each comes with a stated uncertainty. The filter weights each by its reliability and gives a new best guess, with a new, smaller uncertainty.

This chapter covers only what the decoder needs. Our Kalman filter lesson derives the same algorithm from scratch, and RTS smoothing shows the offline version that also looks backwards. Here we stay with the online decoder.

Two moves in every bin

Predict. Push the last estimate x̂ through the physics, and grow the uncertainty by one kick. The result is the prior for this bin, written with a minus sign: x̂− and P−.

x̂−
The predicted state: where the physics says the hand is now, before looking at the spikes.
A
The physics matrix from Chapter 1.
P−
The predicted covariance: how unsure that prediction is. It is always larger than last bin’s P.
W
The kick covariance, the price of one step of not knowing.

Correct. Ask what the prediction says the spikes should have been: H x̂−. Subtract that from the real spikes z. The difference is the innovation: the part of the spikes that the physics did not expect. Add a weighted copy of the innovation to the prediction.

K
The Kalman gain, a 4 × C matrix. Each column says how a surprise in one neuron moves the four state numbers. It is the trust slider.
Q
The measurement noise from Chapter 2. A large Q makes the bracket large and K small: the filter trusts the spikes less.
zk − H x̂−
The innovation: the real spikes minus the spikes the prediction expected.

After the correction the uncertainty shrinks: P = (I − K H) P−. That is the whole bin. Each bin repeats it.

One step by hand

To see the gain at work, boil the C neurons down to one number. Suppose that, once combined, the spikes say velocity is 8.0 cm/s with a noise variance Q of 3.0. Say the prediction is 4.0 cm/s with variance P− of 6.0. In one dimension the gain is K = P− / (P− + Q).

K = 6.0 / (6.0 + 3.0) = 0.667    x̂ = 4.0 + 0.667 × (8.0 − 4.0) = 6.67 cm/s    P = (1 − 0.667) × 6.0 = 2.0

Read it in words. The spikes are twice as reliable as the prediction (variance 3 against 6), so the filter moves two thirds of the way to the spikes. The new variance, 2.0, is lower than both inputs. Check it with precisions: 1/6 + 1/3 = 0.5, and 1/0.5 = 2. Fusing two sources always helps.

Now the next bin. The physics pushes the estimate forward: x̂− = 0.94 × 6.67 = 6.27 cm/s. The variance becomes P− = 0.942 × 2.0 + 4 = 5.77. The spikes arrive again, and the cycle repeats. Step through a run below.

Step the filter

Purple bump: the prior. Pink bump: what the spikes say. Warm bump: the fused answer. Blue line: the true velocity. Press Step, then change the spike noise and step again.

3.0

Illustrative one-axis run: velocity fades 6% per bin with kick variance 4, the combined spikes read the truth plus noise of the variance you set, and the filter starts at 0 cm/s with variance 34.4. The bumps are drawn from the exact numbers in the read-out.

Try a very noisy measurement, a Q of 16. After a few bins the gain settles near 0.36, and the warm bump stays close to the purple one. The estimate is smooth but slow to follow a real change. Try Q = 0.5. The gain settles near 0.90, the warm bump jumps almost onto the pink one at every step, and the answer is as jittery as the raw spikes.

The data flow of the real decoder

Now put the real shapes back. The spike vector zk has C entries. The innovation also has C entries. The gain K is 4 × C (6 × C in the paper). The product K × innovation has 4 entries, and those are added to the 4-number prediction. So each bin turns a C-vector of surprises into a 4-vector of corrections.

The cost is small. The paper reports decoding at 1.54 ms per 70 ms bin for the pinball task, on a 866 MHz Pentium III. A modern processor runs the same arithmetic far faster.

The failure and its test

The symptom of a wrong Q or W is a cursor that is either jumpy or sluggish. The measurement is simple. Collect the innovations over a stretch of data. If the filter is well tuned, they average to zero and their spread matches H P− HT + Q. Innovations that are larger than predicted mean the filter trusts itself too much. Smaller ones mean it is too timid. The fix is to refit W and Q from data, which is the next chapter.

The measurement noise Q becomes much larger. What happens to the gain K and to the cursor?

Chapter 4

Fitting A, W, H and Q from Data

Learn the whole decoder from a few minutes of paired hand and spike recordings, with two least-squares fits and no search

On day one of a new implant, the decoder knows nothing. It does not know which neurons like which directions. It does not know how jerky this person’s movements are. Before the cursor can move, the lab runs a short calibration: a few minutes in which hand and spikes are recorded together.

The result is two tables of numbers. One is X, with M rows (one per bin) and 4 columns (the hand state). The other is Z, with M rows and C columns (the spikes). Everything the decoder needs is hidden in how these two tables relate.

Two regressions, nothing more

Wu and colleagues set the problem up as maximum likelihood: pick A, W, H and Q to make the recorded data as probable as possible. For linear Gaussian models the answer comes in closed form. In plain words it is two ordinary regressions.

Regression one: the physics. Predict the next state from the current state, using all pairs of neighbouring bins. The best-fitting matrix is A. What is left over, the error of the prediction, is the kick. Its covariance is W.

Regression two: the tuning. Predict the spikes from the state, bin by bin. The best-fitting matrix is H, with one row per neuron. What is left over is the measurement noise. Its covariance, across all neurons at once, is Q.

H
The least-squares tuning matrix, C × 4. The first sum is how the spikes line up with the state. The second is how the state lines up with itself.
Q
The covariance of the residuals: the average of each residual list times its own transpose. Its diagonal is each neuron’s leftover variance. Its other cells are the shared errors.

The same recipe gives A and W: replace zk by xk and xk by xk−1. The paper’s appendix A.1 states exactly these closed forms. Our generalized linear models lesson shows the same idea for spike counts with a Poisson noise model.

A fit by hand

Take one neuron and one state number, the velocity. Five centred bins have v = [−4, −2, 0, 2, 4] cm/s and spikes z = [−1.4, −0.4, 0.1, 0.2, 1.5]. Compute the sums.

Σ z v = 5.6 + 0.8 + 0 + 0.4 + 6.0 = 12.8    Σ v2 = 16 + 4 + 0 + 4 + 16 = 40    H = 12.8 / 40 = 0.32

So the neuron rises by 0.32 for each cm/s. Its predictions are [−1.28, −0.64, 0, 0.64, 1.28]. The residuals are [−0.12, 0.24, 0.10, −0.44, 0.22]. Their squares sum to 0.324, so Q = 0.324 / 5 = 0.065, an sd of 0.25. The same steps, in matrix form, fit all C neurons at once.

How much calibration is enough

Each point is a full refit of A, W, H and Q on that many minutes of simulated data, then a test on a path the fit never saw. Drag the training length, then change the neuron count.

2.5 min
24

Illustrative simulation, fitted live in your browser with the closed forms above. Test error is the mean position error on a 30 s test path, using the steady-state gain of Chapter 5. Parameter count = 16 (A) + 10 (W, symmetric) + 4C (H) + C(C + 1)/2 (Q). The paper reports about 2.5 minutes sufficed on its pinball data (its section 3.7).

The Kalman curve falls fast and flattens near 2.5 minutes. That matches the paper: Wu and colleagues found that about 2.5 minutes of pinball training was enough, while their pursuit task used about 10 minutes. The pink line (spikes only) stays high. It never gets the benefit of the physics.

Speed, and where the labels come from

Fitting is fast. The paper reports 0.14 s to train on 3.5 minutes of pinball data, and 1.70 s for 17 minutes of pursuit data, on a 2006 desktop. A lab can refit between two trials.

One catch: the recipe needs the true state X. A monkey’s hand is tracked, so X is measured. A person with paralysis has no hand movement to record. Labs then use a stand-in, such as the path of a target the person is asked to follow in imagination. Gilja and colleagues (2012) went a step further and retrained the decoder on the movement the user intended in closed loop. See their paper for the details.

The failure: too many parameters for the data

The symptom is a decoder that looks perfect on training data and jumpy on new data. The measurement is the held-out error, the same test the device above runs. Drag training to 0.1 minutes at 48 neurons. Q has 1,176 numbers to learn from 86 bins. It cannot be fitted well, and the error jumps.

There are three fixes. Record more data. Use fewer neurons, or fewer state numbers. Or shrink Q toward a diagonal matrix, which has only C numbers. The paper’s remark that the full matrix “may be problematic for large populations” is this same trade.

The core in code

import numpy as np

def fit_kalman(X, Z):
    """X: (M, 4) hand state per bin.   Z: (M, C) square-root firing rates."""
    X = X - X.mean(0);  Z = Z - Z.mean(0)             # centre both
    X0, X1 = X[:-1], X[1:]
    A = np.linalg.solve(X0.T @ X0, X0.T @ X1).T       # (4, 4)  next state from this state
    E = X1 - X0 @ A.T;           W = E.T @ E / len(E)  # (4, 4)  kicks
    H = np.linalg.solve(X.T @ X, X.T @ Z).T           # (C, 4)  spikes from state
    R = Z - X @ H.T;             Q = R.T @ R / len(R)  # (C, C)  measurement noise
    return A, W, H, Q
What is the matrix W, in terms of the regression that produces it?

Chapter 5

The Steady-State Gain

See why the gain stops changing after about a second, and why that turns the decoder into one fixed rule with a fading memory

Picture a new waiter on the first night. At first, every customer is a surprise. The waiter has no habits, so each new table gets full attention. After a week, the routine is set. The waiter still serves each table with care, but the amount of attention per table no longer changes.

The Kalman filter does the same. Its gain K starts large, because the filter begins with little idea where the hand is. As bins pass, it settles. This chapter shows when and why, and what is left once it has settled.

The gain never looks at the spikes

Look again at the filter. The formulas for P−, K and P use only A, W, H and Q. The spikes z appear only in the update of the estimate. So the whole sequence of gains can be computed before a single spike arrives.

Now run the recursion. Each bin the uncertainty grows by the kick (W) and shrinks by the correction (K H). The two forces balance. When A, H, W and Q stay constant, P− and K converge to fixed values. This is the steady state, proved by Kalman and Bucy in 1961. Wu and colleagues write that it happens in practice after about 20 bins, and they derive what it means in their appendix A.3.

What the filter becomes

Once K is fixed, the update is one line. Write M = (I − K H) A. Then:

x̂k
The decoded state in bin k.
M
A 4 × 4 matrix, (I − K H) A. It says how much of the last estimate survives into this one. Its size is below one, so memory fades.
K
The fixed gain, 4 × C: how much each new spike vector moves the estimate.

The right-hand side is the unrolled form. The estimate is a weighted sum of all past spike vectors, with weights Mj K. Because M fades, old spikes get exponentially smaller weights. That is a linear filter with a fading memory, which is how earlier decoders worked. The Wiener filter, the classic fixed linear filter, does the same sum with weights fitted directly. The Kalman filter finds a very similar set of weights from the physics (A, W) and the tuning (H, Q), and runs as a short recursion instead of a long sum. The paper states this link in its appendix.

The worked example

Take the one-dimensional example of Chapter 3: A = 0.94, W = 4, Q = 3. Run the covariance recursion. The gain is 0.92 in bin 1, then 0.68, 0.66, 0.657, 0.657. It has settled within four bins. The steady values are P− = 5.74 and K = 0.657.

M = (1 − 0.657) × 0.94 = 0.323    weights on past spikes: 0.657, 0.212, 0.068, 0.022, …

Each older bin counts 0.323 times as much as the one after it. A spike reading from three bins ago has only 3% of the weight of the newest. With 24 or 48 neurons and four state numbers the shape is different. Weights fall fast at first, then taper slowly. Velocity fades quickly, but position is a running total of velocity, so it forgets slowly. The device runs the full matrices on our simulated neurons.

Watch the gain settle

Top: the size of the gain matrix K in every bin from the start. Bottom: how much weight a spike vector from j bins ago gets in the settled filter. Change the neuron count and how much the filter trusts the spikes.

24
×1.00

Illustrative simulation: A, W, H and Q are fitted on 3.5 minutes of simulated data, then the covariance recursion runs from the training covariance. “Size” is the Frobenius norm of K (the square root of the sum of its squared entries). The spike-noise knob multiplies Q. Settled means within 1% of the final size.

Why it matters in practice

A settled filter is cheap and fixed. The decoder stores one matrix M (16 numbers) and one matrix K (4 × C numbers). Each bin needs about 16 + 4C multiply-adds. For 96 channels at 14 bins per second that is about 5,700 multiply-adds per second: trivial for any chip, even a small low-power one. The design is easy to check, because it is a plain linear filter.

K = steady_state_gain(A, W, H, Q)        # offline: iterate the covariance recursion to convergence
M = (np.eye(4) - K @ H) @ A               # (4, 4)
x = np.zeros(4)
for z in spike_stream():                  # z: (C,) one 70 ms bin, centred
    x = M @ x + K @ z                     # that is the whole decoder
    cursor.move_to(x[:2])

The failure: a stale gain

Two things break the shortcut. The first is the start: for the first 20 or so bins the true gain is larger and still changing, so a fixed K from the settled value will mis-weight the first spikes. The cure is to run the full recursion at start-up, as the paper does, or to hold the cursor still for a second. The second is a model that changes. If a neuron’s tuning drifts, H changes and the fixed K no longer matches. Chapter 7 shows the symptom and the cure.

Why can the whole sequence of gains K be computed before any spikes are recorded?

Chapter 6

Lag and Bin Width

Align spikes with the hand that follows them, choose how wide a bin to count in, and read both choices off the paper’s own tables

A sprinter hears the starting gun, and the legs move a moment later. In between, signals travel and muscles build force. The brain works the same way. A motor neuron that helps drive a reach fires before the hand moves. If we pair each spike count with the hand position at the same instant, we pair the cause with the wrong effect.

The lag

The fix is to shift one table against the other. The model of Chapter 2 becomes zk−i = H xk + q. Spikes from i bins ago explain the hand now. The number i is the lag.

How large is i? Wu and colleagues tried every whole number of bins and kept the best. On the pinball task (70 ms bins) they tried 0, 70, 140, 210 and 280 ms (their Table 2). The error was 7.01 cm2 with no lag, fell to 5.87 at 140 ms, then rose again to 6.80 at 210 ms and 8.81 at 280 ms. On the pursuit task the best lag was about 150 ms. Earlier work by Moran and Schwartz had also found about 145 ms between firing and hand velocity.

Even “zero lag” is not zero. A firing rate counts spikes over the previous 70 ms, so the count already describes the recent past. The paper points this out. A lag of 0 ms in the table still has a built-in delay of up to one bin.

The picture is a U. Too little lag pairs the spikes with a hand that has not moved yet. Too much pairs them with a hand that has moved on. The error is lowest in between.

One lag, or one per neuron

Not every neuron leads by the same amount. The paper let each cell choose its own lag from 0 to 280 ms, using a greedy search: fix all lags, adjust one cell at a time, repeat. This is cheap, because the search only refits H and Q. Per-cell lags brought the pinball error down from 5.87 to 4.75 cm2 (a 19% cut). On the pursuit task they gave no gain. The authors still used the simple 140 ms setting for the rest of their experiments, for simplicity.

def lag_pairs(X, Z, i):
    """Pair the spikes from i bins ago with the hand state now."""
    return (X[i:], Z[:len(Z) - i]) if i else (X, Z)

best = min(range(0, 5), key=lambda i: heldout_error(fit_kalman(*lag_pairs(X, Z, i)), Xtest, Ztest, i))

The width of the bin

Chapter 0 showed the toy version of this trade: wide bins hold more spikes and less noise, and narrow bins react faster. The paper measured it (its Table 7). The authors re-binned the data, refitted the model each time and tested it.

On the pinball task the error was 5.87 cm2 at 70 ms. It fell to 5.29 at 140 ms and to 5.16 at 280 ms, then climbed: 5.95 at 350 ms and 7.25 at 490 ms. On the pursuit task, with its slow movements, wider bins kept helping up to 400 ms: 5.99 at 50 ms, 5.03 at 200 ms, 4.59 at 400 ms, then 4.64 at 500 ms. The authors give a reason: as bins grow, the firing rates are closer to a Gaussian, which is what the model assumes. Bin width also changes the dynamics in A, so every row of the table is a full refit.

Look at the shapes of the two curves together with the speed of the tasks. The pinball hand averaged 14.67 cm/s, about five times the pursuit task’s 2.88 cm/s. The faster hand wants narrower bins. That matches the toy of Chapter 0: lag error grows with acceleration.

Read the paper’s tables

Each bar is a mean squared error from the paper: shorter is better. Pick a table, then drag the slider to move through its settings. The green ring marks the best one.

140 ms

All numbers are Wu et al. 2006, offline reconstruction of monkey hand position. Lag: Table 2 (pinball, position + velocity + acceleration state, 42 units). Bin width: Table 7 (pinball with 140 ms lag; pursuit with 150 ms lag and a position + velocity state, 25 units). Error is mean squared error in cm². Lower is better.

What it means for a live cursor

These tables come from offline tests on recorded monkey data, and the paper says closed-loop tests were still being done. In a live system the picture shifts. The cursor is what the user watches. A 280 ms bin means 3.6 cursor updates per second, which looks choppy, even when its error is the lowest on paper. And the loop itself adds delay: bin, decode, draw.

So a builder reads the tables as a map, not a recipe. The best offline bin is the widest one before the error rises. The best live bin is usually narrower, with the physics prior doing the smoothing that a wide bin would otherwise do. That is the point of the Kalman filter. It gives a narrow bin the benefit of a wide one.

The failure: the wrong alignment

The symptom is a decoded path that trails or leads the true one by a roughly constant amount. Measure it: scan the lag over a small range and plot the held-out error. A clear U-shaped curve with a minimum, like Table 2, tells you the right value. A flat curve tells you it does not matter for your data, as on the pursuit task. The fix is to pick the lag by held-out error, not by the training error, which keeps improving as you add freedom.

In the paper’s Table 2 the error is 7.01 at 0 ms lag, 5.87 at 140 ms and 8.81 at 280 ms. What does this U shape tell you?

Chapter 7

When the Neurons Drift

Break the decoder by shifting the neurons’ baselines, measure the creep, and fix it with a short recalibration

On Monday the decoder is sharp. On Wednesday the same person, the same array and the same settings give a cursor that creeps toward the top-left corner. Nobody changed the code. The brain changed under it.

Neurons are not fixed instruments. The array moves a few micrometres as the head moves. Scar tissue grows. A neuron that fired 12 spikes per second last week fires 15 now. A neuron drops out and another appears. Scientists call this nonstationarity: the statistics change over time. The model of Chapter 2 assumed that H and the baseline firing stay the same. In real life they do not.

Noise averages out. Drift does not.

Chapters 0 to 3 dealt with noise: random errors that point in a new direction each bin. A filter shrinks them, because they cancel. Drift is different. It is a steady push in one direction. It does not cancel. The filter reads it as real motion.

Take the simplest drift: one neuron’s baseline rises by 0.4 units. The decoder expects zero baseline (we centred the rates). If this neuron has weight h = 0.2 per cm/s, then 0.4 / 0.2 = 2 cm/s of “movement” appears out of nothing. With 24 neurons whose shifts point in random directions, the pushes partly cancel, to about 2 / √24 = 0.4 cm/s. That sounds small. But the physics of Chapter 1 adds up velocity into position. A bias of 0.4 cm/s becomes 24 cm after a minute, far off the edge of a screen.

The cursor creeps in one direction, and the user pushes back with imagined effort, which makes the decoder yet more wrong. In practice, users learn to compensate. Still, the lab has to measure the drift and fix it.

The test: are the innovations centred?

The symptom is a cursor that creeps. The measurement comes from Chapter 3. In a well-tuned filter, the innovation z − H x̂− averages to zero over time. A running mean that wanders away from zero is the alarm. It also tells you which neurons moved.

Here is the arithmetic. Say a neuron’s noise has sd 1.0 and its baseline rose by 0.4. After 100 bins (7 s) the mean of its innovations has an sd of 1.0 / √100 = 0.1 from chance alone. A mean of 0.4 sits four sds from zero. That is a clear alarm after seven seconds, long before the cursor has crept far. A decoder that logs this number per neuron can warn the lab, or recalibrate itself.

The fix has three levels, from cheap to costly.

  1. Re-centre the baselines. Run a short block in which the intended movement is known, such as moving to a cued target. Compute the mean of z − H x over the block and subtract it afterwards. It takes seconds.
  2. Refit H and Q. Rerun Chapter 4 on a fresh block of data. It takes minutes. Adaptive filters, like those in our adaptive Kalman lesson, do this continuously.
  3. Align the model to the new neurons. Later work, such as Degenhart and colleagues (2020), aligns the low-dimensional space of neural activity across days so that the old decoder keeps working. Silversmith and colleagues (2020) built a decoder that stabilises itself. The FALCON benchmark (Karpowicz et al., 2024) tests how well decoders hold up across days. See the links in the Field Guide.

Let the neurons drift

Over 90 seconds, each of 24 simulated neurons slowly changes its baseline. Red: a decoder that is never recalibrated. Green: the same decoder, re-centred at the interval you choose. Raise the drift, then change the interval.

1.00

Illustrative simulation, not recorded data: 24 neurons, drift = each baseline ramps to ± that many units (random sign) by 90 s, fixed steady-state gain from Chapter 5. A recalibration subtracts the mean of z − H x over the last 20 bins, using the true state as a cued target would. Error is the position error, averaged over a moving 5 s window.

Set drift to 1.00 and recalibration to never. The red line rises steadily, to over 13 cm by the end. Switch to every 30 seconds and the line drops back to a couple of centimetres. Now put drift to zero and switch to every 10 seconds. The green line sits above the red one. Why? Each recalibration is estimated from only 20 noisy bins, so it injects a small error of its own. Calibrate as often as the drift needs, and no more.

Model mismatch is a cousin of drift

The Kalman filter is optimal only when its model is true: linear tuning, Gaussian noise, physics that does not change. Wu and colleagues were frank about the tuning: the linear fit is “crude but reasonable”. Counts are not Gaussian, and the kicks in our simulation are bursts when a new target appears. The filter degrades gracefully under such mismatch. But it cannot repair it, and neither can a tighter fit. If the model is wrong in a way the data show, move to a model that can represent it, which is the subject of the next chapter.

Why does a small steady shift in a neuron’s baseline hurt a Kalman decoder more than the same amount of random noise?

Chapter 8

Against the Field

Compare the Kalman decoder with the population vector and the linear filter on the paper’s own table, then see where the field went next

A new tool needs a fair fight. Wu and colleagues put their Kalman decoder against two older ones on the same monkey data. Each decoder has a clear idea of its own. Put the ideas in order and the results make sense.

Three ideas for the same job

The population vector. Every neuron votes for its preferred direction, and the vote is as strong as its firing rate. Add the votes as arrows. The sum points where the hand is going. Georgopoulos and colleagues described this in 1986, and Taylor and colleagues used it in closed loop in 2002. It needs no training of dynamics, and no noise model. But it assumes the preferred directions cover all directions evenly. In the paper’s data they do not (its Figure 7), and the vote is biased toward the favoured directions.

The linear filter. Predict the hand position as a weighted sum of the firing rates of the last N bins. Fit the weights by least squares. In the paper, N was 14 bins for the pinball task, about one second of history. This is the Wiener filter idea behind the closed-loop work of Serruya and colleagues in 2002. It has no model of how the hand moves, only a long memory of spikes.

The Kalman decoder. A physics prior, a tuning model, and a recursive fusion, as in this lesson. Chapter 5 showed that its settled form is itself a linear filter over the past spikes. The difference is where the weights come from: from a small physical model, not from a free fit on every lag.

The paper’s scorecard

Table 6 of the paper gives the mean squared error of each method on each task. On the pinball task the population vector scored 75.0 cm2, the linear filter 6.48 and the Kalman decoder 4.55. The 4.55 used 140 ms bins and a lag chosen for each cell, not the 70 ms bins of the earlier chapters. On the pursuit task the scores were 13.2, 4.74 and 4.66.

Pinball: (6.48 − 4.55) / 6.48 = 30% lower error than the linear filter    Pursuit: (4.74 − 4.66) / 4.74 = 1.7% lower

Read these numbers with care. The population vector loses badly on both tasks, and the paper explains why: the movements are complex, and the preferred directions are uneven. The Kalman decoder wins clearly on the fast, free pinball movements. On the slow pursuit task it barely beats the linear filter. The gain depends on how much the physics prior can help, and a slow, smooth hand is already easy.

Read the scorecard

Bars show each decoder’s mean squared error from the paper. Switch the task, then use Zoom to cut off the population vector and compare the two filters.

All numbers are Wu et al. 2006, Table 6, offline reconstruction of monkey hand position. Pinball: Kalman with 140 ms bins and a lag chosen per cell (nonuniform); linear filter with N = 14 bins; 42 units. Pursuit: Kalman with 300 ms bins and 150 ms lag; linear filter with N = 30; 25 units. Correlation values are in the paper.

What the paper adds beyond the score

The Kalman decoder gives more than a point estimate. It keeps the covariance P, which says how unsure each estimate is. The paper lists this as a design goal: indicate the uncertainty. It also gave a way to check the model. The authors tested different kinematic states, lags and bin widths, and showed that the full covariance Q matters.

A note on parameters. The linear filter with N = 14 on 42 units has 14 × 42 = 588 weights for each of the two axes, 1,176 in all. The Kalman model with six state numbers has 36 + 21 + 252 + 903 = 1,212. The count is about the same. The difference is structure: a Kalman model puts its numbers where the physics and the neurons say they belong.

Where the field went next

The linear-Gaussian model has limits, and the later literature pushes on each. All the names below come from the reading list of this series; look each up for its own numbers.

Why does the Kalman decoder still matter? It is the baseline every new method must beat, and it trains in seconds. It is simple enough to check line by line. And its idea, a state, a model of change and a model of measurement, survives in each of its successors.

On the pursuit task in Table 6, the Kalman decoder (4.66) barely beats the linear filter (4.74). What is the best reading?

Chapter 9

Field Guide: The Kalman Decoder on One Page

Keep the numbers, the checklist and the code, and find the sources and the lessons to read next

The one-paragraph summary

A hand moves by physics plus small kicks, so the state xk+1 = A xk + w predicts the next bin, with growing doubt (W). Spikes are noisy rulers of that state, z = H x + q, with correlated noise (Q). Four matrices, A, W, H and Q, come from two least-squares fits on a few minutes of paired data. Each bin the filter predicts, then moves the prediction toward the spikes by the gain K, which depends only on those four matrices. After about 20 bins K settles, and the decoder becomes one fixed rule, x̂ = M x̂ + K z, with a fading memory of past spikes. Lag, bin width and drift are the tuning knobs that the data, not the algebra, must set.

The numbers that matter

WhatNumberWhy it matters
Paper and dataWu et al. 2006; pinball: 42 units, 70 ms bins, 3.5 min train, 1 min test; pursuit: 25 units, 50 ms binsOffline reconstruction of monkey hand position
State order (pinball, Table 3)MSE 7.72 (position), 6.31 (+ velocity), 5.87 (+ acceleration), 5.72 (+ jerk)Higher derivatives help, with diminishing returns
Full vs diagonal Q (Table 5)5.87 vs 6.91 (pinball); 5.99 vs 6.35 (pursuit)Shared neuron noise is worth modelling
Training data neededabout 2.5 min (pinball); about 10 min (pursuit)Calibration is minutes, not hours
Fit and decode time (2006 PC)0.14 s to train on 3.5 min; 1.54 ms to decode one 70 ms binReal-time cost is negligible
Best uniform lag (Table 2)140 ms: MSE 5.87 (0 ms: 7.01; 280 ms: 8.81); per-cell lags: 4.75Spikes lead the hand
Bin width (Table 7)pinball best 5.16 at 280 ms; pursuit best 4.59 at 400 msWider bins help until they lag
Scorecard (Table 6, pinball)population vector 75.0; linear filter 6.48; Kalman 4.55Kalman 30% below the linear filter
Scorecard (Table 6, pursuit)13.2; 4.74; 4.66A slow hand narrows the gap to 1.7%
Steady stateK converges after about 20 bins (1.4 s at 70 ms)Compute K once, offline

The checklist before you trust a Kalman decoder

  1. State. Position and velocity at least. Add acceleration only if held-out error falls.
  2. Preprocess. Square-root the rates, then centre rates and states on the training means.
  3. Fit. A and W from neighbouring states; H and Q from state and spikes. Keep Q full unless data are short.
  4. Align. Scan the lag and the bin width on held-out data, never on training data.
  5. Start up. Run the full covariance recursion for the first 20 bins, then switch to the fixed gain.
  6. Monitor. Track the running mean of each neuron’s innovation. Recalibrate when it leaves zero.
  7. Compare. Beat the linear filter on the same data before you claim a gain.

The core, as code

import numpy as np
from scipy.linalg import solve_discrete_are

def fit(X, Z):                                   # X: (M, 4)  Z: (M, C)  sqrt rates
    mx, mz = X.mean(0), Z.mean(0);  X, Z = X - mx, Z - mz
    X0, X1 = X[:-1], X[1:]
    A = np.linalg.solve(X0.T @ X0, X0.T @ X1).T                  # (4, 4)
    E = X1 - X0 @ A.T;  W = E.T @ E / len(E)                     # (4, 4)
    H = np.linalg.solve(X.T @ X, X.T @ Z).T                      # (C, 4)
    R = Z - X @ H.T;    Q = R.T @ R / len(R)                     # (C, C)
    return A, W, H, Q, mx, mz

def steady_gain(A, W, H, Q):
    Pp = solve_discrete_are(A.T, H.T, W, Q)                      # predicted covariance, (4, 4)
    return Pp @ H.T @ np.linalg.inv(H @ Pp @ H.T + Q)            # K: (4, C)

def decode(Z, A, H, K, mx, mz):
    M, x, out = (np.eye(4) - K @ H) @ A, np.zeros(4), []
    for z in Z - mz:                                             # one 70 ms bin at a time
        x = M @ x + K @ z
        out.append(x + mx)
    return np.array(out)                                         # (M, 4): position and velocity

Where this sits

This lesson is the estimation core of motor-cortex decoding. Before it come spikes, tuning and population vectors. After it come dynamics-aware decoders, latent-variable models and pretrained neural networks, which keep the state-space idea and swap out the straight-line parts. The Kalman filter is also the base of sensor fusion in robots and vehicles, where the “spikes” are GPS or an IMU.

Keep going

Sources

The ideas and every number from the paper come from Wu, Gao, Bienenstock, Donoghue and Black, “Bayesian population decoding of motor cortical activity using a Kalman filter”, Neural Computation 18, 80 to 118 (2006), read in the open copy hosted by Brown University: the model (its sections 2.2 and appendix A.1), the steady state (appendix A.3), and Tables 2, 3, 5, 6 and 7. The closed-loop follow-up, which the paper cites, is Wu et al., IEEE EMBS 2004. Everything labelled illustrative is our own simulation.

Named in Chapters 2, 4, 7 and 8, and worth opening: Georgopoulos, Schwartz and Kettner 1986 (population vector); Serruya et al. 2002 (closed-loop linear filter); Taylor, Tillery and Schwartz 2002; Gilja et al. 2012 (ReFIT); Churchland et al. 2012 (rotations); Kao et al. 2015; Pandarinath et al. 2018 (LFADS); Degenhart et al. 2020; Silversmith et al. 2020; the FALCON benchmark; and the tutorial Glaser et al. 2020, “Machine learning for neural decoding”. We opened the Wu paper; for the others we relied on their titles and the series reading list, so check their own numbers before quoting them.

The takeaway. The Kalman decoder is two small models and one trust weight. The physics model says where the hand probably is. The tuning model says what the spikes mean. The gain, computed from both, says how far to move toward the spikes. Fit the models from minutes of data, check the lag and bin on held-out data, and watch the innovations for drift.
A lab refits a Kalman decoder every morning. After a month the cursor creeps again within an hour. Which check comes first?

Now press Teach and explain the decoder out loud from memory: what the state and the prior are, how the spikes become a measurement, how A, W, H and Q are fitted, and why the gain settles. Then go back to the hero, switch between Each bin alone and Kalman at 8 neurons and at 48, and say what the physics prior is doing.

The Kalman Decoder, BCI series lesson 16
Wu et al. 2006 · Kalman filter lesson