A decoder that scores well on recorded data can still steer a cursor badly. Retrain it on what the user meant, and in this toy a 31% hit rate becomes 100%.
See why you must judge a brain-computer interface with the user in the loop, and how the ReFIT idea of Gilja and colleagues fixes a decoder by relabelling its training data.
Choose a decoder, then press Run 16 trials. Each line is one trip from the centre to a target. Then we build, piece by piece, the loop, the labels, the Kalman filter and the two learners.
You need a rough idea of a straight-line fit (the line that passes closest to a cloud of points). We build the rest from zero.
Based on Gilja et al., Nature Neuroscience 2012, and Taylor, Tillery and Schwartz, Science 2002. All neurons, cursors and numbers in the devices are a seeded toy unless a caption says they are the paper’s.
Toy: 14 simulated neurons, a simulated user who steers toward the target and sees the cursor 150 ms late, and a linear decoder. A trial counts as a hit if the cursor sits in the target for 500 ms within 5 s, the task rules of Gilja et al. The first run of every visit is seeded, so it shows the same trips each time; later runs draw new noise. The numbers are this toy’s, not the paper’s.
Chapter 0
See why the same decoder behaves differently on a recording and in a person’s hands
Picture a game controller that is not plugged in. You push the stick and nothing moves on the screen. Now plug it in, but with a fault: the cursor veers 30 degrees to the right of where you push. For one second you miss. Then you push a little to the left, and you hit the target.
You did not repair the controller. You changed your push, because you could see the cursor. Your eyes closed a circle: the controller moved the cursor, the cursor reached your eyes, and your eyes changed your next push.
A brain-computer interface (BCI) is that controller, with the motor cortex as the stick. Tiny electrodes record the spikes (brief electrical pulses) of many neurons. A program called a decoder turns the spike counts into a cursor move, about every 50 milliseconds. The decoder is a function: neural activity goes in, a velocity comes out.
You can test a decoder in two very different ways. In an open-loop test you record neural activity while a person does a known task, such as watching a cursor move or reaching with a hand. Later, offline, you feed the recording to the decoder and compare its output to the known answer. Nobody sees the decoder’s output. What it says cannot change the next sample.
In a closed-loop test the person sees the cursor that the decoder moves. The output feeds back into the brain through the eyes. This feedback is what makes the system a loop. The person steers, and the decoder’s mistakes become part of what the person reacts to.
The two tests can disagree, and that is the subject of this lesson. A decoder can look excellent in an open-loop test and perform badly in a closed loop. The reverse can also happen. The test that matters is the closed-loop one, because a BCI is only ever used with the person in the loop.
The loop adds three things that a recording cannot show. First, the plant changes. In control language the plant is the thing you steer. In a recording the plant is the person’s own arm. In a BCI it is a cursor with different dynamics, so the person may use a different strategy to steer it.
Second, the person corrects errors. The user partly cancels a small bias in the decoder by pushing the other way. An offline test cannot see this help, so it cannot tell a good decoder from one that the user is rescuing.
Third, the neurons themselves can change. Taylor, Tillery and Schwartz reported in 2002 that cells changed their tuning when monkeys moved a cursor with brain control. We return to that in Chapter 6. For now, keep one picture: the decoder and the brain are two parts of one circuit, and you cannot test one part alone.
Try it. The device below uses the same decoder twice. The first time, the simulated user cannot see the cursor, so the user steers as if the decoder were perfect. The second time, the user watches the cursor and corrects.
Eight trips from the centre, one to each target. Choose Loop open and the simulated user steers blind. Choose Loop closed and the user sees the cursor and corrects. The decoder is the same in both.
Toy: the arm-trained decoder of the hero page, 14 simulated neurons, noise level 5. The blind user steers by an imagined cursor that always obeys; the closed-loop user steers by the real cursor, seen 150 ms late. A trip ends when the cursor has held a target for 500 ms or after 5 s. Miss is the distance from the cursor to the target centre at the end of the trip, in screen units where the targets sit 0.70 from the centre. Illustrative, seeded.
Look at the open-loop paths first. Each one leaves the centre at the wrong angle, because the decoder is slightly wrong for these neurons, and the error grows with every step. Now close the loop. The same wrong decoder, the same noisy neurons, and the paths bend back toward the targets. The user is doing work that no offline score counts.
Suppose you build a decoder and measure its accuracy on yesterday’s recording. You then ship it. Three things can go wrong that your test never exercised. The user may steer differently. The user may rescue a weak decoder today and tire of it tomorrow. The decoder may change the user’s neurons, which changes the data you train the next decoder on.
So a good BCI team treats the loop as the unit of design. It trains the decoder on data from the loop, labels the data with care, and judges success by what the person achieves with the cursor. This lesson walks that path, with the two papers that started it.
Chapter 1
Build the simplest decoder from scratch: neurons with favourite directions, and a straight-line fit that reads them
Imagine a room with fourteen people. Each person has a favourite direction. When you point the cursor in that direction, the person shouts loudly. When you point the opposite way, the person goes quiet. For other directions the shout is in between.
You stand outside the room and hear only the shouts. Can you tell which way the cursor should go? Yes, if you know each person’s favourite. Add up the favourites, each weighted by how loudly the person shouts, and the total points the right way.
Motor cortex works like this room. In 1986 Georgopoulos, Schwartz and Kettner reported that single neurons fire most for one direction of arm reach, and that the combined activity of many neurons points along the direction of the reach. A neuron’s firing rate is its number of spikes per second. The direction where it fires most is its preferred direction. The shape of rate against direction is its tuning curve, close to a cosine.
Write the intended move as a velocity with two parts: ux to the right and uy up. A cosine-tuned neuron i then fires like this.
This is a generative story: the intention comes first and the spikes follow. A decoder runs the story backwards. It gets the spikes and must guess the intention.
The simplest decoder is a weighted sum. Subtract the baseline from each rate, multiply by a weight, and add up. Do it once for the x part of the velocity and once for the y part. In matrix form this is one product.
v̂ = D (y − b) shapes: y is 14 numbers, D is 2 × 14, v̂ is 2 numbers
Every 50 ms bin gives the decoder 14 spike rates. It answers with two numbers, the x and y speed of the cursor. The cursor then moves by that speed times 50 ms. The decoder has no memory in this version. Each bin stands alone.
Where does D come from? From training data: a list of bins where you know both the rates and the intended velocity. Least squares picks the weights that make the squared error between the decoder’s output and the known velocity as small as possible, summed over all bins.
Try a tiny case with one neuron and one velocity axis. Three bins give centred rates of 4, −2 and 6, and known velocities of 0.5, −0.3 and 0.7. The best single weight is the sum of rate times velocity divided by the sum of rate squared. That is (4×0.5 + (−2)×(−0.3) + 6×0.7) divided by (16 + 4 + 36), or 6.8 divided by 56, which is 0.121. The decoder then says 0.486, −0.243 and 0.729. Each is close to the truth.
With 14 neurons the same rule gives the matrix D. Stack the T training bins into a table Z of centred rates (T rows, 14 columns) and a table L of known velocities (T rows, 2 columns). Then D = LT Z (ZT Z + λI)−1. The small number λ is a safety margin called ridge. It keeps the fit steady when training data is scarce or two neurons are alike.
Try it below. Drag the direction and watch the fourteen rates. Then change how many training bins the decoder had. With few bins the decoded arrow wanders. With hundreds it settles on the truth.
Left: the rate of each neuron above or below its baseline. Right: the true direction (teal) and the decoder’s guess (warm). Drag the direction, then change the number of training bins.
Toy: 14 neurons with seeded preferred directions, baselines of 12 to 20 spikes per second and gains of 7 to 11. Training bins are random velocities with noise level 5 and ridge λ = 1. The average error is taken over 36 test directions with fresh seeded noise. Illustrative, not recorded data.
Here is the failure you will see first. The symptom: the decoded arrow wanders and disagrees with the true direction, even though the neurons are healthy. The measurement: average the angle between the true and decoded arrows over many test directions. In the device, 10 training bins give an average error of 29 degrees. At 40 bins it falls to 14 degrees. At 200 bins it reaches 11 degrees, and at 600 bins it stays at 11.
The last two numbers tell you something. More data stopped helping because the remaining error comes from the noise in each single reading, not from the weights. The fix therefore depends on the cause. If the fit has too little data, collect more bins or raise the ridge margin. If the readings are noisy, no weights can help, and you need smoothing, which is the job of Chapter 4.
Notice the cost of scarce data. You cannot ask a real BCI user for thousands of clean reaches every morning. This pressure drives much of what follows: the labels you fit on, and how often you refit.
Two notes on realism. Real decoders use many more channels, often hundreds, and they bin spikes at 20 to 50 ms. The cosine model is also a simplification. Later chapters keep the same plain linear decoder as the test bench, because it shows the loop effects without extra machinery. The sibling lessons on linear decoders and spikes to rates cover the neural side in more depth, and generalized linear models cover fits where spike counts are not Gaussian.
Chapter 2
Score three decoders offline and online, and watch the ranking flip
A teacher grades a student driver on a video of someone else driving. The student watches the video and says when to brake and when to turn. The grade is excellent. On the first real road the student drives badly. The test never put the student behind a wheel.
Offline decoder scores have the same weakness. This chapter shows three ways a score can look good while the cursor fails, using the decoders from the hero page.
The usual offline score is R², read “R squared”. It answers one question: what fraction of the ups and downs in the true velocity does the decoder’s output explain? A score of 1 means a perfect match. A score of 0 means the decoder does no better than guessing the average velocity.
To compute it you need a recording and a label for each bin. The label is the velocity you call “true”. The decoder reads the spikes of each held-out bin, and you compare its output to the label. The choice of label decides everything. That choice is where the three traps hide.
A small example makes R² concrete. Say the true velocities in three bins are 1, 2 and 3. Their average is 2, so their total variation is (1−2)² + 0 + (3−2)² = 2. A decoder outputs 1.2, 1.8 and 3.3. Its squared errors are 0.04, 0.04 and 0.09, which add to 0.17. Then R² = 1 − 0.17 / 2 = 0.915. The decoder explains about 92 percent of the variation.
We fitted the first decoder in the hero on data from before the loop. In the arm-reach stage of Gilja and colleagues, a monkey moves a hand and the cursor follows it. The decoder learns how spikes relate to hand velocity. It scores well on held-out reaches of the same kind.
Then the monkey steers by brain control alone. Gilja et al. wrote that the prosthetic device “constitutes a new physical plant with different dynamical properties than the native arm”, and that earlier algorithms assumed the subject uses the same control strategy as with the native arm. Taylor and colleagues had seen neurons change their tuning under brain control in 2002. If the tuning moves, the arm-trained map is wrong in the loop, and no score on arm data can warn you.
In a closed loop the user fixes small errors by steering the other way. An offline replay has no user to do that. So the same decoder can look worse offline than it feels online, and a poor decoder that a skilled user rescues can look fine in a short session. The score tells you about the decoder alone. The performance you care about belongs to the pair.
This trap is the sharpest. Suppose you refit the decoder on closed-loop data, and you use the cursor’s own velocity as the label, since it is the only velocity you can see. The decoder produced that velocity from the spikes. So the label is a function of the data already.
Work a one-neuron case. Take a decoder weight of 0.2 and centred rates of 4, −2 and 6. The cursor velocities are 0.8, −0.4 and 1.2. Now fit a new weight to these labels by least squares: (4×0.8 + (−2)×(−0.4) + 6×1.2) divided by 56 equals 11.2 divided by 56, which is 0.2. The new weight is the old weight. The fit has no error left to remove, so its R² is exactly 1.
The same holds for 14 neurons. Refitting on what the cursor did returns the decoder you started with, and scores a perfect 1.00 against its own labels. A decoder that cannot change cannot improve. It also cannot fail the test.
The device below measures all three decoders both ways. Switch the offline yardstick and watch the bars.
For each decoder: the offline R² (purple) and the share of 64 closed-loop trials that hit their target (warm). Choose which yardstick the offline score uses.
Toy, seeded. Arm-trained: scored on held-out arm-phase bins (the data it was built on). Refit on cursor: scored against the cursor velocity it produced. Refit on intent: scored against its own intent labels. The second yardstick scores all three on one closed-loop session against the simulated user’s true intended velocity, which a real experiment never has. Hit rate: 64 trials, 5 s limit.
Under the usual test, the refit-on-cursor decoder wins with a perfect 1.00, the intent decoder follows, and the arm-trained one trails. Online, the cursor-refit decoder hits exactly as many targets as the arm-trained one, because it is the same decoder. The offline ranking and the online ranking disagree.
Under the true-intent yardstick the ranking changes. Now the arm and cursor-refit decoders tie, and the intent-refit decoder leads. That ranking matches the hit rate. A simulation can use this yardstick. A real experiment cannot, because nobody can read the user’s true intention. The next chapter shows how to estimate it.
Here is how this failure looks in practice, as symptom, measurement and fix. The symptom: users say the cursor drifts and will not stop, yet the offline report shows a healthy score of 0.78. The measurement: run the decoder in the loop and count hits and overshoots, and compare the offline score against the true intent where a simulation or a calibration task gives it. In the device, the true-intent score of the arm-trained decoder is only 0.12. The fix: stop tuning the offline number and refit on closed-loop data with labels that carry new information.
Treat an offline score as a smoke alarm, not as a verdict. A bad score is a warning. A good score proves little. Judge a decoder in the loop, with the metrics of Chapter 7: success rate, time to target and throughput. The offline score still has a use. It is cheap, it runs on every recording, and it catches broken channels and bugs before a person sits down.
Chapter 3
Turn a messy closed-loop recording into clean training data by guessing the move the user was aiming for
You steer a toy car toward a door with a remote that pulls slightly left. The car drifts left, you correct, it drifts again.
A camera films the car. Later you ask what the driver wanted. If you answer from the footage alone, you say “to wiggle left and right”. That is wrong. The driver wanted the door.
The footage shows the cursor, not the wish. This gap is the core idea of Gilja and colleagues. During a closed-loop session, the observed cursor velocity is the decoder’s output, with all its faults. The user’s intention is a different thing: the move the user was trying to make.
Chapter 2 showed that labels made from the cursor’s own velocity teach the decoder nothing new. We need labels that carry new information. In a calibration task the experimenter knows where the target is. That is information the decoder never had.
Gilja et al. used it in a simple way. They “hypothesized that the monkey wished to move directly towards the target”. So for each time bin they take the observed cursor velocity and rotate it until it points at the target. The speed stays as it was. The paper describes this as rotating the velocities “towards the target” to generate the estimated intended velocities. They also set the intended velocity to zero while the cursor is on the target during the hold, because the best way to hold still is to keep the cursor still.
The prosthesis does not know the target while it runs. The paper states that it has no knowledge of targets during online control. The team uses targets only afterwards, to build the labels for the next fit. This is why the method is called a refit: a second fit, made after a closed-loop session, on relabelled data.
Take one time bin. The cursor is at (0.20, 0.10). The target is at (0.70, 0.00). The decoder moved the cursor with velocity (0.60, −0.30).
The observed move pointed 26.6 degrees below the horizontal. The intended move points 11.3 degrees below. The label turned the arrow by 15.3 degrees. A bin on the target gets the label (0, 0).
Think of what the neurons did in that bin. They fired for the user’s intention. The old decoder read them with the wrong map and sent the cursor 15 degrees off. If we now fit the new decoder to say “these spikes mean the intended arrow”, the new map fixes exactly that error. Repeat over thousands of bins and the fit finds the map from spikes to intent.
The data flow is short. During the session, log for each bin the 14 rates, the cursor position, the target position and the decoded velocity. After the session, build the table Z of centred rates (T rows, 14 columns) and the table L of intent labels (T rows, 2 columns). Fit D by least squares as in Chapter 1. Deploy the new D. In the paper the training sets were “typically composed of about 500” target acquisitions.
The device below makes this concrete. The top panel shows one trip of the arm-trained decoder.
Warm arrows are the velocities the cursor had. Teal arrows are the relabelled intents. The slider mixes the two kinds of label. At 0 the labels are the cursor’s velocities. At 1 they are the full intent labels. For each setting the page refits the decoder, then runs 64 closed-loop trials.
Top: one trip, with the cursor’s velocity (warm) and the intent label (teal) in each bin. Bottom: the hit rate of the refit decoder for each mix of labels. Drag the mix.
Toy, seeded. The labels are a straight mix of the observed velocity and the rotated intent label, computed bin by bin. Each setting refits on 160 trials of the arm-trained decoder’s closed-loop data, then runs 64 new trials with a 5 s limit. The rotation rule is the one Gilja et al. describe. The hit rates belong to this toy.
At 0% the refit decoder matches the arm-trained one: it learned its own output. As the share of intent grows, the hit rate climbs. The fit now sees which way the user was heading, so it moves the weights toward the user’s real map.
It helps to see why this is not circular. The new label uses one piece of information the decoder never had: where the target is. The label does not have to be perfect. It only has to be closer to the truth than the cursor’s own velocity. In the worked bin the observed arrow was 15 degrees off the straight line, and the label is on the line. A decoder fitted to thousands of such bins averages away small mistakes in the guess, much as it averages away noise in the spikes.
Intention estimation rests on an assumption: the user aims straight at the target. If the user takes a curved path on purpose, to avoid an obstacle for example, the label is wrong. The method also needs known targets, which a calibration task provides and free use does not. Chapter 8 returns to this limit. The point to keep is the principle: put information into the labels that the old decoder did not have.
Chapter 4
Smooth a jittery decoder by blending each new reading with a prediction, and tune how much to trust each
Watch the cursor in the hero when the neural noise is high. It trembles. Each 50 ms bin gives a fresh, noisy reading, and the cursor follows every wobble. A real hand does not move that way. Hands speed up and slow down smoothly. A decoder that knows this can ignore some of the shake.
The Kalman filter is the standard tool for this. If you have met it, skip ahead to the equations. If not, here is the whole idea in three steps. First, predict: from the last estimate and a rule for how things move, guess where the velocity is now. Second, measure: read the neurons for a noisy hint of the velocity. Third, blend: move the guess toward the hint by a fraction.
That fraction is the Kalman gain, written K. It is a trust slider. At K = 1 you believe the new reading completely. At K = 0 you ignore it and trust your prediction. The filter chooses K from two numbers: how noisy the readings are, and how fast the true velocity can change. Noisy readings give a small K. A fast-changing world gives a large K.
A full treatment lives in the Kalman filter lesson, and the sibling lesson on the Kalman decoder builds the whole decoder for neural data. Here we need only the pieces that matter for the loop.
A Kalman decoder holds two models. The first says how the hidden state moves. The state x is the cursor velocity, with an x part and a y part. The second says how the state shows up in the neurons.
Notice that the observation model runs from state to spikes. It is the generative story, not the decoder of Chapter 1. We fit it by least squares too, but the other way round: regress the spike rates on the velocity labels to get C. Then A and the nudge size come from how the label velocities change from bin to bin, and the noise q comes from what C leaves unexplained. Wu and colleagues put cursor decoding into this state-space form in 2006.
Here is how the motion model is fitted, in one small case. Suppose the label velocities along one axis run 0.80, 0.72, 0.64, 0.58. Regress each value on the one before it: the pairs are (0.80, 0.72), (0.72, 0.64) and (0.64, 0.58). The best slope is the sum of products divided by the sum of squares, 1.408 / 1.568 = 0.90. So A is about 0.90: the velocity keeps nine tenths of its value from bin to bin. The leftover errors give the size of the nudge w.
So the Kalman decoder needs labels in exactly one place: the fit of A, C and the noise sizes. This is why Chapter 3 matters. The better the labels, the better C, and the better K.
Take one axis of the velocity.
The model predicts 0.40 for this bin. The neurons suggest 0.80. Say the filter has settled on K = 0.25. The blend is 0.40 + 0.25 × (0.80 − 0.40) = 0.50. The estimate moves a quarter of the way to the reading. With K = 0.5 it would be 0.60.
Where does a steady K come from? For a velocity that changes by a random nudge of variance q each bin, read with noise variance 1, the long-run gain solves P² + qP − q = 0, and K equals P. For q = 0.1 this gives P = (−0.1 + √0.41) / 2 = 0.270, so K = 0.27. For q = 1 it gives K = 0.62. A larger q means a more agile world, so the filter trusts new readings more.
The device below applies this blend to the decoded velocity, one axis at a time, inside the closed loop. Slide q and watch the smoothness and the time to target trade against each other.
Left: one trip at the chosen gain. Right: the wobble of the cursor and the mean time to target as the gain changes, across all eight targets. Drag the process noise q.
Toy, seeded. The intent-refit decoder of the hero page feeds a one-axis Kalman blend with measurement noise 1 and process noise q (0.003 to 3). The gain K is the steady-state value from the formula above. Wobble is the mean change in step size from one bin to the next, in thousandths of a screen unit. The paper filters the neural rates directly; this shows the same smoothing on the decoder’s output. Illustrative.
At a high gain the cursor follows every wobble. At a low gain the cursor glides but reacts late, so it takes longer to arrive and, at the extreme, may miss. The best setting sits between the ends. It also depends on the user: a person who steers in quick moves needs a higher gain than one who steers slowly.
The filter adds a delay to the loop. A user who sees the cursor 150 ms late already pays for the screen. A low gain adds more lag, and a user steering a laggy cursor may overshoot, the way a driver does on an icy road. So the best gain for a closed loop is not the one that gives the smoothest picture. You must tune it with the person steering. That is one more reason to test in the loop.
Chapter 5
Assemble the two changes that Gilja and colleagues made to a Kalman decoder, and see what each one buys
Two monkeys sit before a screen. Each has 96-electrode Utah arrays in premotor and motor cortex. A cursor must reach a target and stay there for half a second. With a standard velocity Kalman filter, the cursor wanders to the target and often shoots past it. With the new decoder, it glides in and stops.
That decoder is ReFIT-KF, short for Recalibrated Feedback Intention-Trained Kalman Filter, from Gilja and colleagues in Krishna Shenoy’s group at Stanford (Nature Neuroscience, 2012). The name lists its ingredients. It is recalibrated with feedback from the closed loop, and trained on estimated intention. This chapter builds it from Chapters 3 and 4.
Stage one builds a first decoder with no neural control at all. The paper fits it either from arm reaches or from watching an automated cursor with the arms held still. The authors found observation-based training gave performance “roughly equivalent” to arm-based training.
Stage two puts the monkey in the loop. The monkey steers the cursor with the stage-one decoder. The session logs neural data, cursor positions and target positions. Then the team fits ReFIT-KF on the logged data. The paper reports training sets of about 500 target acquisitions.
So there are two fits, and the second is the one that counts. It is the same loop you saw in Chapter 3: record in the loop, relabel, refit.
Why two stages? The stage-one decoder only has to be good enough to let the monkey reach most targets. A decoder that cannot move the cursor at all would produce data in which the user never reaches the targets, and the labels would be poor. A decoder that is merely rough produces closed-loop data in which the neurons show what the user does when the cursor responds. That data is what stage two needs. The authors hypothesised that starting from the arm-to-neuron relationship lets stage two reach a higher level of optimisation.
This is Chapter 3 applied to a Kalman filter. The labels that fit the matrices A, C and the noise sizes are the estimated intended velocities: the observed velocity turned toward the target, and zero on the target during the hold. The label zero matters for stopping. A decoder trained on cursor velocities sees overshoot and slow wobbling at the target and learns to reproduce them. A decoder trained on intent learns what hold-still looks like in the neurons.
The second change is subtler. A neuron can respond to where the cursor is, not only to where it is heading. If the decoder ignores this, it reads the position response as a velocity signal and the cursor drifts.
Gilja et al. wrote that they “explicitly model velocity as the user’s intention and cursor position as an additional variable”. The cursor position is simply the sum of past velocity outputs. The decoder knows it, so it can predict how much of each neuron’s rate the position explains and subtract it. What remains is the velocity signal.
Work the idea for one neuron. Suppose the neuron fires at baseline plus 6 spikes per second per unit of position. The cursor rests at x = 0.50 and the user intends no move. The neuron fires 3 spikes per second above baseline. A decoder that reads that as velocity says “move”. A decoder that subtracts 6 × 0.50 = 3 reads zero and holds still.
The device below tests this idea in a toy world where neurons respond strongly to position. A parked cursor and a user who intends nothing: how fast does each decoder think the cursor should move?
The user intends to stay still. The curves show the average speed each decoder outputs for a parked cursor at each position. Lower is better. Drag the position, and switch decoders on and off.
Toy, seeded, and deliberately harsh: here the neurons’ position terms are three times those of the other devices. Each curve averages the decoder’s output over 200 noisy readings with intent zero. This lesson quotes no per-change breakdown from the paper, which presents the two changes together; the toy only shows why position matters.
The arm-trained decoder drifts the most, and the drift grows with distance from the centre. The intent refit already removes most of it, because intent labels teach it that stillness is common. Adding the position term removes nearly all of the rest. The two changes work on different parts of the problem.
Take the failure mode that motivated it, as symptom, measurement and fix. The symptom: with the velocity Kalman filter the cursor reaches the target but overshoots, then has to be pulled back, so holds fail. The measurement: count trials that cross the target and return, and time to a completed hold. The fix: train on intent, with zero on the target, so the decoder has learned that arrival means stop. In the lesson’s toy the effect on path shape is easy to measure. Among the hits, the arm-trained decoder’s paths are 1.92 times as long as the straight line to the target, and the intent refit’s paths are 1.01 times as long.
All numbers here are from the paper’s text and its Table 1. In the centre-out task, ReFIT-KF performed at 75 to 85 percent of the native arm, and at least twice as well as the velocity Kalman filter. Within five seconds, success was above 99 percent against 95 percent for the velocity filter. In the harder pinball task, where targets can appear anywhere, the velocity filter’s success was below 40 percent, and in the maze task it could not complete the task. ReFIT-KF took 710 ms on average in pinball against 519 ms for the native arm. The authors also report repeatable high performance for years after implantation in two monkeys.
In 2015 Gilja and colleagues moved this system to two people with amyotrophic lateral sclerosis (ALS), who had microelectrode arrays in motor cortex (Nature Medicine). Their abstract reports the highest published performance by a person to date, more than double that of earlier pilot clinical trial participants.
Chapter 6
Find out what happens when the brain and the decoder both change to reduce the same error
Two people carry a sofa through a narrow hall. The sofa tilts. The front carrier sees the tilt and lifts. The rear carrier sees it too and lowers.
Each carrier fixes the tilt, but both fix it at once. The sofa tips the other way. They both correct again, and it tips back further. Two helpers, one error, and the result is worse than one helper alone.
A BCI has two learners in one loop. The user learns: neurons change how they fire so that the cursor goes where the user wants. The decoder learns: a refit changes the map so that the cursor goes where the neurons suggest. Both read the same error, which is the gap between the cursor and the target. This is co-adaptation.
In 2002 Taylor, Tillery and Schwartz let monkeys control a 3D cursor in real time from the activity of motor cortex cells, while the monkeys watched the cursor form its path. They found that the cells changed their tuning when they were used for brain control. The team’s control algorithm, which they called coadaptive, tracked these changes. Daily practice improved both movement accuracy and the directional tuning of the recorded cells. The monkeys also made long 3D movement sequences from far fewer cells than earlier estimates suggested.
Orsborn and colleagues studied the same theme in 2014 (Neuron). They used decoder adaptation to shape neural plasticity in two settings: changes in the recorded neurons, and changes in the control context. Monkeys steered a cursor with and without simultaneous arm movements. The abstract reports that useful plasticity can occur alongside decoder adaptation, giving better performance, skill retention and resistance to interference from native motor networks.
So the picture is hopeful. The brain learns the decoder, and a decoder that adapts well can help it learn. The question for an engineer is how to adapt without fighting the user.
Reduce the situation to one number. Let e be the mismatch in degrees between how the user’s neurons encode a move and how the decoder reads it. At e = 0 the two agree. Each batch of trials, the system measures e. The measurement reaches both learners one batch late.
The user moves toward the decoder by a fraction au of the mismatch they saw. The decoder moves toward the user by a fraction ad. Both steps shrink e, so the new mismatch is the old one minus both corrections, each based on the mismatch from one batch earlier.
Run it by hand with a starting mismatch of 30 degrees in the first two batches and g = 0.3. Batch 2: 30 − 0.3 × 30 = 21. Batch 3: 21 − 0.3 × 30 = 12. Batch 4: 12 − 0.3 × 21 = 5.7. Batch 5: 5.7 − 0.3 × 12 = 2.1. The mismatch dies away.
Now try g = 1.2.
Batch 2: 30 − 36 = −6. Batch 3: −6 − 36 = −42. Batch 4: −42 + 7.2 = −34.8. Batch 5: −34.8 + 50.4 = 15.6. Batch 6: 15.6 + 41.8 = 57.4. The mismatch swings from side to side and grows. The sofa has tipped.
The rule falls out of the algebra. With a one-batch delay, the mismatch decays for g below 1 and grows for g above 1. For g above 0.25 it also rings, swinging from positive to negative while it shrinks. Try the numbers yourself.
The line is the mismatch after each batch, starting at 30 degrees. Slide the user’s rate and the decoder’s rate, or pick a preset. Watch the line settle, ring or blow up.
Toy, not a model of any paper’s algorithm: one number for the mismatch, a one-batch measurement delay, no noise. It shows why two learners that share one error must keep their combined rate below 1. Real systems add noise, nonlinearity and slower or faster delays, so the exact limit moves.
With the decoder frozen (ad = 0), only the user learns, and the loop is calm for any user rate below 1. As the decoder speeds up, the shared error gets double-corrected. A decoder that updates faster than the user can follow adds to the problem it was meant to cure.
The same trouble shows up from the user’s side. The symptom: the person says the cursor keeps changing and they cannot get used to it. The measurement: the size of each update, written as the length of the change in the weight table divided by the length of the table. In the lesson’s toy, the intent refit changes the weights by 83 percent in one step, a very large move for someone who has just learned a map. A refit on the cursor’s own velocity changes them by about two parts in a million. The fix is to cap the size of each update, or blend the new weights with the old, so the user always meets a map that is mostly familiar.
There are three common habits, and they follow from the sofa. Update the decoder slowly compared with how fast the user learns. Update in batches rather than every bin, so the user sees a steady map for a while. And stop updating once performance is good, so the user can build a skill on a fixed target. These are engineering conclusions from the toy and the papers above, not rules quoted from them.
Chapter 7
Turn hits, misses and seconds into numbers you can compare between decoders, tasks and papers
Two runners finish a race. One ran 100 metres in 12 seconds. The other ran 400 metres in 50 seconds. Who is faster? The times alone cannot say, because the distances differ. You need a number that divides out the distance.
Cursor tasks have the same problem. A decoder that hits a large nearby target in half a second may be worse than one that hits a small distant target in a second. To compare them you need a measure that accounts for how hard the task was. Closed-loop BCI work uses a handful of measures for this, and this chapter builds them in order.
The success rate is the share of trials where the user held the target for the required time before the time limit. In the centre-out comparison of Gilja et al. the hold was 500 ms and the limit was 5 seconds. Their pinball task used a 2 second limit. The time to target is how long a successful trial took. Together these answer the plain questions: does it work, and how fast?
Add two softer measures. A path is straight if it runs close to the line from start to target. Overshoot counts trips that pass through the target and have to come back. The paper noted that the velocity Kalman filter cursor “often significantly overshot the target”, and that ReFIT-KF “stopped better”.
Success rate needs enough trials to mean anything.
Let p be the true hit rate and n the number of trials. The typical error of the measured rate is the square root of p(1 − p)/n. After 16 trials with 31% hits, that error is about 12 percentage points. The true rate could easily be 20% or 45%. In the paper, each monkey made 545 to 659 movements per method across four days for its main comparison. That many trials makes small differences believable.
Here are the simple measures on the hero page. The arm-trained decoder held the target on 5 of 16 trips, 31 percent, and the hits took 3.92 seconds on average. The intent refit held it on 16 of 16, and took 2.52 seconds. The measure of straightness is the path length divided by the straight-line distance to the target. A perfect path scores 1.00. Among the hits, the arm-trained decoder scored 1.92 and the intent refit 1.01. Four numbers describe the difference without a single appeal to R².
To compare across tasks, BCI papers borrow a rule from human-computer interaction: Fitts’s law. It says the time to point at a target depends on how hard the target is, and it measures hardness in bits. The index of difficulty of a target at distance D with width W is
ID = log2(D / W + 1) bits
Each doubling of the ratio D/W adds about one bit, the same way each extra binary digit doubles the number of values you can tell apart. Throughput is the difficulty divided by the time: bits per second. A far, small target is hard. Reaching it quickly earns a high throughput.
Work an example. Targets of width 0.24 at distance 0.70 have D/W = 2.92, so ID = log2(3.92) = 1.97 bits. If the mean time to hit is 2.5 seconds, throughput is 1.97 / 2.5 = 0.79 bits per second. If a better decoder cut the time to 1.25 seconds, throughput would double to 1.58.
The paper reports the metric directly. Its Table 1 lists an index of difficulty of 1.32 bits, centre-out acquire times of 0.76 to 0.89 seconds, and Fitts’s law throughput of 1.48 to 1.74 bits per second across six sessions. Check the arithmetic: 1.32 / 0.89 is 1.48, and 1.32 / 0.76 is 1.74. Use the device to explore your own numbers and the paper’s.
The picture shows the start, the target and the distance. Move the three sliders, or press a preset to load the paper’s index of difficulty and one of its acquire times.
The formula is Fitts’s law in the Shannon form. The paper’s values (index of difficulty 1.32 bits; acquire times 0.76 to 0.89 s; throughput 1.48 to 1.74 bits/s) are from its Table 1. The paper’s target geometry is not reproduced here, so the preset sets the difficulty directly and the picture is a stand-in. The starting values (0.70, 0.24, 2.5 s) are this lesson’s simulation and include its 500 ms hold.
Throughput has traps. It depends on how you count time: some papers include the hold and some stop at first touch. It depends on how you define the width, and on whether failures count. Two papers can quote bits per second for different tasks and rules. Before you compare numbers across papers, check the definitions.
Consider how throughput can mislead, as symptom, measurement and fix. The symptom: a new decoder shows a higher bits per second but users like it less. The measurement: look at the success rate and at the failed trials, which a time-to-target average drops. A decoder that gives up on hard targets raises the average speed of the easy ones. The fix is to report throughput together with success rate, and to compute the time over all trials, with failures counted at the time limit, as well as over hits only.
A fair test follows a few habits. Use the same task, targets and time limit for every decoder. Test the decoders on the same days, so the user’s skill and the electrodes match. Report the number of trials beside each rate. And, where you can, report long-run results, since a decoder that works on day one but not on day thirty is not a good decoder. The paper tracked performance across 280 experimental days.
Chapter 8
Turn the refit into a daily routine with a gate that decides whether a new decoder reaches the user
It is Monday morning. The user sits down. Since Friday, a few electrodes have picked up different neurons, and some neurons now fire a little differently. Friday’s decoder was good for Friday. Is it good for today? Nobody knows until the cursor moves.
Recordings change from day to day. Orsborn and colleagues treated this directly: one of their two settings was changing neural recordings, and they used decoder adaptation to cope. A BCI that works for years needs a routine that follows the neurons. This chapter lays out that routine and the one decision at its centre: should the new decoder replace the old one?
Gilja et al. report that training sets were typically about 500 target acquisitions. At the toy’s pace, 160 trials take around twelve minutes. A real routine has to balance data against the user’s time and tiredness.
The acceptance test has the problem from Chapter 7. A hit rate measured on a few trials is noisy. Say the old decoder hits 47% of 32 trials and the new one hits 100%. The difference is large compared with the noise. Say the new one hits 3 of 4 and the old one 2 of 4. That tells you nothing.
A simple gate compares the two rates with their combined error. Take the difference in hit rate, and divide by the typical error of that difference. Call the result z. If z is above 2, the new decoder is better with good confidence: deploy. If z is below −2, it is worse: reject. In between, you cannot tell yet: keep the old decoder and collect more trials.
For the error, use p̃ = (hits + 1) / (trials + 2) for each decoder, so that a perfect 4 of 4 does not look perfectly certain. The error of the difference is then the square root of p̃1(1 − p̃1)/n + p̃2(1 − p̃2)/n.
The current decoder is the arm-trained one. Choose a candidate, then slide the number of acceptance trials. The bars show hit rates with their error whiskers. The verdict follows the gate rule.
Toy, seeded. Both decoders face the same simulated user and neurons (noise level 5) and the same target order. The candidate was refit on 160 closed-loop trials. The error whiskers are one standard error. The z rule (deploy above 2, reject below −2) is a simple example, not the rule of any lab or company. With few trials the gate says it cannot tell, even for a decoder that is truly better.
The second candidate teaches the other lesson. A refit on cursor labels returns the old decoder, so its hit rate matches the old one at every trial count. The gate never deploys it. That is the right outcome: nothing improved, so nothing ships. Chapter 2 predicted this.
Think about what the log must hold, because a replay is only as good as the record. For each 50 ms bin store the 14 (or 140, or 1,000) firing rates as one row, the cursor position and velocity as four numbers, the target position as two, the decoder version, and the trial number. A table of 160 trials of about 50 bins each is then roughly 8,000 rows. With it, any day can be refit again with new labels, or tested against a new decoder offline, and only the closed-loop acceptance test needs the person.
A dead channel shows the fix path well. The symptom: one electrode reads a flat line, and the decoder weight on it multiplies noise or zero. The measurement: the smoke test flags a channel whose variance dropped to near zero. The fix: mask that channel, refit without it, and run the acceptance test before the new decoder replaces the old.
Some failures show up only over many days, so log numbers that move early.
Track the mean decoded speed. Intent labels keep the observed speed, so a weak decoder passes its weak speed to the next fit, and speeds could shrink over repeated refits. This is a caution from reasoning, not a result of the paper, so check it on your own data. Track the hold failures, where the cursor reaches the target and then slips off. Track the time to target. And keep a known-good decoder to roll back to.
The routine rests on calibration tasks, where the target is known. In free use there are no targets to point the labels at. Any method that estimates intention in free use needs another source of information, and it carries a risk: if the estimate is wrong, the decoder is trained toward the wrong thing. The honest position is that Chapter 3 gives a principle, and each system must find a safe way to apply it outside the lab.
Chapter 9
Review the whole loop in one page: the numbers, the code, the sources and the next lessons
You can now explain, from memory, why a BCI decoder must be judged with the user in the loop, and how one relabelling idea repaired it. The pieces are small. A decoder is a table of weights. Offline scores can flatter it. Intent labels add information that the decoder never had. A Kalman filter smooths it. Position is modelled apart. Two learners share one error and must not shout over each other. A gate decides what ships.
In an open-loop test, nobody sees the decoder’s output. In a closed loop, the user watches the cursor and corrects it, so the plant, the strategy and even the neurons can change. A decoder refit on the cursor’s own velocity returns itself. ReFIT (Gilja et al., 2012) refits on closed-loop data with intention estimation: each velocity is rotated toward the target, kept at its speed, and set to zero on the target. It also treats cursor position as a separate variable. Co-adaptation (Taylor et al., 2002; Orsborn et al., 2014) means the brain and the decoder both learn, so their rates must be balanced. Judge the loop with success rate, time to target and Fitts throughput, and gate every update on closed-loop evidence.
| What | Number | Where it comes from |
|---|---|---|
| Animals and arrays | 2 monkeys, 96-electrode Utah arrays | Gilja 2012 |
| Task rule | 500 ms hold; success counted within 5 s | Gilja 2012 |
| Training set for the refit | about 500 target acquisitions | Gilja 2012 |
| ReFIT-KF vs native arm, centre-out | 75 to 85% of native arm performance | Gilja 2012 |
| ReFIT-KF vs velocity Kalman filter | at least twice the performance; success above 99% vs 95% in 5 s | Gilja 2012 |
| Harder tasks | pinball: velocity filter success below 40%; maze: it could not complete it | Gilja 2012 |
| Pinball time to target | ReFIT-KF 710 ± 317 ms vs native arm 519 ± 196 ms | Gilja 2012 |
| Fitts throughput | 1.48 to 1.74 bits/s; index of difficulty 1.32 bits; acquire 0.76 to 0.89 s | Gilja 2012, Table 1 |
| Long run | performance tracked over 280 experimental days | Gilja 2012 |
| Brain adaptation | cell tuning changed under 3D brain control; practice improved accuracy | Taylor et al. 2002 |
| Hero toy | arm-trained 31% hits, intent refit 100% (16 trials) | This lesson, seeded toy |
| Refit on cursor labels | weights unchanged; R² 1.00 on its own labels | This lesson, Chapter 2 toy |
| Steady Kalman gain | K = 0.27 at q = 0.1; 0.62 at q = 1 | Formula, Chapter 4 |
| Two-learner limit | stable only if au + ad is below 1 (one-batch delay) | Chapter 6 toy |
This sketch builds the intent labels and refits a linear decoder. It is the idea of Chapter 3 in a few lines of NumPy, not the paper’s full Kalman fit.
pythonimport numpy as np def intent_labels(v_obs, pos, target, hold_radius): # v_obs, pos, target: (T, 2) arrays, one row per 50 ms bin to_target = target - pos dist = np.linalg.norm(to_target, axis=1, keepdims=True) speed = np.linalg.norm(v_obs, axis=1, keepdims=True) labels = to_target / np.maximum(dist, 1e-9) * speed # rotate toward the target, keep the speed labels[dist[:, 0] < hold_radius] = 0.0 # on the target: intend to stay still return labels def refit(rates, baseline, v_obs, pos, target, hold_radius, ridge=1.0): Z = rates - baseline # (T, 14) centred firing rates L = intent_labels(v_obs, pos, target, hold_radius) # (T, 2) what the user meant A = Z.T @ Z + ridge * np.eye(Z.shape[1]) return (L.T @ Z) @ np.linalg.inv(A) # D: (2, 14) def gate(hits_new, hits_old, n): p1, p2 = (hits_new + 1) / (n + 2), (hits_old + 1) / (n + 2) se = np.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n) z = (p1 - p2) / se return "deploy" if z > 2 else "reject" if z < -2 else "cannot tell yet"
This lesson sits between the decoders and the systems. Before it come the linear decoders and the Kalman decoder, which build the models we refit here. After it comes the point-process GLM, a way to model spikes without binning them. Underneath sit the neural data lessons: spikes to rates and neural manifolds. The theme of the loop returns wherever a system learns from its own outputs: the decoder trains on data that its previous version shaped.
I opened these myself. Each line says what it supports.
One correction to a common note: a reading list I was given quotes ReFIT at “about 4 bits/s”. The paper’s Table 1 gives 1.48 to 1.74 bits/s of Fitts throughput, so this lesson uses those. Companies that build BCI products publish their own figures; treat any such figure as a company claim until you have read a paper that backs it.
Now press Present or Teach and explain the loop out loud, from memory: why an offline score can flatter a decoder, what an intent label is and how you build one, why a cursor-label refit changes nothing, what the two ReFIT changes are, and how two learners can shake a loop apart. Then go back to the cursor at the top and run all three decoders again.